Dot-line feature fusion type visual inertia mileage measuring and calculating method and system based on Fast-EDLines algorithm

Abstract

The invention discloses a point-line feature fusion type visual inertia mileage measuring and calculating method and system based on a Fast-EDLines algorithm, and the method comprises the steps: carrying out the Gaussian filtering smoothing of a gray collection image, obtaining a processed image, carrying out the gradient calculation of the processed image, and obtaining a gradient amplitude; screening the significant gradient change pixel points according to the gradient magnitude and obtaining edge points, and performing edge drawing according to the edge points and obtaining an edge pixel chain; according to the method, a function of performing point-line feature fusion type visual inertia mileage measurement and calculation on image data acquired by an image sensor based on a Fast-EDLines algorithm is effectively realized, and continuous long line segment features can be extracted by accelerating gradient calculation through an AVX2 instruction set and combining a long and short line segment merging strategy; compared with a traditional LSD algorithm, the long-line segment detection efficiency in the dark light environment is improved by several times, the geometric constraint of front-end pose estimation is remarkably enhanced, the positioning error is reduced, and the method is suitable for being widely popularized and used.

Description

Technical Field

[0001] The present invention relates to the technical field of image sensor data processing, and in particular to a point-line feature fusion visual-inertial odometry method and system based on the Fast-EDLines algorithm. Background Art

[0002] Visual Simultaneous Localization and Mapping (Visual SLAM) is a technology that uses cameras (monocular, binocular, RGB-D, etc.) as primary sensors to estimate a vehicle's motion trajectory in real time and simultaneously construct a map of its surroundings. It is widely used in robotics, autonomous driving, augmented reality (AR), virtual reality (VR), and other fields. Despite decades of development, SLAM technology still faces numerous challenges in practical application.

[0003] At present, the existing visual SLAM system is highly sensitive to illumination changes, texture sparseness and carrier motion noise, and the image contrast is reduced in low-light environments, which makes feature extraction easily fail. At the same time, the number of trackable corner points in low-texture scenes is also small, which easily causes front-end tracking failure. The jitter of the handheld device not only aggravates the motion blur and feature loss, but also causes the accumulation of pose estimation errors. Most existing visual SLAMs rely on point features for pose estimation, which makes it easy to mismatch in scenes with repeated textures or structured scenes. For example, the point features of scenes such as corridors and walls lack uniqueness, which easily leads to inter-frame matching errors. Therefore, it is necessary to design a point-line feature fusion visual-inertial odometer method and system based on the Fast-EDLines algorithm. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and to better and effectively solve the problems that the existing visual SLAM system is highly sensitive to illumination changes, sparse textures and carrier motion noise, and the image contrast is reduced in low-light environments, which makes feature extraction easily fail. At the same time, the number of trackable corner points in low-texture scenes is also small, which easily leads to front-end tracking failure. The jitter of the handheld device not only aggravates the motion blur and feature loss, but also causes the accumulation of pose estimation errors. The existing visual SLAM mostly relies on point features for pose estimation, which leads to mismatching in scenes with repeated textures or structured scenes, such as corridors, walls, etc. The lack of uniqueness of features can easily lead to inter-frame matching errors. In this paper, a point-line feature fusion visual-inertial odometry method and system based on the Fast-EDLines algorithm are provided. The method realizes the function of using the point-line feature fusion visual-inertial odometry based on the Fast-EDLines algorithm for image data collected by the image sensor. The AVX2 instruction set is used to accelerate gradient calculation and combine the long and short line segment merging strategy to extract continuous long line segment features. Not only is the efficiency of long line segment detection in dark environments improved by several times compared with the traditional LSD algorithm, but it also significantly enhances the geometric constraints of the front-end pose estimation and reduces the positioning error.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] A point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm includes the following steps:

[0007] Step A, performing Gaussian filtering and smoothing processing on the grayscale collected image to obtain a processed image, and then performing gradient calculation on the processed image to obtain a gradient amplitude;

[0008] Step B: Screening pixels with significant gradient changes according to gradient amplitude and obtaining edge points, and then performing edge drawing based on the edge points to obtain edge pixel chains;

[0009] Step C: extracting the edge pixel chain using a linear least squares fitting method to obtain a first set of straight line segments, and then removing short line segments from the first set of straight line segments according to the Helmholtz principle to obtain a second set of straight line segments;

[0010] Step D, using the false alarm number NFA to verify the significance of the straight line ends in the second straight line segment set and eliminate the pseudo line segments in the second straight line segments to obtain a third straight line segment set;

[0011] Step E: performing line segment matching and fusion on the line segments in the third line segment set and eliminating isolated line segments to obtain a fourth line segment set;

[0012] Step F, using the Shi-Tomasi corner detection method to extract corner points in the processed image and use them as a first feature point set, then tracking, eliminating, screening, and supplementing the first feature point set to obtain a second feature point set;

[0013] Step G, using discrete time integration to perform recursive integration calculation on the inertial measurement values ​​between consecutive image frames and obtain IMU pre-integral;

[0014] In step H, the fourth straight line segment set, the second feature point set, and the IMU pre-integral are input as observation quantities into the back-end optimization module to complete the visual inertial odometry operation.

[0015] The aforementioned point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, step A, performs Gaussian filtering and smoothing on the grayscale image to obtain a processed image, and then performs gradient calculation on the processed image to obtain the gradient amplitude. The specific steps are as follows:

[0016] Step A1, performing Gaussian filtering and smoothing processing on the grayscale collected image to obtain a processed image, wherein the Gaussian filtering and smoothing processing specifically uses a 5*5 Gaussian filter for smoothing processing;

[0017] Step A2, calculate the gradient of the processed image and obtain the gradient amplitude, wherein the gradient calculation is shown in formula (1):

[0018]

[0019] Among them, I(x, y) represents the grayscale value of the pixel point (x, y) in the image, G(x, y) represents the gradient direction of the pixel point (x, y), and g x (x, y) represents the horizontal gradient information at the pixel point (x, y), g y (x,y) represents the vertical gradient information at the pixel point (x,y), and g(x,y) represents the gradient amplitude.

[0020] In the aforementioned point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, step B is to screen pixels with significant gradient changes according to the gradient amplitude and obtain edge points, and then draw edges based on the edge points to obtain edge pixel chains. The specific steps are as follows:

[0021] Step B1: Filter pixels with significant gradient changes according to their gradient amplitude and obtain edge points. If the gradient amplitude of a pixel is greater than the gradient amplitude of its neighboring pixels and is also greater than a preset gradient threshold ρ, the pixel is identified as an edge point. The calculation process of the gradient threshold ρ is shown in formula (2).

[0022]

[0023] Among them, τ represents the angle tolerance and δ represents the boundary error;

[0024] Step B2: Draw the edge based on the edge points and obtain an edge pixel chain. Specifically, after calculating the gradient threshold ρ, traverse the gradient map and remove pixels with gradient values ​​less than the gradient threshold ρ, thereby obtaining the edge area of ​​the image. Then, the local gradient peak in the edge area is used as the anchor point and starting from the anchor point, pixels are gradually connected according to the gradient direction and amplitude to form an edge pixel chain.

[0025] In the aforementioned point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, step C uses the linear least squares fitting method to extract the edge pixel chain and obtain a first set of straight line segments. Then, according to the Helmholtz principle, short line segments in the first set of straight line segments are removed to obtain a second set of straight line segments. The specific steps are as follows:

[0026] Step C1: extract and obtain the first straight line segment set by using the linear least squares fitting method for the edge pixel chain. The specific steps are as follows:

[0027] Step C11: Assume that the edge pixel sequence obtained by edge detection is (x1, y1), (x2, y2) ... (x i ,y i ) and the current subsequence length is n, then the straight line fitting equation of the fitting line is shown in formula (3),

[0028] y=a·x+b (3)

[0029] Where a and b both represent coefficients;

[0030] Step C12, use least squares to solve coefficients a and b, set and Represent the mean of n pixels in the x and y directions respectively, and the solution process is as shown in formula (4).

[0031]

[0032] Step C13: Calculate the vertical distance d from each pixel to the straight line segment i , as shown in formula (5),

[0033]

[0034] In step C14, an error threshold δ is set. If the mean square error E of all pixels is not greater than the error threshold δ, the current subsequence can be fitted into a valid straight line segment. If the mean square error E of all pixels is greater than the error threshold δ, the current straight line segment is truncated and the above fitting process is repeated for the remaining pixels until the pixel chain is completely processed. The mean square error E is shown in formula (6).

[0035]

[0036] Step C2: According to the Helmholtz principle, the short segments in the first set of straight line segments are removed to obtain the second set of straight line segments. The minimum length of the straight line segment n is set to min , then according to the Helmholtz principle the minimum length n min The calculation process of is shown in formula (7).

[0037]

[0038] Where N represents the length of the image side and p represents the probability of the gradient direction being consistent.

[0039] In the aforementioned point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, step D uses the false alarm number (NFA) to verify the significance of the straight line ends in the second straight line segment set and eliminates the pseudo line segments in the second straight line segment to obtain the third straight line segment set. The specific steps are as follows:

[0040] Step D1 introduces the false alarm number NFA as a judgment criterion. Assuming that for a line segment A with a length of n, there are at least k pixels whose gradient direction is consistent with the line segment direction, the calculation process of the false alarm number NFA of line segment A is shown in formula (8):

[0041]

[0042] Among them, N 4 represents the number of potential straight line segments in the image, To represent the method of selecting i pixels from n pixels;

[0043] Step D2: Set the NFA threshold ∈. If the number of false alarms NFA for segment A is not greater than the NFA threshold ∈, segment A is significant and has a high confidence level, and thus the segment is retained. If the number of false alarms NFA for segment A is less than the NFA threshold ∈, the pseudo segment is removed.

[0044] In the aforementioned point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, step E is to perform line segment matching and fusion on the line segments in the third line segment set and eliminate isolated line segments to obtain the fourth line segment set. The specific steps are as follows:

[0045] Step E1: Assume that the initial line segment set extracted through line segment detection and verification is L = {l1, l2, ..., l n} and each line segment l is represented by the starting point and the end point as l = (s, e), according to the length threshold length threshold Divide the line segment set L into the long line segment set L by Euclidean distance calculation long and the set of short line segments L short , as shown in formula (9),

[0046]

[0047] Step E2, calculate the initialized fused line segment set L′, where L′=L long , specifically for the short line segment set L short Each short line segment l in short =(s short , e short )Calculation and long segment set L long Each long line segment l in long =(s long , e long ) and find the long line segment L that meets the direction and position conditions. long , the specific steps are as follows,

[0048] Step E21, find a long line segment L that meets the direction condition long , the specific steps are as follows,

[0049] Step E211: calculate the angle angle (l short , l long ) is calculated as shown in formula (10),

[0050]

[0051] Among them, the short line direction vector V short =e short -s short , long segment direction vector V long =e long -s long ;

[0052] Step E212, determine the angle angle (l short , l long ) is less than the angle threshold θ threshold =3°, as shown in formula (11),

[0053] angle(l short , l long )≤θ threshold+β (11)

[0054] Where β represents the small tolerance value; if formula (11) holds, it means that the short segment and the long segment are close in direction;

[0055] Step E22, find a long line segment that meets the position conditions, specifically calculate the point p (xp, yp ) to the shortest distance between line segment l = (s, e). The specific steps are as follows:

[0056] Step E221, calculate the direction vector and vector modulus of the line segment, as shown in formula (12) and formula (13),

[0057] V=es=(x e -x s ,y e -y s ) (12)

[0058] ||V|| 2 =(x e -x s ) 2 +(y e -y s ) 2 (13)

[0059] Among them, the coordinates of the starting point s are (x s ,y s ), the coordinates of the end point e are (x e ,y e ), V represents the displacement vector from the starting point s to the end point e;

[0060] Step E222, calculate the projection of the point to the line, specifically calculate the projection point of point p on the line pproj The normalized position parameter t of is shown in formula (14),

[0061]

[0062] The numerator is the inner product of the projection of point p onto the straight line, and the inner product represents the projection length of point p on the straight line. The denominator represents the square of the modulus of the line segment direction vector, and the square of the modulus is used to normalize the projection scale.

[0063] Step E223, if 0≤t≤1, then the projection point p proj (x proj ,y proj ) is located inside the line segment as shown in formula (15), and the shortest distance d(p, l) from the point to the line is shown in formula (16),

[0064] p proj=s+t·V=(x s +t·(x e -x s ), y s +t·(y e -y s )) (15)

[0065]

[0066] In step E224, if t<0, the projection point is located on the extension line of the starting point of the line segment, and the shortest distance is the length from the point to the starting point of the line segment, as shown in formula (17).

[0067]

[0068] Step E225: If t>1, the projection point is located on the extension line of the end point of the line segment, and the shortest distance is the length from the point to the end point of the line segment, as shown in formula (18).

[0069]

[0070] Step E226, calculate the short line segment l short Starting point short and the end point e short Arrival at long segment l long The specific formula is as follows:

[0071] dist(l short , l long )=min(d(s short , l long ), d(e short , l long )) (19)

[0072] Among them, dist(l short , l long ) is the shortest distance from the short segment to the long segment;

[0073] Step E227, determine the shortest distance dist(l short , l long ) is less than the distance threshold dist threshold =5, as shown in formula (20). If formula (20) is satisfied, it means that the short segment and the long segment are close in position. If not, the short segment is regarded as an isolated segment and removed.

[0074] dist(l short ,l iong )≤dist treshold (20);

[0075] Step E23, if there is a long line segment l long If both the direction condition and the position condition are met, the short line segment l short Merge to long line segment l long Then, set the maximum number of iterations for merging line segments and remove the line segments involved in the merging from the third straight line segment set. Update the endpoints of the fused long line segments as shown in formula (21).

[0076]

[0077] Among them, (S new , S end ) are the coordinates of the endpoints of the fused long line segment.

[0078] In the aforementioned point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, step F uses the Shi-Tomasi corner detection method to extract corner points in the processed image and use them as the first feature point set. The first feature point set is then tracked, eliminated, filtered, and supplemented to obtain a second feature point set. The specific steps are as follows:

[0079] Step F1: Use the Shi-Tomasi corner detection algorithm to extract corner points in the image and use the KLT sparse optical flow method to track the feature points. Then, use the image pyramid structure to enhance the scale invariance of the point features. If tracking fails or the feature points exceed the image boundary during the feature tracking process, invalid points are removed using the feature tracking status mark.

[0080] Step F2: Use the basic matrix to eliminate outliers and improve the accuracy of feature matching. Then, filter the extracted corner points and sort the tracking points according to the number of tracking times to optimize the distribution of feature points.

[0081] Step F3: Perform homogenization processing on the densely populated areas of the feature set, and then dynamically add new extracted corner points.

[0082] In the aforementioned point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, step G utilizes discrete-time integration to perform recursive integration calculations on the inertial measurement values ​​between consecutive image frames and obtain IMU pre-integration. Specifically, by accumulating accelerometer and gyroscope data between adjacent key frames, the relative motion parameters of the sensor at adjacent image acquisition moments can be calculated. The relative motion parameters include displacement, velocity, and attitude change.

[0083] In the aforementioned point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, in step H, the fourth straight line segment set, the second feature point set, and the IMU pre-integration are input as observation quantities into the back-end optimization module to complete the visual-inertial odometry operation. The specific implementation process of the back-end optimization module is implemented using the VINS-Mono algorithm.

[0084] A point-line feature fusion visual-inertial odometry system based on the Fast-EDLines algorithm includes a gradient calculation module, an edge drawing module, a line segment extraction module, a line segment verification module, a line segment fusion module, a feature point extraction module, an IMU pre-integration acquisition module and an observation quantity optimization module. The gradient calculation module is used to perform Gaussian filtering and smoothing on the grayscale acquisition image to obtain a processed image, and then perform gradient calculation on the processed image to obtain a gradient amplitude; the edge drawing module is used to screen pixels with significant gradient changes according to the gradient amplitude and obtain edge points, and then perform edge drawing based on the edge points to obtain an edge pixel chain; the line segment extraction module is used to extract and obtain a first straight line segment set from the edge pixel chain using a linear least squares fitting method, and then remove short line segments in the first straight line segment set according to the Helmholtz principle to obtain a second straight line segment set; the line segment verification module is used to perform a linear least squares fitting method on the edge pixel chain to obtain a first straight line segment set, and then remove short line segments in the first straight line segment set according to the Helmholtz principle to obtain a second straight line segment set; The block is used to verify the significance of the straight line ends in the second straight line segment set by using the false alarm number NFA and eliminate the pseudo line segments in the second straight line segment to obtain the third straight line segment set; the line segment fusion module is used to perform line segment matching and fusion on the straight line segments in the third straight line segment set and eliminate isolated line segments to obtain the fourth straight line segment set; the feature point extraction module is used to extract the corner points in the processed image by using the Shi-Tomasi corner point detection method and use them as the first feature point set, and then track, eliminate, filter and supplement the feature points of the first feature point set to obtain the second feature point set; the IMU pre-integration acquisition module is used to use discrete time integration to perform recursive integration calculation on the inertial measurement values ​​between consecutive image frames and obtain the IMU pre-integral; the observation quantity optimization module is used to input the fourth straight line segment set, the second feature point set and the IMU pre-integral as observation quantities into the back-end optimization module and complete the visual inertial odometry operation.

[0085] The beneficial effects of the present invention are as follows: a point-line feature fusion visual-inertial odometry method and system based on the Fast-EDLines algorithm of the present invention first performs Gaussian filtering and smoothing processing on the grayscale acquisition image to obtain a processed image, then performs gradient calculation on the processed image to obtain the gradient amplitude, then screens the pixels with significant gradient changes according to the gradient amplitude and obtains the edge points, then draws the edge according to the edge points and obtains the edge pixel chain, then extracts the edge pixel chain using the linear least squares fitting method and obtains the first straight line segment set, and then fits the first straight line segment set according to the Helmholtz principle. The short line segments in the image are removed to obtain the second set of straight line segments. Then, the false alarm number NFA is used to verify the significance of the straight line ends in the second set of straight line segments and the pseudo line segments in the second set of straight line segments are eliminated to obtain the third set of straight line segments. Then, the straight line segments in the third set of straight line segments are matched and fused and isolated line segments are eliminated to obtain the fourth set of straight line segments. Then, the Shi-Tomasi corner point detection method is used to extract the corner points in the processed image and use them as the first set of feature points. The feature points of the first set of feature points are tracked, eliminated, screened and supplemented to obtain the second set of feature points. Finally, the continuous image is integrated using discrete time integral. The inertial measurement values ​​between image frames are recursively integrated and IMU pre-integral is obtained. The fourth straight line segment set, the second feature point set and the IMU pre-integral are then input into the back-end optimization module as observation quantities to complete the visual inertial odometer operation. The visual inertial odometer measurement method and system are effectively realized. The system has the function of using the point-line feature fusion visual inertial odometer measurement based on the Fast-EDLines algorithm for the image data collected by the image sensor, and the gradient calculation is accelerated by the AVX2 instruction set and the long and short line segment merging strategy is combined to extract the continuous long line segment features, which solves the problem of point feature fusion in low-texture scenes in the existing technology. The problem of sparse features; the efficiency of long line segment detection in dark light environment is improved by several times compared with the traditional LSD algorithm, and the proportion of long line segments is 100%, which significantly enhances the geometric constraints of front-end pose estimation; the present invention can significantly improve the accuracy of carrier motion estimation in weak texture and dark light environment through the fusion of point and line features, and reduces the positioning error compared with the traditional single-point feature method. At the same time, the trajectory drift is effectively suppressed, ensuring the stable operation of the system, solving the bottlenecks of existing technologies in positioning accuracy, real-time performance and equipment adaptability in complex environments, and breaking through the limitations of traditional point feature dependence and environmental interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0086] Figure 1 This is an overall flow chart of the point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm of the present invention;

[0087] Figure 2This is a schematic diagram of the operation process of a point-line feature fusion visual-inertial odometry system based on the Fast-EDLines algorithm of the present invention;

[0088] Figure 3 This is the code diagram of the algorithm for accelerating the Sobel gradient calculation process using AVX2 of the present invention;

[0089] Figure 4 It is the algorithm code diagram of the long and short line processing process of the present invention;

[0090] Figure 5 This is a comparison diagram of the detection algorithms of LSD, EDLines, and Fast-EDLines proposed in the embodiment of the present invention;

[0091] Figure 6 It is a sequence plan diagram of underground space A and a motion trajectory diagram in an embodiment of the present invention;

[0092] Figure 7 2. It is a schematic diagram of the comparison of VIO trajectories of sequence A in the underground space according to an embodiment of the present invention;

[0093] Figure 8 It is a schematic diagram of the evo measurement results of the A-sequence trajectory of the underground space in an embodiment of the present invention. DETAILED DESCRIPTION

[0094] The present invention will be further described below with reference to the accompanying drawings.

[0095] like Figure 1-2 As shown, the present invention provides a point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, comprising the following steps:

[0096] like Figure 3 As shown, in step A, the grayscale acquisition image is smoothed by Gaussian filtering to obtain a processed image, and then the gradient of the processed image is calculated to obtain the gradient amplitude. The specific steps are as follows:

[0097] Step A1, performing Gaussian filtering and smoothing processing on the grayscale collected image to obtain a processed image, wherein the Gaussian filtering and smoothing processing specifically uses a 5*5 Gaussian filter for smoothing processing;

[0098] It effectively suppresses high-frequency noise and random noise, thereby improving image quality and providing a stable foundation for subsequent gradient calculation;

[0099] Step A2, calculate the gradient of the processed image and obtain the gradient amplitude, wherein the gradient calculation is shown in formula (1):

[0100]

[0101] Among them, I(x, y) represents the grayscale value of the pixel point (x, y) in the image, G(x, y) represents the gradient direction of the pixel point (x, y), and g x (x, y) represents the horizontal gradient information at the pixel point (x, y), g y (x, y) represents the vertical gradient information at the pixel point (x, y), and g(x, y) represents the gradient amplitude.

[0102] The traditional method needs to calculate the horizontal gradient, vertical gradient and gradient amplitude separately, resulting in repeated pixel reading and multiple memory accesses. The present invention fully utilizes the 256-bit parallel computing capability of the AVX2 instruction set, loading 16 pixels at a time, realizing the joint calculation of horizontal and vertical gradients and their absolute values, and directly obtaining the gradient amplitude through the _mm256_abs_epi16 function, thereby greatly reducing the calculation steps and memory overhead.

[0103] Step B: Filter the pixels with significant gradient changes according to the gradient amplitude and obtain edge points, then draw edges based on the edge points and obtain edge pixel chains. The specific steps are as follows:

[0104] Step B1: Filter pixels with significant gradient changes according to their gradient amplitude and obtain edge points. If the gradient amplitude of a pixel is greater than the gradient amplitude of its neighboring pixels and is also greater than a preset gradient threshold ρ, the pixel is identified as an edge point. The calculation process of the gradient threshold ρ is shown in formula (2).

[0105]

[0106] Among them, τ represents the angle tolerance and δ represents the boundary error;

[0107] Step B2: Draw the edge based on the edge points and obtain an edge pixel chain. Specifically, after calculating the gradient threshold ρ, traverse the gradient map and remove pixels with gradient values ​​less than the gradient threshold ρ, thereby obtaining the edge area of ​​the image. Then, the local gradient peak in the edge area is used as the anchor point and starting from the anchor point, pixels are gradually connected according to the gradient direction and amplitude to form an edge pixel chain.

[0108] The maximum boundary error between two consecutive pixels is usually 2, and according to the setting parameters in the EDLines algorithm and the LSD algorithm, when the experimental angle tolerance is τ = 22.5°, the length and alignment of the line segments are better;

[0109] Step C: extract the edge pixel chain using the linear least squares fitting method to obtain a first set of straight line segments, and then remove the short line segments in the first set of straight line segments according to the Helmholtz principle to obtain a second set of straight line segments. The specific steps are as follows:

[0110] Step C1: extract and obtain the first straight line segment set by using the linear least squares fitting method for the edge pixel chain. The specific steps are as follows:

[0111] Step C11: Assume that the edge pixel sequence obtained by edge detection is (x1, y1), (x2, y2) ... (x i ,y i ) and the current subsequence length is n, then the straight line fitting equation of the fitting line is shown in formula (3),

[0112] y=a·x+b (3)

[0113] Where a and b both represent coefficients;

[0114] Step C12, use least squares to solve coefficients a and b, set and Represent the mean of n pixels in the x and y directions respectively, and the solution process is as shown in formula (4).

[0115]

[0116] Step C13: Calculate the vertical distance d from each pixel to the straight line segment i , as shown in formula (5),

[0117]

[0118] In step C14, an error threshold δ is set. If the mean square error E of all pixels is not greater than the error threshold δ, the current subsequence can be fitted into a valid straight line segment. If the mean square error E of all pixels is greater than the error threshold δ, the current straight line segment is truncated and the above fitting process is repeated for the remaining pixels until the pixel chain is completely processed. The mean square error E is shown in formula (6).

[0119]

[0120] Step C2: According to the Helmholtz principle, the short segments in the first set of straight line segments are removed to obtain the second set of straight line segments. The minimum length of the straight line segment n is set to min , then according to the Helmholtz principle the minimum length n min The calculation process of is shown in formula (7).

[0121]

[0122] Where N represents the length of the image side and p represents the probability of the gradient direction being consistent.

[0123] like Figure 4As shown, in step D, the false alarm number NFA is used to verify the significance of the straight line ends in the second straight line segment set and to eliminate the pseudo line segments in the second straight line segment to obtain the third straight line segment set. The specific steps are as follows:

[0124] Step D1 introduces the false alarm number NFA as a judgment criterion. Assuming that for a line segment A with a length of n, there are at least k pixels whose gradient direction is consistent with the line segment direction, the calculation process of the false alarm number NFA of line segment A is shown in formula (8):

[0125]

[0126] Among them, N 4 represents the number of potential straight line segments in the image, To represent the method of selecting i pixels from n pixels;

[0127] Step D2: Set the NFA threshold ∈. If the number of false alarms NFA for segment A is not greater than the NFA threshold ∈, segment A is significant and has a high confidence level, and thus the segment is retained. If the number of false alarms NFA for segment A is less than the NFA threshold ∈, the pseudo segment is removed.

[0128] Step E: performing line segment matching and fusion on the line segments in the third line segment set and eliminating isolated line segments to obtain a fourth line segment set. The specific steps are as follows:

[0129] Step E1: Assume that the initial line segment set extracted after line segment detection and verification is L = {l1, l2, ..., l n} and each line segment l is represented by the starting point and the end point as l = (s, e), according to the length threshold length threshold Divide the line segment set L into the long line segment set L by Euclidean distance calculation long and the set of short line segments L short , as shown in formula (9),

[0130]

[0131] Step E2, calculate the initialized fused line segment set L′, where L′=L long , specifically for the short line segment set L short Each short line segment l in short =(s short , e short )Calculation and long segment set L long Each long line segment l in long =(s long , e long ) and find the long line segment L that meets the direction and position conditions. long , the specific steps are as follows,

[0132] Step E21, find a long line segment L that meets the direction condition long , the specific steps are as follows,

[0133] Step E211: calculate the angle angle (l short , l long ) is calculated as shown in formula (10),

[0134]

[0135] Among them, the short line direction vector V short =e short -s short , long segment direction vector V long =e long -s long ;

[0136] Step E212, determine the angle angle (l short , l long ) is less than the angle threshold θ threshold =3°, as shown in formula (11),

[0137] angle(l short , l long )≤θ threshold +β (11)

[0138] Where β represents the small tolerance value; if formula (11) holds, it means that the short segment and the long segment are close in direction;

[0139] Step E22, find a long line segment that meets the position conditions, specifically calculate the point p (xp, yp ) to the shortest distance between line segment l = (s, e). The specific steps are as follows:

[0140] Step E221, calculate the direction vector and vector modulus of the line segment, as shown in formula (12) and formula (13),

[0141] V=es=(x e -x s ,y e -y s ) (12)

[0142] ||V|| 2 =(x e -x s ) 2 +(y e -y s ) 2 (13)

[0143] Among them, the coordinates of the starting point s are (x s ,y s ), the coordinates of the end point e are (x e ,y e ), V represents the displacement vector from the starting point s to the end point e;

[0144] Step E222, calculate the projection of the point to the line, specifically calculate the projection point of point p on the line pproj The normalized position parameter t of is shown in formula (14),

[0145]

[0146] The numerator is the inner product of the projection of point p onto the straight line, and the inner product represents the projection length of point p on the straight line. The denominator represents the square of the modulus of the line segment direction vector, and the square of the modulus is used to normalize the projection scale.

[0147] Step E223, if 0≤t≤1, then the projection point p proj (x proj ,y proj ) is located inside the line segment as shown in formula (15), and the shortest distance d(p, l) from the point to the line is shown in formula (16),

[0148] p proj =s+t·V=(x s +t·(x e -x s ), y s +t·(y e -y s )) (15)

[0149]

[0150] In step E224, if t<0, the projection point is located on the extension line of the starting point of the line segment, and the shortest distance is the length from the point to the starting point of the line segment, as shown in formula (17).

[0151]

[0152] Step E225: If t>1, the projection point is located on the extension line of the end point of the line segment, and the shortest distance is the length from the point to the end point of the line segment, as shown in formula (18).

[0153]

[0154] Step E226, calculate the short line segment l short Starting point short and the end point e short Arrival at long segment l longThe specific formula is as follows:

[0155] dist(l short , l long )=min(d(s short , l long ), d(e short , l long )) (19)

[0156] Among them, dist(l short , l long ) is the shortest distance from the short segment to the long segment;

[0157] Step E227, determine the shortest distance dist(l short , l long ) is less than the distance threshold dist threshold =5, as shown in formula (20). If formula (20) is satisfied, it means that the short segment and the long segment are close in position. If not, the short segment is regarded as an isolated segment and removed.

[0158] dist(l short , l long )≤dist threshold (20);

[0159] Step E23, if there is a long line segment l long If both the direction condition and the position condition are met, the short line segment l short Merge to long line segment l long Then, set the maximum number of iterations for merging line segments and remove the line segments involved in the merging from the third straight line segment set. Update the endpoints of the fused long line segments as shown in formula (21).

[0160]

[0161] Among them, (S new , S end ) are the coordinates of the endpoints of the fused long line segment.

[0162] Step F, using the Shi-Tomasi corner detection method to extract corner points in the processed image and use them as the first feature point set, then tracking, eliminating, screening and supplementing the first feature point set to obtain the second feature point set. The specific steps are as follows:

[0163] Step F1: Use the Shi-Tomasi corner detection algorithm to extract corner points in the image and use the KLT sparse optical flow method to track the feature points. Then, use the image pyramid structure to enhance the scale invariance of the point features. If tracking fails or the feature points exceed the image boundary during the feature tracking process, invalid points are removed using the feature tracking status mark.

[0164] Step F2: Use the basic matrix to eliminate outliers and improve the accuracy of feature matching. Then, filter the extracted corner points and sort the tracking points according to the number of tracking times to optimize the distribution of feature points.

[0165] Step F3: Perform homogenization processing on the densely populated areas of the feature set, and then dynamically add new extracted corner points.

[0166] Step G, using discrete time integration to perform recursive integration calculation on the inertial measurement values ​​between consecutive image frames and obtain IMU pre-integration, specifically by accumulating accelerometer and gyroscope data between adjacent key frames to calculate the relative motion parameters of the sensor at adjacent image acquisition moments, the relative motion parameters including displacement, velocity and attitude change.

[0167] In step H, the fourth straight line segment set, the second feature point set, and the IMU pre-integral are input as observation quantities into the back-end optimization module to complete the visual inertial odometry operation. The specific implementation process of the back-end optimization module adopts the VINS-Mono algorithm.

[0168] A point-line feature fusion visual-inertial odometry system based on the Fast-EDLines algorithm includes a gradient calculation module, an edge drawing module, a line segment extraction module, a line segment verification module, a line segment fusion module, a feature point extraction module, an IMU pre-integration acquisition module and an observation quantity optimization module. The gradient calculation module is used to perform Gaussian filtering and smoothing on the grayscale acquisition image to obtain a processed image, and then perform gradient calculation on the processed image to obtain a gradient amplitude; the edge drawing module is used to screen pixels with significant gradient changes according to the gradient amplitude and obtain edge points, and then perform edge drawing based on the edge points to obtain an edge pixel chain; the line segment extraction module is used to extract and obtain a first straight line segment set from the edge pixel chain using a linear least squares fitting method, and then remove short line segments in the first straight line segment set according to the Helmholtz principle to obtain a second straight line segment set; the line segment verification module is used to perform a linear least squares fitting method on the edge pixel chain to obtain a first straight line segment set, and then remove short line segments in the first straight line segment set according to the Helmholtz principle to obtain a second straight line segment set; The block is used to verify the significance of the straight line ends in the second straight line segment set by using the false alarm number NFA and eliminate the pseudo line segments in the second straight line segment to obtain the third straight line segment set; the line segment fusion module is used to perform line segment matching and fusion on the straight line segments in the third straight line segment set and eliminate isolated line segments to obtain the fourth straight line segment set; the feature point extraction module is used to extract the corner points in the processed image by using the Shi-Tomasi corner point detection method and use them as the first feature point set, and then track, eliminate, filter and supplement the feature points of the first feature point set to obtain the second feature point set; the IMU pre-integration acquisition module is used to use discrete time integration to perform recursive integration calculation on the inertial measurement values ​​between consecutive image frames and obtain the IMU pre-integral; the observation quantity optimization module is used to input the fourth straight line segment set, the second feature point set and the IMU pre-integral as observation quantities into the back-end optimization module and complete the visual inertial odometry operation.

[0169] like Figure 5-8 As shown, the specific embodiments of the present invention show that the efficiency of long line segment detection in dark light environments is improved by 3 times compared with the traditional LSD algorithm, and the proportion of long line segments reaches 100%, which significantly enhances the geometric constraints of front-end pose estimation; and through the fusion of point and line features, the present invention can significantly improve the accuracy of carrier motion estimation in weak texture and dark light environments; the implementation results show that compared with the traditional single-point feature method, the positioning error of the present invention is reduced by 63.99%, and the trajectory drift is effectively suppressed, ensuring the stable operation of the system.

[0170] In summary, the present invention provides a point-line feature fusion visual-inertial odometry method and system based on the Fast-EDLines algorithm. First, the grayscale acquisition image is smoothed by Gaussian filtering to obtain a processed image. Then, the processed image is gradient calculated to obtain the gradient amplitude. Then, the pixels with significant gradient changes are screened according to the gradient amplitude to obtain edge points. Then, the edge is drawn based on the edge points to obtain an edge pixel chain. Then, the edge pixel chain is extracted by the linear least squares fitting method to obtain a first straight line segment set. Then, the short straight line segments in the first straight line segment set are selected according to the Helmholtz principle. The line segments are removed to obtain the second set of straight line segments, and then the false alarm number NFA is used to verify the significance of the straight line ends in the second set of straight line segments and eliminate the pseudo line segments in the second set of straight line segments to obtain the third set of straight line segments. The straight line segments in the third set of straight line segments are then matched and fused, and isolated line segments are eliminated to obtain the fourth set of straight line segments. The Shi-Tomasi corner point detection method is then used to extract the corner points in the processed image and use them as the first set of feature points. The feature points of the first set of feature points are then tracked, eliminated, screened and supplemented to obtain the second set of feature points. Finally, discrete time integration is used to integrate the continuous image frames. The inertial measurement values ​​between the four points are recursively integrated to obtain the IMU pre-integral, and then the fourth straight line segment set, the second feature point set and the IMU pre-integral are input as observation quantities into the back-end optimization module to complete the visual inertial odometer measurement operation; the visual inertial odometer measurement method and system are effectively realized to have the function of using the point-line feature fusion visual inertial odometer measurement based on the Fast-EDLines algorithm for the image data collected by the image sensor, and the gradient calculation is accelerated by the AVX2 instruction set and combined with the long and short line segment merging strategy to extract the continuous long line segment features, which solves the problem of point features in low-texture scenes in the existing technology. The problem of sparseness is solved. The efficiency of long line segment detection in dark light environment is improved by several times compared with the traditional LSD algorithm, and the proportion of long line segments is 100%, which significantly enhances the geometric constraints of front-end pose estimation. The present invention can significantly improve the accuracy of carrier motion estimation in weak texture and dark light environment through the fusion of point and line features, and reduces the positioning error compared with the traditional single-point feature method. At the same time, the trajectory drift is effectively suppressed, ensuring the stable operation of the system, solving the bottlenecks of existing technologies in positioning accuracy, real-time performance and equipment adaptability in complex environments, and breaking through the limitations of traditional point feature dependence and environmental interference.

[0171] The basic principles, main features, and advantages of the present invention are shown and described above. Those skilled in the art should understand that the present invention is not limited to the foregoing embodiments. The foregoing embodiments and descriptions are merely illustrative of the principles of the present invention. Various changes and modifications may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and modifications are intended to fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. A point-line feature fusion visual-inertial odometry method based on the Fast-EDLines algorithm, characterized by: The following steps are included: Step A, performing Gaussian filtering and smoothing processing on the grayscale collected image to obtain a processed image, and then performing gradient calculation on the processed image to obtain a gradient amplitude; Step B: Screening pixels with significant gradient changes according to gradient amplitude and obtaining edge points, and then performing edge drawing based on the edge points to obtain edge pixel chains; Step C: extracting the edge pixel chain using a linear least squares fitting method to obtain a first set of straight line segments, and then removing short line segments from the first set of straight line segments according to the Helmholtz principle to obtain a second set of straight line segments; Step D, using the false alarm number NFA to verify the significance of the straight line ends in the second straight line segment set and eliminate the pseudo line segments in the second straight line segments to obtain a third straight line segment set; Step E: performing line segment matching and fusion on the line segments in the third line segment set and eliminating isolated line segments to obtain a fourth line segment set; Step F, using the Shi-Tomasi corner detection method to extract corner points in the processed image and use them as a first feature point set, then tracking, eliminating, screening, and supplementing the first feature point set to obtain a second feature point set; Step G, using discrete time integration to perform recursive integration calculation on the inertial measurement values ​​between consecutive image frames and obtain IMU pre-integral; In step H, the fourth straight line segment set, the second feature point set, and the IMU pre-integral are input as observation quantities into the back-end optimization module to complete the visual inertial odometry operation.

2. The method for visual-inertial odometry based on point-line feature fusion using the Fast-EDLines algorithm according to claim 1, characterized in that: Step A: Gaussian filter smoothing is performed on the grayscale image to obtain a processed image, and then gradient calculation is performed on the processed image to obtain the gradient amplitude. The specific steps are as follows: Step A1, performing Gaussian filtering and smoothing processing on the grayscale collected image to obtain a processed image, wherein the Gaussian filtering and smoothing processing specifically uses a 5*5 Gaussian filter for smoothing processing; Step A2, calculate the gradient of the processed image and obtain the gradient amplitude, wherein the gradient calculation is shown in formula (1): Among them, I(x, y) represents the grayscale value of the pixel point (x, y) in the image, G(x, y) represents the gradient direction of the pixel point (x, y), and g x (x, y) represents the horizontal gradient information at the pixel point (x, y), g y (x, y) represents the vertical gradient information at the pixel point (x, y), and g(x, y) represents the gradient amplitude.

3. The method for visual-inertial odometry based on point-line feature fusion using the Fast-EDLines algorithm according to claim 2, characterized in that: Step B: Filter the pixels with significant gradient changes according to the gradient amplitude and obtain edge points, then draw edges based on the edge points and obtain edge pixel chains. The specific steps are as follows: Step B1: Filter pixels with significant gradient changes according to their gradient amplitude and obtain edge points. If the gradient amplitude of a pixel is greater than the gradient amplitude of its neighboring pixels and is also greater than a preset gradient threshold ρ, the pixel is identified as an edge point. The calculation process of the gradient threshold ρ is shown in formula (2). Among them, τ represents the angle tolerance and δ represents the boundary error; Step B2: Draw the edge based on the edge points and obtain an edge pixel chain. Specifically, after calculating the gradient threshold ρ, traverse the gradient map and remove pixels with gradient values ​​less than the gradient threshold ρ, thereby obtaining the edge area of ​​the image. Then, the local gradient peak in the edge area is used as the anchor point and starting from the anchor point, pixels are gradually connected according to the gradient direction and amplitude to form an edge pixel chain.

4. The method for visual-inertial odometry based on point-line feature fusion using the Fast-EDLines algorithm according to claim 3, characterized in that: Step C: extract the edge pixel chain using the linear least squares fitting method to obtain a first set of straight line segments, and then remove the short line segments in the first set of straight line segments according to the Helmholtz principle to obtain a second set of straight line segments. The specific steps are as follows: Step C1: extract and obtain the first straight line segment set by using the linear least squares fitting method for the edge pixel chain. The specific steps are as follows: Step C11: Assume that the edge pixel sequence obtained by edge detection is (x1, y1), (x2, y2) ... (x i ,y i ) and the current subsequence length is n, then the straight line fitting equation of the fitting line is shown in formula (3), y=a·x+b (3) Where a and b both represent coefficients; Step C12, use least squares to solve coefficients a and b, set and Represent the mean of n pixels in the x and y directions respectively, and the solution process is as shown in formula (4). Step C13: Calculate the vertical distance d from each pixel to the straight line segment i , as shown in formula (5), In step C14, an error threshold δ is set. If the mean square error E of all pixels is not greater than the error threshold δ, the current subsequence can be fitted into a valid straight line segment. If the mean square error E of all pixels is greater than the error threshold δ, the current straight line segment is truncated and the above fitting process is repeated for the remaining pixels until the pixel chain is completely processed. The mean square error E is shown in formula (6). Step C2: According to the Helmholtz principle, the short segments in the first set of straight line segments are removed to obtain the second set of straight line segments. The minimum length of the straight line segment n is set to min , then according to the Helmholtz principle the minimum length n min The calculation process of is shown in formula (7). Where N represents the length of the image side and p represents the probability of the gradient direction being consistent.

5. The method for visual-inertial odometry based on point-line feature fusion using the Fast-EDLines algorithm according to claim 4, characterized in that: Step D: Use the false alarm number NFA to verify the significance of the straight line ends in the second straight line segment set and eliminate the pseudo line segments in the second straight line segment to obtain the third straight line segment set. The specific steps are as follows: Step D1 introduces the false alarm number NFA as a judgment criterion. Assuming that for a line segment A with a length of n, there are at least k pixels whose gradient direction is consistent with the line segment direction, the calculation process of the false alarm number NFA of line segment A is shown in formula (8): Among them, N 4 represents the number of potential straight line segments in the image, To represent the method of selecting i pixels from n pixels; Step D2: Set the NFA threshold ∈. If the number of false alarms NFA for segment A is not greater than the NFA threshold ∈, segment A is significant and has a high confidence level, and thus the segment is retained. If the number of false alarms NFA for segment A is less than the NFA threshold ∈, the pseudo segment is removed.

6. The method for visual-inertial odometry based on point-line feature fusion using the Fast-EDLines algorithm according to claim 5, characterized in that: Step E: performing line segment matching and fusion on the line segments in the third line segment set and eliminating isolated line segments to obtain a fourth line segment set. The specific steps are as follows: Step E1: Assume that the initial line segment set extracted after line segment detection and verification is L = {l1, l2, ..., l n } and each line segment l is represented by the starting point and the end point as l = (s, e), according to the length threshold length threshold Divide the line segment set L into the long line segment set L by Euclidean distance calculation long and the set of short line segments L short , as shown in formula (9), Step E2, calculate the initialized fused line segment set L′, where L′=L long , specifically for the short line segment set L short Each short line segment l in short =(s short , e short )Calculation and long segment set L long Each long line segment l in long =(s long , e long ) and find the long line segment L that meets the direction and position conditions. long , the specific steps are as follows, Step E21, find a long line segment L that meets the direction condition long , the specific steps are as follows, Step E211: calculate the angle angle (l short , l long ) is calculated as shown in formula (10), Among them, the short line direction vector V short =e short -s short , long segment direction vector V long =e long -s long ; Step E212, determine the angle angle (l short , l long ) is less than the angle threshold θ threshold =3°, as shown in formula (11), angle(l short ,l long )≤θ threshold +b (11) Where β represents the small tolerance value; if formula (11) holds, it means that the short segment and the long segment are close in direction; Step E22, find a long line segment that meets the position condition, specifically calculate the point p(x p ,y p ) to the shortest distance between line segment l = (s, e). The specific steps are as follows: Step E221, calculate the direction vector and vector modulus of the line segment, as shown in formula (12) and formula (13), V=e-s=(x e -x s ,y e -y s ) (12) ||V|| 2 =(x e -x s ) 2 +(and e -and s ) 2 (13) Among them, the coordinates of the starting point s are (x s ,y s ), the coordinates of the end point e are (x e ,y e ), V represents the displacement vector from the starting point s to the end point e; Step E222, calculate the projection of the point to the line, specifically calculate the projection point p of point p on the line proj The normalized position parameter t of is shown in formula (14), The numerator is the inner product of the projection of point p onto the straight line, and the inner product represents the projection length of point p on the straight line. The denominator represents the square of the modulus of the line segment direction vector, and the square of the modulus is used to normalize the projection scale. Step E223, if 0≤t≤1, then the projection point p proj (x proj ,y proj ) is located inside the line segment as shown in formula (15), and the shortest distance d(p, l) from the point to the line is shown in formula (16), p proj =s+t·V=(x s +t·(x e -x s ),y s +t·(y e -y s )) (15) In step E224, if t<0, the projection point is located on the extension line of the starting point of the line segment, and the shortest distance is the length from the point to the starting point of the line segment, as shown in formula (17). Step E225: If t>1, the projection point is located on the extension line of the end point of the line segment, and the shortest distance is the length from the point to the end point of the line segment, as shown in formula (18). Step E226, calculate the short line segment l short Starting point short and the end point e short Arrival at long segment l long The specific formula is as follows: dist(l short ,l long )=min(d(s short ,l long ),d(e short ,l long )) (19) Among them, dist(l short , l long ) is the shortest distance from the short segment to the long segment; Step E227, determine the shortest distance dist(l short , l long ) is less than the distance threshold dist threshold =5, as shown in formula (20). If formula (20) is satisfied, it means that the short segment and the long segment are close in position. If not, the short segment is regarded as an isolated segment and removed. dist(l short ,l long )≤dist threshold (20); Step E23, if there is a long line segment l long If both the direction condition and the position condition are met, the short line segment l short Merge to long line segment l long Then, set the maximum number of iterations for merging line segments and remove the line segments involved in the merging from the third straight line segment set. Update the endpoints of the fused long line segments as shown in formula (21). Among them, (S new , S end ) are the coordinates of the endpoints of the fused long line segment.

7. The method for visual-inertial odometry based on point-line feature fusion using the Fast-EDLines algorithm according to claim 6, characterized in that: Step F, using the Shi-Tomasi corner detection method to extract corner points in the processed image and use them as the first feature point set, then tracking, eliminating, screening and supplementing the first feature point set to obtain the second feature point set. The specific steps are as follows: Step F1: Use the Shi-Tomasi corner detection algorithm to extract corner points in the image and use the KLT sparse optical flow method to track the feature points. Then, use the image pyramid structure to enhance the scale invariance of the point features. If tracking fails or the feature points exceed the image boundary during the feature tracking process, invalid points are removed using the feature tracking status mark. Step F2: Use the basic matrix to eliminate outliers and improve the accuracy of feature matching. Then, filter the extracted corner points and sort the tracking points according to the number of tracking times to optimize the distribution of feature points. Step F3: Perform homogenization processing on the densely populated areas of the feature set, and then dynamically add new extracted corner points.

8. The method for visual-inertial odometry based on point-line feature fusion using the Fast-EDLines algorithm according to claim 7, characterized in that: Step G, using discrete time integration to perform recursive integration calculation on the inertial measurement values ​​between consecutive image frames and obtain IMU pre-integration, specifically by accumulating accelerometer and gyroscope data between adjacent key frames to calculate the relative motion parameters of the sensor at adjacent image acquisition moments, the relative motion parameters including displacement, velocity and attitude change.

9. The method for visual-inertial odometry based on point-line feature fusion using the Fast-EDLines algorithm according to claim 8, characterized in that: In step H, the fourth straight line segment set, the second feature point set, and the IMU pre-integral are input as observation quantities into the back-end optimization module to complete the visual inertial odometry operation. The specific implementation process of the back-end optimization module adopts the VINS-Mono algorithm.

10. A point-line feature fusion visual-inertial odometry system based on the Fast-EDLines algorithm, wherein the specific measurement process of the visual-inertial odometry system is based on the visual-inertial odometry method according to any one of claims 1 to 9, and is characterized by: It includes a gradient calculation module, an edge drawing module, a line segment extraction module, a line segment verification module, a line segment fusion module, a feature point extraction module, an IMU pre-integration acquisition module and an observation quantity optimization module. The gradient calculation module is used to perform Gaussian filtering and smoothing on the grayscale acquisition image to obtain a processed image, and then perform gradient calculation on the processed image to obtain the gradient amplitude; The edge drawing module is used to screen the pixels with significant gradient changes according to the gradient amplitude and obtain edge points, and then perform edge drawing according to the edge points to obtain edge pixel chains; The line segment extraction module is used to extract the edge pixel chain using a linear least squares fitting method to obtain a first set of straight line segments, and then remove short line segments in the first set of straight line segments according to the Helmholtz principle to obtain a second set of straight line segments; The line segment verification module is used to verify the significance of the straight line ends in the second straight line segment set using the false alarm number NFA and eliminate the pseudo line segments in the second straight line segments to obtain the third straight line segment set; The line segment fusion module is used to perform line segment matching and fusion on the straight line segments in the third straight line segment set and eliminate isolated line segments to obtain a fourth straight line segment set; The feature point extraction module is used to extract corner points in the processed image using the Shi-Tomasi corner point detection method and use them as a first feature point set, and then track, eliminate, filter and supplement the feature points of the first feature point set to obtain a second feature point set; The IMU pre-integration acquisition module is used to perform recursive integration calculation on the inertial measurement values ​​between consecutive image frames using discrete time integration and obtain IMU pre-integral; The observation quantity optimization module is used to input the fourth straight line segment set, the second feature point set and the IMU pre-integration as observation quantities into the back-end optimization module and complete the visual inertial odometry operation.

Images