A multi-point optical flow based image line segment fast tracking and fusion method

Abstract

This invention relates to the field of computer vision technology, specifically to a method for fast tracking and fusion of image line segments based on multi-point optical flow. The method includes: acquiring an image sequence arranged in chronological order; verifying whether the image sequence is continuously captured; preprocessing all images in the continuously captured image sequence and calculating gradients; processing the images in the preprocessed image sequence one by one in sequence order; for the currently processed image, extracting multiple line segments from the currently processed image; fusing and filtering the multiple line segments to obtain a set of line segments to be tracked in the currently processed image; for each line segment to be tracked in the set of line segments to be tracked, using a multi-point optical flow method for tracking, obtaining a set of tracked line segments in the currently processed image as the result of fast line segment tracking and fusion. This invention can improve the accuracy and efficiency of image line segment tracking.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision, and in particular to a method for fast tracking and fusing of image line segments based on multi-point optical flow. BACKGROUND

[0002] Simultaneous Localization and Mapping (SLAM) is a computer vision and robotics technology that aims to help robots or devices construct a map of the environment in real-time and determine their own position in the map simultaneously through sensors such as cameras, lidar, and IMU. One of the core tasks of SLAM is to achieve feature matching between images to ensure the accuracy and stability of localization and mapping.

[0003] Traditional SLAM systems mainly rely on point features (such as ORB, SIFT, etc.) to perform localization and mapping tasks. However, in some low-texture or sparse feature point scenes, point features are not sufficient to provide stable and reliable matching results. At this time, line features (such as the edges of objects, the outlines of building structures, and the junctions of walls and ceilings) can be a powerful complement to point features, providing more rich environmental information and thus improving the matching accuracy and robustness of the SLAM system.

[0004] In structured environments with more prominent line features (such as indoor scenes and urban streets), line features are often more stable than point features, such as the outlines of buildings, the junctions of room walls and ceilings, etc., which exhibit high consistency and robustness under different viewing angles. This characteristic enables line features to significantly improve the matching accuracy and overall positioning accuracy of the SLAM system in certain scenarios.

[0005] Currently, for image-to-image line segment matching, mainstream methods usually rely on descriptor matching algorithms (such as MSLD, LBD, etc.). These algorithms generate corresponding descriptors by calculating local features around the line segment for matching, but in environments containing repetitive textures, they can cause line segment mis-matching. In addition, the descriptor generation process has high computational complexity, resulting in slow matching speed. Thus, in scenarios with complex textures or high computational requirements, how to quickly and accurately track line segments between multiple images becomes a technical problem that needs to be solved. SUMMARY

[0006] In view of the above problems, the present application provides a method for fast tracking and fusing of image line segments based on multi-point optical flow, which solves the technical problem of low precision and efficiency of line segment matching in the prior art.

[0007] The present application provides a method for fast tracking and fusing of image line segments based on multi-point optical flow, comprising the following steps:

[0008] Step S1: Use a camera to capture an image sequence arranged in chronological order, verify whether the image sequence is a continuous capture, and obtain a continuously captured image sequence;

[0009] Step S2: Preprocess all images in the continuously captured image sequence and calculate the gradient to obtain the preprocessed image sequence and the gradient of each image;

[0010] Step S3: Process the images in the preprocessed image sequence one by one in sequence. For the currently processed image:

[0011] Extract multiple line segments from the currently processed image, and fuse and filter the multiple line segments according to the gradient of the currently processed image and the distance between the multiple line segments to obtain a set of line segments to be tracked in the current processed image;

[0012] For each line segment to be tracked in the set of line segments to be tracked: based on multiple points on the line segment to be tracked and the next image of the current image being processed, the tracking line segment corresponding to the line segment to be tracked in the next image is obtained by using a multi-point optical flow method; finally, the set of tracking line segments of the current image is obtained.

[0013] Step S4: Take the set of tracking line segments for each image in the preprocessed image sequence as the result of fast line segment tracking and fusion.

[0014] Preferably, in step S1, the specific steps for verifying whether the image sequence is continuously captured are as follows: for the images in the image sequence, calculate the chi-square distance of the gray value distribution histograms of two consecutive images. When the distance is less than a preset chi-square distance threshold, it is determined that the image parallax is small and meets the condition for continuous capture; the image sequence that meets the condition for continuous capture is taken as the continuously captured image sequence.

[0015] Preferably, step S2 specifically includes:

[0016] Step S2-1: Perform histogram equalization on all grayscale images in the continuously captured image sequence to obtain the preprocessed image sequence;

[0017] Step S2-2: Calculate the gradient of each image in the preprocessed image sequence, including: using the Sobel operator to calculate the horizontal gradient value and vertical gradient value of each pixel in the image, and calculating the gradient direction and magnitude of each pixel based on the horizontal gradient and vertical gradient.

[0018] Preferably, step S3 specifically includes:

[0019] Step S3-1: Set the first image in the preprocessed image sequence as the current image to be processed;

[0020] Step S3-2: Apply the EDLines algorithm to the current image being processed and filter the results of the EDLines algorithm to extract multiple line segments; set a merge mask, a mask mask, and multiple feature points for each of the multiple line segments, and fuse the multiple line segments based on the merge mask and the multiple feature points;

[0021] Step S3-3: Based on the mask, filter the fused line segments to obtain multiple valid line segments;

[0022] Step S3-4: If the number of the multiple valid line segments is less than the preset threshold for the number of valid line segments, then reduce the detection parameters of the EDLines algorithm and return to step S3-2 until the number of valid line segments is greater than the threshold for the number of valid line segments; the final set of valid line segments is used as the set of line segments to be tracked in the current image.

[0023] Steps S3-5: Construct image pyramids for the current image and the next image of the current image; determine the initial amount of top-level optical flow tracking based on the set of line segments to be tracked in the current image;

[0024] Step S3-6: Perform multi-point optical flow tracking of straight line segments layer by layer based on the initial optical flow tracking amount of the top layer, and obtain the tracking straight line segment corresponding to the line segment to be tracked in the current image in the next image; finally, obtain the set of tracking straight line segments of the preprocessed image.

[0025] Step S3-7: Set the next image of the currently processed image as the current processed image, return to step S3-2, and continue processing until all images in the preprocessed image sequence have been processed.

[0026] Preferably, in step S3-2, the step of filtering the results of the EDLines algorithm specifically includes: determining the minimum length len based on the image's width cols and height rows. min The expression is:

[0027] len min =min(rows,cols)×0.125

[0028] Where min(·) represents calculating the minimum value; for multiple line segments extracted by the EDLines algorithm, length filtering is performed to select those with a length greater than the minimum length len. min Multiple line segments;

[0029] In step S3-2, the steps of setting the merge mask, mask mask, and multiple feature points specifically include:

[0030] All ranges less than 5 pixels from the line segment are defined as the merge mask; all ranges less than 20 pixels from the line segment are defined as the mask.

[0031] The straight line segment is evenly divided into 7 regions. The regions near the two ends of the straight line segment are removed to obtain the middle 5 regions. The gradient of each point in the middle 5 regions is obtained based on the gradient of the current image being processed. The 3 points with the largest gradient values ​​are determined based on the gradient of each point in the middle 5 regions. The two endpoints of the straight line segment are then added to form a set containing 5 points, which are used as the multiple feature points of the straight line segment.

[0032] Preferably, in step S3-2, the step of fusing the multiple line segments based on the merge mask and the multiple feature points specifically includes:

[0033] If at least one of the five feature points of a line segment lies within the merge mask range of other line segments, then merging is required. The least squares method is used to determine the unified line after merging, and the following system of equations is solved:

[0034]

[0035] a′ 2 +b′ 2 =1

[0036] Where E is the error function of the least squares method, (u0,v0) and (u4,v4) are the coordinates of the two endpoints of the line segment, and a′, b′, c′ are the parameters of the equation of the unified line;

[0037] Solve the above system of equations to obtain the parameters a′, b′ and c′ of the unified line, and thus determine the unified line; project the endpoints of the line segments to be merged onto the unified line, select the two farthest endpoints as the endpoints of the merged line segments, and obtain the merged line segments.

[0038] Preferably, step S3-3 specifically includes:

[0039] For the multiple line segments after fusion, it is determined whether the feature points on the line segment are within the mask range of other line segments. If at least one feature point on the line segment is within the mask of other line segments, the line segment is removed, and finally multiple valid line segments are obtained.

[0040] Preferably, step S3-5 specifically includes:

[0041] For the current image I1 and the next image I2, set I1 and I2 as the bottom layer of the pyramid. The resolution of each layer from the bottom to the top is 100%, 50%, 25% and 12.5% ​​of the original image, respectively, to construct a four-layer image pyramid L3, L2, L1 and L0.

[0042] The set p of feature points on all valid line segments in image I1 i =(u i ,v i ), i∈{0,1,2,3,4} are scaled to the top layer L3 of the image pyramid as the initial quantity for optical flow tracking of the top layer, where u i ,v i Represents the x and y coordinates of the i-th point.

[0043] Preferably, steps S3-6 specifically include:

[0044] Starting from the top layer L3 of the image pyramid, during tracking, each layer first scales the tracking positions of the points on the line segments of the previous layer to the current layer, then performs multi-point optical flow tracking at the resolution of that layer, updates the tracking positions of the points on the line segments of that layer, and finally obtains multiple tracking line segments at the bottom layer, resulting in a set of tracking line segments of the preprocessed image.

[0045] The multi-point optical flow tracking processing steps specifically include:

[0046] (1) Determine the expression for multi-point optical flow tracing:

[0047]

[0048] Among them, I u I represents the grayscale derivative of the pixel in the u direction. v L represents the grayscale derivative of the pixel in the v direction. nm Let θ represent the distance between the m-th and n-th feature points on the line segment, θ represent the angle between the line segment and the horizontal direction of the image, a and b represent the displacement changes of the line segment in the u and v directions, and c represent the angular change of the angle between the line segment and the horizontal axis in images I1 and I2. t This represents the grayscale derivative of a pixel's grayscale value with respect to time t.

[0049] (2) The optimal value of (a,b,c) is calculated iteratively using the Gauss-Newton method. After obtaining the parameters (a,b,c), the position of each point on the current line segment to be tracked in the image I1 is calculated in the image I2 according to the following expression, so as to obtain the tracking line segment of the current line segment to be tracked.

[0050] u″ m =um +a+L nm csinθ

[0051] v″ m =v m +bL nm ccosθ

[0052] Where, u″ m ,v″ m u represents the position coordinates of the m-th point on the line segment in image I2. m ,v m This represents the position coordinates of the m-th point on the line segment in image I1.

[0053] Compared with the prior art, the present invention has at least the following beneficial effects:

[0054] (1) The fast image line segment tracking and fusion method based on multi-point optical flow provided in this invention has advantages such as low computational cost, fast operation speed, sound theoretical basis, and strong algorithm interpretability. Compared with commonly used descriptor methods, this method does not require additional descriptor calculation, which greatly reduces the computational cost. At the same time, it effectively avoids the problem of mismatch on repetitive texture line segments that is prone to occur in descriptor methods.

[0055] (2) This invention can further fuse the line segments directly extracted by EDLines to obtain more accurate and continuous long line segments. This improvement provides more reliable feature support in the subsequent line segment tracking of the SLAM system, which helps to improve the overall localization and mapping accuracy.

[0056] (3) The technical solution of this invention has strong interpretability. Unlike traditional optical flow methods, which rely only on the two endpoints of a straight line segment and are prone to drift and inaccurate tracking results, this invention ensures higher tracking accuracy through a multi-point optical flow strategy. The theoretical advantages and algorithmic transparency make this method more reliable and operable in practical applications. Attached Figure Description

[0057] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0058] Figure 1 The flowchart shows the fast tracking and fusion method for image straight line segments based on multi-point optical flow provided by the present invention.

[0059] Figure 2 This is a schematic diagram of the set of points selected based on gradient for a straight line segment, as provided by the present invention.

[0060] Figure 3 A flowchart of the straight segment fusion method provided by the present invention.

[0061] Figure 4 This is a schematic diagram of the image pyramid construction provided by the present invention.

[0062] Figure 5 This is a schematic diagram of straight line segment tracking provided by the present invention.

[0063] Figure 6 This is a schematic diagram of the straight line segment extraction and fusion tracking results provided by the present invention.

[0064] Figure 7 This diagram illustrates a comparison of the line segment tracking performance provided by this invention with other methods. Detailed Implementation

[0065] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0066] This invention discloses a fast tracking and fusion method for line segments in images based on multi-point optical flow, which aims to achieve efficient extraction, fusion and tracking of line segment features between continuously captured images, so as to quickly and accurately complete line segment recognition and tracking.

[0067] The method provided by this invention requires a series of continuously captured images as input and calculates a histogram of grayscale value distribution based on the grayscale images. The chi-square distance between consecutive images must be less than a set threshold to meet the requirements of continuous shooting. After histogram equalization preprocessing of the images, straight line segments are extracted from the current image, and the original straight line segments are fused and filtered. To obtain more accurate straight line tracking results, the method provided by this invention constructs an image pyramid, and reduces drift error by tracking layer by layer and using multi-point optical flow tracking. Compared with traditional descriptor methods or two-point optical flow methods, the method provided by this invention has lower computational cost, higher tracking accuracy, and smaller drift error.

[0068] To illustrate the effectiveness of the method proposed in this invention, the following detailed description of the above technical solution is provided through a specific embodiment, such as... Figure 1 As shown, a fast image line segment tracking and fusion method based on multi-point optical flow is disclosed. The specific implementation steps are as follows:

[0069] Step S1: Obtain an image sequence arranged in chronological order, verify whether the image sequence is continuously captured, and obtain a continuously captured image sequence.

[0070] In this step, a sequence of images taken in a laboratory environment and arranged in chronological order is used. First, the image data of different formats are uniformly represented as grayscale images.

[0071] A histogram of grayscale value distribution is calculated based on the grayscale image. The chi-square distance between the histograms of two consecutive images is calculated. When the distance is less than a preset chi-square distance threshold, the image disparity is considered small, satisfying the condition for continuous shooting. The image sequence that meets the condition for continuous shooting is defined as the continuously shot image sequence.

[0072] The above steps ensure that the input images are obtained from continuous shooting, providing a reliable data foundation for subsequent processing. They also calculate the gradient values ​​and directions of all images, preparing for line segment extraction.

[0073] Step S2: Preprocess all images in the continuously captured image sequence and calculate the gradient to obtain the preprocessed image sequence and the gradient of each image.

[0074] In this step, histogram equalization is performed on all grayscale images in the continuously captured image sequence to reduce the impact of lighting on the images, enhance image contrast, and highlight image details, ultimately obtaining the preprocessed image sequence.

[0075] Based on the image after histogram equalization, the Sobel operator is used to calculate the gradient of the image. The specific steps include: using the Sobel operator to calculate the horizontal gradient value and vertical gradient value of each pixel in the histogram equalized image, and calculating the gradient direction and magnitude of each pixel based on the horizontal gradient and vertical gradient.

[0076] Step S3: Process the images in the preprocessed image sequence one by one in sequence. For the currently processed image:

[0077] Extract multiple line segments from the currently processed image, and fuse and filter the multiple line segments according to the gradient of the currently processed image and the distance between the multiple line segments to obtain a set of line segments to be tracked in the current processed image;

[0078] For each line segment to be tracked in the set of line segments to be tracked: based on multiple points on the line segment to be tracked and the next image of the current image being processed, the tracking line segment corresponding to the line segment to be tracked in the next image is obtained by using a multi-point optical flow method; finally, the set of tracking line segments of the current image is obtained.

[0079] In this step, each image in the preprocessed image sequence is traversed in chronological order, and each image is processed.

[0080] First, the tracking line segment features of each image are obtained based on information from the current image and the next image. The specific steps are as follows.

[0081] (1) Apply the EDLines algorithm to the currently processed image to extract multiple line segments. The steps for extracting line segments from an image using EDLines are as follows:

[0082] Image filtering and smoothing. First, Gaussian filtering is applied to the image to suppress noise and achieve smoothing. In the smoothed image, the gradient strength and direction of each pixel are calculated, typically using operators like Sobel. Based on the calculated gradient information, a set of pixels called "anchor points" is identified. Anchor points are those pixels where the gradient operator produces a maximum value; these points are usually candidate pixels for edge regions. Anchor points correspond to pixels where the gradient operator produces a maximum value, and these pixels are also highly likely to be edge regions. The edges of the image are depicted by connecting these anchor points. Specifically, starting from an anchor point, the next anchor point is searched along the direction of the maximum gradient value using the gradient strength and direction of its neighboring pixels, thus completing the connection.

[0083] Line segment extraction and fitting. The algorithm iterates through the pixel grid, applying a least-squares line fitting method to fit each line segment until the fitting error exceeds a specific threshold (e.g., one pixel). After outputting the extracted line segments, the algorithm continues iteratively processing the remaining pixels in the pixel chain until all pixels have been fitted.

[0084] False line segment detection. Based on the Helmholtz criterion, multiple line segments are obtained. Short line segments are typically filtered out as they may be false positives; longer line segments, due to their significant deviation from the background, are usually retained to ensure the reliability and accuracy of the extraction results.

[0085] This invention performs length filtering on multiple line segments extracted by the EDLines algorithm, selecting multiple line segments with a length greater than a minimum length. The specific steps are as follows: assuming the image width is *cols* and the height is *rows*, the minimum length *len* is determined based on the image width and height. min To ensure that the extracted line segments have a certain degree of significance and reliability, the expression is:

[0086] len min =min(rows,cols)×0.125

[0087] Where min(·) represents calculating the minimum value.

[0088] (2) After length filtering is completed, the filtered line segments are merged. A merge mask and a mask are set for each line segment. In the image, all ranges less than 5 pixels away from the line segment are determined as the merge mask, which is used to filter neighboring line segments; all ranges less than 20 pixels away from the line segment are determined as the mask mask, which is used to filter line segments that are evenly distributed in the current image.

[0089] For each of the filtered line segments, based on the gradient of the current processed image calculated in step S2, feature points with significant gradient changes are obtained by dividing the line segment into regions. For example... Figure 2 As shown, the straight line segment is first evenly divided into 7 regions, and the regions near the two ends of the line segment are discarded, retaining the middle 5 regions to ensure that the selected feature points are more representative and stable. Within the retained middle 5 regions, the 3 points with the largest gradient values ​​are selected at fixed intervals, and then the two endpoints of the straight line segment are added to form a set {(u... i ,v i )}, i∈{0,1,2,3,4}. Where, u i ,v i Let represent the x and y coordinates of the i-th point, where i = 0 and i = 4 correspond to the starting and ending points of the line, respectively. The starting point represents the point closest to the lower left corner of the image, and the ending point represents the point closest to the upper right corner of the image.

[0090] The positional relationship between the five feature points of a line segment and the merge mask of other line segments determines whether a line segment needs to be merged. If at least one of the five feature points is located within the merge mask range of another line segment (i.e., within 5 pixels of the line segment), it indicates that this line segment is very close to the other line segments. In this case, the current line segment needs to be merged with the line segment to which the merge mask belongs. During the merging process, the least squares method is used to determine the unified line after merging. Assume that the function expressions of the two extracted line segments are line segment 1: a1u + b1v + c1 = 0, and line segment 2: a2u + b2v + c2 = 0, where a1, b1, and c1 are the parameters of line segment 1, a2, b2, and c2 are the parameters of line segment 2, and x and y are the horizontal and vertical coordinates of the pixels in the image.

[0091] The error function E of the least squares method is defined as the square of the perpendicular distance from a line segment to the uniform line a′u+b′v+c′=0, and its expression is:

[0092]

[0093] Where (u0,v0) and (u4,v4) are the coordinates of the two endpoints of the line segment, |·| represents the calculation of the absolute value, a′,b′,c′ are the parameters of the unified line, and u,v are the horizontal and vertical coordinates of the pixels in the image.

[0094] To find the parameters a′, b′, and c′ of the unified line, it is necessary to minimize the error function E. This invention achieves this by taking the partial derivatives of E with respect to a′, b′, and c′ and setting the partial derivatives to 0. The expression is as follows:

[0095]

[0096] Add constraint a′ 2 +b′ 2 =1 to ensure parameter uniqueness, solve the above system of equations to obtain the parameters a′, b′ and c′ of the unified line and thus determine the unified line. At this time, project the endpoints of line segment 1 and line segment 2 onto a′x+b′y+c′=0, select the two farthest endpoints as the endpoints of the merged line segment, and determine the mask and merge mask of the merged line segment.

[0097] (3) After merging, filter the multiple line segments. Determine the point p on the line segment. i =(u i ,v i Check if i∈{0,1,2,3,4} is within the mask range of other line segments. If at least one point is within the mask, it indicates that the distribution of the line segment is uneven, and the line segment is discarded. The above method is used to filter multiple line segments in the current image.

[0098] (4) After confirming that all points on the remaining line segments are not within the range of the merge mask and mask mask, the number of currently valid line segments is detected. If the number of valid line segments is less than the preset threshold for the number of valid line segments, the detection parameters of the EDLines algorithm are reduced, and step (1) is returned until the number of valid line segments is greater than the threshold for the number of valid line segments. The final set of valid line segments is used as the set of line segments to be tracked in the current image. Figure 3 The above steps demonstrate the process of line segment selection and fusion.

[0099] After obtaining the set of tracking line segments for each image, this invention uses a multi-point optical flow method based on multiple points on the line segment to be tracked and the next image of the current image to obtain the corresponding tracking line segment in the next image; finally, the set of tracking line segments in the next image is obtained. The processing procedure is as follows:

[0100] (1) As Figure 4As shown, an image pyramid is constructed for the currently processed image I1 and the next image I2. Images I1 and I2 are set as the bottom layer of the pyramid, maintaining a resolution of 1.0 without any scaling. Subsequently, images of different resolutions are generated by downsampling layer by layer to form the image pyramid. Specifically, the image pyramids constructed according to ratios of 1.0, 0.5, 0.25, and 0.125 mean that the resolution of each layer is 100%, 50%, 25%, and 12.5% ​​of the original image, respectively. In this structure, the top layer has a resolution of 0.125, the lowest in size and resolution, displaying the least detail, thus providing a good foundation for subsequent optical flow tracing at multiple resolution levels.

[0101] The downsampling process employs a Gaussian filter to smooth the image, reducing aliasing effects introduced during downsampling. After smoothing, the image is downsampled. In the construction of the image pyramid, layer 0.5 is obtained by sampling every other pixel from layer 1.0, while layer 0.25 is obtained by sampling every other pixel from layer 0.5. The first pyramid layer processed is the top layer L3 (four layers in total: 0, 1, 2, 3), starting with the lowest resolution 0.125 pyramid layer. The set p of feature points on all valid line segments in image I1 is then processed. i =(u i ,v i ), i∈{0,1,2,3,4} scaled to the top layer L3 of the image pyramid.

[0102] (2) Layer-by-layer multi-point optical flow tracking of straight line segments. First, tracking begins at the top layer L3 of the image pyramid. Optical flow calculations are performed using the resolution of this layer to obtain the tracking positions of points on the straight line segments. By proceeding layer by layer downwards (i.e., from L3 to L0), image details are gradually restored to ensure the accuracy and robustness of optical flow tracking. During tracking at each layer, the tracking positions of points on the straight line segments in the previous layer are first scaled to the current layer. Then, optical flow calculations are performed at the resolution of this layer to update the tracking positions of the points on the straight line segments. Finally, accurate straight line segment matching and tracking are achieved throughout the entire image pyramid.

[0103] The principle of single-point optical flow tracking is as follows: For a single point, optical flow tracks by minimizing photometric error. Assume that at time t, a point is at position (u,v) with a grayscale value of I(u,v,t). After time dt, at time t+dt, the point moves to a new position (u+du,v+dv). Based on this, and according to the assumption of grayscale invariance (i.e., the image grayscale value remains unchanged within a small range of motion), the following expression can be obtained:

[0104] I(u+du,v+dv,t+dt)=I(u,v,t)

[0105] Performing a Taylor expansion of the equation and ignoring higher-order terms yields the following expression:

[0106]

[0107] Among them, I u I represents the grayscale derivative of the pixel in the u direction. v I represents the grayscale derivative of the pixel in the v direction; t This represents the grayscale derivative of a pixel's grayscale value with respect to time t. This represents the pixel's movement speed in the u direction. This represents the pixel's movement speed in the v direction.

[0108] like Figure 5 As shown, for multi-point optical flow tracking of a straight line segment, assuming that there are multiple points p on the straight line segment in image I1. i =(u i ,v i ), i∈{0,1,2,3,4}, the positions of these points in image I2 are {p″ i =(u″ i ,v″ i )}, i∈{0,1,2,3,4}. In this case, the positional change of the line segment in image I2 relative to the line segment in I1 is solved. The positional change can be described by parameters (a,b,c), which define the transformation of the line segment's position in image i2 relative to i1. a and b represent the displacement changes of the line segment in the u and v directions, c represents the angular change of the angle between the line segment and the horizontal axis in images I1 and I2, and angle θ represents the angle between the line segment and the horizontal direction of the image. L nm Point p on the line segment m With point p n The distance between them. The expression for multi-point optical flow tracking of a straight line segment is:

[0109]

[0110] In multi-point optical flow tracing on a straight line segment, assuming each point on the segment satisfies the optical flow tracing equation, there are a total of 5 points, and 3 unknowns need to be solved, namely (a, b, c). Therefore, the Gauss-Newton method is used to iteratively solve for the optimal value of (a, b, c). The Gauss-Newton method updates (a, b, c) by minimizing the cost function, gradually approaching the optimal solution. When the cost function fails to converge and the maximum number of iterations is reached, the current straight line segment is considered untrackable and is therefore discarded.

[0111] (3) After obtaining the parameters (a,b,c), calculate the position of each point on the current line segment to be tracked in the image I2 according to the following expression, and obtain the tracking line segment of the current line segment to be tracked.

[0112] u″ m =u m +a+L nm csinθ

[0113] v″ m =v m +bL nm ccosθ

[0114] Where, u″ m ,v″ m u represents the position coordinates of the m-th point on the line segment in image I2. m ,v m This represents the position coordinates of the m-th point on the line segment in image I1.

[0115] Repeat the line segment extraction and tracking process until all images in the preprocessed image sequence have been processed.

[0116] Step S4: Take the set of tracking line segments for each image in the preprocessed image sequence as the result of fast line segment tracking and fusion.

[0117] After all image processing is complete, the set of tracking line segments for each image is taken as the result of fast line segment tracking and fusion.

[0118] In some embodiments, each line segment in the tracking line segment set is marked in the captured image to display the line segment extraction results, such as... Figure 6 As shown, Figure 6 The left side shows the result of extracting line segments, while... Figure 6 The right side shows the result of merging line segments, where adjacent line segments in the horizontal frame are merged and filtered, and the lines are made more uniform.

[0119] Figure 7 Table 1 compares the method of this invention with conventional line segment extraction and tracking methods on multiple datasets. MY represents the method of this invention, Edline+LK represents line extraction using EDLine and optical flow tracking using LK, and LSD+LBD represents line segment extraction using LSD and line feature descriptors using LBD. Table 1 shows the time consumed; this invention reduces the time consumed by up to 80% compared to LSD+LBD. Although it takes about 10ms longer than Edline+LK, Figure 7 In the extraction results, MY showed better results in line segment extraction and tracking. In the horizontal box, MY successfully fused the repeatedly extracted lines at the edges, and in the vertical box, MY removed and fused the densely distributed line segments.

[0120] Table 1

[0121]

[0122] The results above demonstrate that the line segment fusion and tracking method based on multi-point optical flow provided in this invention can effectively extract, fuse, and track line segments in a series of continuously captured images, exhibiting strong adaptability. It can handle different lighting conditions and repetitive texture environments, ensuring reliability in complex scenes. This method has low computational cost and fast processing speed, making the processing more efficient and suitable for real-time applications. Furthermore, the algorithm has a solid theoretical foundation, is easy to understand and apply, and overcomes the shortcomings of existing descriptor methods and traditional two-endpoint optical flow tracking methods, avoiding potential mismatches and drift phenomena during processing. Therefore, the line segment fusion and tracking method based on multi-point optical flow of this invention can quickly and accurately extract, fuse, and track line segments in continuously captured images, providing a solid foundation for subsequent visual processing and applications.

[0123] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A fast image line segment tracking and fusion method based on multi-point optical flow, characterized in that, Includes the following steps: Step S1: Use a camera to capture an image sequence arranged in chronological order, verify whether the image sequence is a continuous capture, and obtain a continuously captured image sequence; Step S2: Preprocess all images in the continuously captured image sequence and calculate the gradient to obtain the preprocessed image sequence and the gradient of each image; Step S3: Process the images in the preprocessed image sequence one by one in sequence. For the currently processed image: Extract multiple line segments from the currently processed image, and fuse and filter the multiple line segments according to the gradient of the currently processed image and the distance between the multiple line segments to obtain a set of line segments to be tracked in the current processed image; For each line segment to be tracked in the set of line segments to be tracked: based on multiple points on the line segment to be tracked and the next image of the current image, the tracking line segment corresponding to the line segment to be tracked in the next image is obtained by using a multi-point optical flow method; The final set of tracking line segments in the currently processed image is obtained; Step S4: Take the set of tracking line segments for each image in the preprocessed image sequence as the result of fast line segment tracking and fusion.

2. The method for fast tracking and fusion of image straight line segments based on multi-point optical flow according to claim 1, characterized in that, In step S1, the specific steps for verifying whether the image sequence is continuously captured are as follows: For the images in the image sequence, calculate the chi-square distance of the gray value distribution histograms of two consecutive images. When the distance is less than a preset chi-square distance threshold, determine that the image parallax meets the condition for continuous capture; and take the image sequence that meets the condition for continuous capture as the continuously captured image sequence.

3. The method for fast tracking and fusion of image straight line segments based on multi-point optical flow according to claim 2, characterized in that, Step S2 specifically includes: Step S2-1: Perform histogram equalization on all grayscale images in the continuously captured image sequence to obtain the preprocessed image sequence; Step S2-2: Calculate the gradient of each image in the preprocessed image sequence, including: using the Sobel operator to calculate the horizontal gradient value and vertical gradient value of each pixel in the image, and calculating the gradient direction and magnitude of each pixel based on the horizontal gradient and vertical gradient.

4. The method for fast tracking and fusion of image straight line segments based on multi-point optical flow according to claim 3, characterized in that, Step S3 specifically includes: Step S3-1: Set the first image in the preprocessed image sequence as the current image to be processed; Step S3-2: Apply the EDLines algorithm to the currently processed image and filter the results of the EDLines algorithm to extract multiple line segments; set the parameters for each of the multiple line segments. mask Mask and multiple feature points, based on the The mask and the multiple feature points are fused together to form the multiple line segments; Step S3-3, based on the above The mask filters the merged line segments to obtain multiple valid line segments. Step S3-4: If the number of the multiple valid line segments is less than the preset threshold for the number of valid line segments, then reduce the detection parameters of the EDLines algorithm and return to step S3-2 until the number of valid line segments is greater than the threshold for the number of valid line segments; the final set of valid line segments is used as the set of line segments to be tracked in the current image. Steps S3-5: Construct image pyramids for the current image and the next image of the current image; determine the initial amount of top-level optical flow tracking based on the set of line segments to be tracked in the current image; Step S3-6: Perform multi-point optical flow tracking of straight line segments layer by layer based on the initial optical flow tracking amount of the top layer, and obtain the tracking straight line segment corresponding to the line segment to be tracked in the current image in the next image; finally, obtain the set of tracking straight line segments of the current image. Step S3-7: Set the next image of the currently processed image as the current processed image, return to step S3-2, and continue processing until all images in the preprocessed image sequence have been processed.

5. The method for fast tracking and fusion of image straight line segments based on multi-point optical flow according to claim 4, characterized in that, In step S3-2, the step of filtering the results of the EDLines algorithm specifically includes: based on the image width and height Determine the minimum length The expression is: in, This indicates the calculation of the minimum value; for multiple line segments extracted by the EDLines algorithm, length filtering is performed to select those with a length greater than the minimum length. Multiple line segments; In step S3-2, the setting mask The specific steps for masking and multiple feature points include: Define all ranges whose distance from the line segment is less than 5 pixels as the specified range. Mask; defines all ranges less than 20 pixels from the line segment as... Mask; The straight line segment is evenly divided into 7 regions. The regions near the two ends of the straight line segment are removed to obtain the middle 5 regions. The gradient of each point in the middle 5 regions is obtained based on the gradient of the current image being processed. The 3 points with the largest gradient values ​​are determined based on the gradient of each point in the middle 5 regions. The two endpoints of the straight line segment are then added to form a set containing 5 points, which are used as the multiple feature points of the straight line segment.

6. The method for fast tracking and fusion of image straight line segments based on multi-point optical flow according to claim 5, characterized in that, In step S3-2, the step based on the The steps of fusing the mask and the multiple feature points with the multiple line segments specifically include: If at least one of the five feature points of a line segment lies on another line segment... If the area is within the mask range, it is determined that fusion is required. The least squares method is used to determine the unified straight line after fusion, and the following system of equations is solved: in, Let be the error function of the least squares method. and Let be the coordinates of the two endpoints of the line segment. The parameters for the equation of a uniform straight line; Solving the above system of equations yields the parameters of the uniform straight line. and This determines a unified straight line; the endpoints of the line segments to be merged are projected onto the unified straight line, and the two farthest endpoints are selected as the endpoints of the merged line segments to obtain the merged line segments.

7. The method for fast tracking and fusion of image straight line segments based on multi-point optical flow according to claim 6, characterized in that, Step S3-3 specifically includes: For the multiple line segments after merging, determine whether the feature points on the line segments are located on other line segments. Within the masked area, if at least one feature point on a line segment lies within another line segment... Within the mask, the line segment is removed, ultimately resulting in multiple valid line segments.

8. The method for fast tracking and fusion of image straight line segments based on multi-point optical flow according to claim 7, characterized in that, Step S3-5 specifically includes: For the currently processed image and the next image of the currently processed image. ,Will and The image is set as the bottom layer of the pyramid, and the resolution of each layer from the bottom to the top is 100%, 50%, 25%, and 12.5% ​​of the original image, respectively, constructing a four-layer image pyramid. ; Image The set of feature points on all valid line segments in the equation Scale to the top of the image pyramid , as the initial quantity for the top-level optical flow tracking, wherein, Indicates the first The x and y coordinates of each point.

9. The method for fast tracking and fusion of line segments in images based on multi-point optical flow according to claim 8, characterized in that, Steps S3-6 specifically include: From the top of the image pyramid Initially, during tracking, each layer first scales the tracking positions of the points on the line segments of the previous layer to the current layer, then performs multi-point optical flow tracking at the resolution of that layer, updates the tracking positions of the points on the line segments of that layer, and finally obtains multiple tracking line segments of the bottom layer, resulting in a set of tracking line segments of the preprocessed image. The multi-point optical flow tracking processing steps specifically include: (1) Determine the expression for multi-point optical flow tracing: in, Indicates the pixel point at gray-scale derivative in direction, Indicates the pixel point at gray-scale derivative in direction, Indicates the first line segment on the line segment. The feature point and the first The distance between feature points , and Indicates that the line segment is in and Displacement change in direction Indicates the line segment in the image and images The change in the angle between the center and the horizontal axis. Indicates the grayscale value of a pixel as a function of time. The gray-scale derivative; (2) Iterative calculation using the Gauss-Newton method. The optimal value is obtained after obtaining the parameters. Then, the image is calculated according to the following expression. In the image, each point on the line segment to be tracked is... The position in the middle is used to obtain the current line segment to be tracked; in, Representing an image On the middle straight segment The position coordinates of each point Representing an image On the middle straight segment The coordinates of each point.

Images