Image straight line segment rapid tracking and fusion method based on multi-point optical flow

Abstract

The invention relates to the technical field of computer vision, in particular to an image straight-line segment rapid tracking and fusion method based on multi-point optical flow, which comprises the following steps: acquiring an image sequence arranged according to a time sequence, and verifying whether the image sequence is continuously shot or not; preprocessing all the images in the continuously shot image sequence, calculating the gradient, processing the images in the preprocessed image sequence one by one according to the sequence, extracting a plurality of straight line segments from the currently processed image, fusing and screening the plurality of straight line segments, and obtaining a to-be-tracked straight line segment set of the currently processed image; and tracking each to-be-tracked straight line segment in the to-be-tracked straight line segment set by adopting a multi-point optical flow method to obtain a tracking straight line segment set of the current processing image, and taking the tracking straight line segment set as a straight line segment rapid tracking and fusion result, so that the precision and efficiency of image straight line segment tracking can be improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and particularly relates to a method for fast tracking and fusion of image straight line segments based on multi-point optical flow. Background Art

[0002] Simultaneous Localization and Mapping (SLAM) is a computer vision and robotics technology that aims to help robots or devices build an environmental map in real time in an unknown environment through sensors (such as cameras, lidar, and IMU), and at the same time determine their own positions in the map. One of the core tasks of SLAM is to achieve feature matching between images to ensure the accuracy and stability of positioning and map building.

[0003] Traditional SLAM systems mainly rely on point features (such as ORB, SIFT, etc.) to perform positioning and mapping tasks. However, in some scenes with low texture or sparse feature points, point features are not sufficient to provide stable and reliable matching results. At this time, line features (such as the edges of objects, the contour lines of building structures, the intersection lines of walls and ceilings) can be a powerful supplement to point features, providing richer environmental information, and thus improving the matching accuracy and robustness of the SLAM system.

[0004] In structured environments with prominent line features (such as indoor scenes, urban blocks), line features are often more stable than point features. For example, the contour lines of buildings, the intersection lines of room walls and ceilings, etc., show high consistency and robustness from different perspectives. This characteristic enables line features to significantly improve the matching accuracy and overall positioning accuracy of the SLAM system in specific scenarios.

[0005] Currently, for the matching of straight line segments between images, mainstream methods usually rely on descriptor matching algorithms (such as MSLD, LBD, etc.). These algorithms generate corresponding descriptors by calculating the local features around the straight line segments for matching, but in an environment containing repeated textures, it may lead to incorrect matching of straight line segments. In addition, the calculation complexity of the descriptor generation process is relatively high, resulting in a slow matching speed. Therefore, in scenarios with complex textures or high computational requirements, how to quickly and accurately track straight line segments between multiple images has become a technical problem to be solved urgently. Summary of the Invention

[0006] In view of the above problems, the present invention provides a method for fast tracking and fusion of image straight line segments based on multi-point optical flow, which solves the technical problems of low accuracy and efficiency of straight line segment matching in the prior art.

[0007] The present invention provides a method for fast tracking and fusion of image straight line segments based on multi-point optical flow, including the following steps:

[0008] Step S1: Use a camera to capture an image sequence arranged in chronological order, verify whether the image sequence is continuously captured, and obtain the continuously captured image sequence;

[0009] Step S2: Preprocess all the images in the continuously captured image sequence and calculate the gradients to obtain a preprocessed image sequence and the gradients of each image;

[0010] Step S3: Process the images in the preprocessed image sequence one by one in sequence order. For the currently processed image:

[0011] Extract multiple straight line segments from the currently processed image, fuse and filter the multiple straight line segments according to the gradient of the currently processed image and the distances between the multiple straight line segments, and obtain the set of straight line segments to be tracked of the currently processed image;

[0012] For each straight line segment to be tracked in the set of straight line segments to be tracked: Based on multiple points on the straight line segment to be tracked and the next image of the currently processed image, use the multi-point optical flow method to obtain the corresponding tracked straight line segment of the straight line segment to be tracked in the next image; Finally, obtain the set of tracked straight line segments of the currently processed image;

[0013] Step S4: Take the set of tracked straight line segments of each image in the preprocessed image sequence as the result of straight line segment fast tracking and fusion.

[0014] Preferably, in step S1, the specific steps for verifying whether the image sequence is continuously captured are: For the images in the image sequence, calculate the chi-square distance of the grayscale value distribution histograms of two consecutive images before and after. When the distance is less than a preset chi-square distance threshold, it is determined that the image parallax is small and meets the condition of continuous shooting; Take the image sequence that meets the condition of continuous shooting as the continuously captured image sequence.

[0015] Preferably, step S2 specifically includes:

[0016] Step S2-1: Perform histogram equalization processing on all the grayscale images in the continuously captured image sequence to obtain the preprocessed image sequence;

[0017] Step S2-2: Calculate the gradients of each image in the preprocessed image sequence, including: Use the sobel operator to calculate the horizontal gradient value and vertical gradient value of each pixel in the image, and calculate the gradient direction and amplitude of each pixel based on the horizontal gradient and vertical gradient.

[0018] Preferably, step S3 specifically includes:

[0019] Step S3-1: Set the first image in the preprocessed image sequence as the current processed image;

[0020] Step S3-2: Apply the EDLines algorithm to the current processed image and filter the results of the EDLines algorithm to extract multiple line segments; Set a merge mask, a mask mask, and multiple feature points for each line segment among the multiple line segments, and fuse the multiple line segments based on the merge mask and the multiple feature points;

[0021] Step S3-3: Screen and filter the fused multiple line segments based on the mask mask to obtain multiple valid line segments;

[0022] Step S3-4: If the number of the multiple valid line segments is less than the preset valid line segment number threshold, reduce the detection parameters of the EDLines algorithm and return to Step S3-2 until the number of the valid line segments is greater than the valid line segment number threshold; Use the set of finally formed valid line segments as the set of line segments to be tracked in the current processed image;

[0023] Step S3-5: Build image pyramids for the current processed image and the next image of the current processed image respectively; Determine the initial amount of top-layer optical flow tracking based on the set of line segments to be tracked in the current processed image;

[0024] Step S3-6: Perform multi-point optical flow tracking of line segments layer by layer from the initial amount of top-layer optical flow tracking to obtain the corresponding tracked line segments of the line segments to be tracked in the current processed image in the next image; Finally obtain the set of tracked line segments of the preprocessed image;

[0025] Step S3-7: Set the next image of the current processed image as the current processed image, and return to Step S3-2 until all images in the preprocessed image sequence are processed.

[0026] Preferably, in Step S3-2, the step of filtering the results of the EDLines algorithm specifically includes: determining the minimum length len according to the width cols and height rows of the picture min , and the expression is:

[0027] len min = min(rows, cols) × 0.125

[0028] where min(·) represents calculating the minimum value; perform length filtering on the multiple line segments extracted by the EDLines algorithm, and screen to obtain multiple line segments with lengths greater than the minimum length len min ;

[0029] In step S3-2, the steps of setting the merge mask, the mask mask, and multiple feature points specifically include:

[0030] Determine the range where the distance to the straight line segment is less than 5 pixels as the merge mask; determine the range where the distance to the straight line segment is less than 20 pixels as the mask mask;

[0031] Evenly divide the straight line segment into 7 regions, remove the regions near both ends of the straight line segment, obtain the middle 5 regions, obtain the gradients of each point in the middle 5 regions based on the gradient of the current processed image, determine 3 points with the largest gradient values based on the gradients of each point in the middle 5 regions, and then add the two endpoints of the straight line segment. Finally, form a set containing 5 points as the multiple feature points of the straight line segment.

[0032] Preferably, in step S3-2, the step of fusing the multiple straight line segments based on the merge mask and the multiple feature points specifically includes:

[0033] If at least one of the 5 feature points of a straight line segment is located within the merge mask range of other straight line segments, it is determined that fusion is required, and the least squares method is used to determine the unified straight line after fusion, and the following system of equations is solved:

[0034]

[0035] a′ 2 +b′ 2 =1

[0036] where E is the error function of the least squares method, (u 0 , v 0 ) and (u 4 , v 4 ) are the coordinates of the two endpoints of the straight line segment respectively, and a′, b′, c′ are the parameters of the equation of the unified straight line;

[0037] Solve the above system of equations to obtain the parameters a′, b′, and c′ of the unified straight line, thereby determining the unified straight line; project the endpoints of the straight line segments to be fused onto the unified straight line, select the two farthest endpoints as the endpoints of the fused straight line segment, and obtain the fused straight line segment.

[0038] Preferably, step S3-3 specifically includes:

[0039] For the fused multiple straight line segments, determine whether the feature points on the straight line segment are located within the mask mask range of other straight line segments. If at least one feature point on the straight line segment is located within the mask mask of other straight line segments, then remove this straight line segment, and finally obtain multiple effective straight line segments.

[0040] Preferably, the step S3-5 specifically includes:

[0041] For the current processed image I 1 and the next image I of the current processed image 2 , set I 1 and I 2 as the bottom layer of the pyramid, and construct a four-layer image pyramid L with the resolution of each layer of images from the bottom layer to the top layer being 100%, 50%, 25%, and 12.5% of the original image 3 , L 2 , L 1 , L 0 ;

[0042] Scale the set p 1 of feature points on all valid line segments in the image I i =(u i , v i ), i∈{0,1,2,3,4} to the top layer L 3 of the image pyramid as the initial amount of optical flow tracking at the top layer, where u i , v i represent the horizontal and vertical coordinates of the i-th point.

[0043] Preferably, the step S3-6 specifically includes:

[0044] Starting from the top layer L 3 of the image pyramid, during tracking for each layer, first scale the tracking positions of the points on the line segments of the previous layer to the current layer, then perform multi-point optical flow tracking at the resolution of this layer to update the tracking positions of the points on the line segments of this layer, and finally obtain multiple tracked line segments at the bottom layer to obtain the set of tracked line segments of the pre-processed image;

[0045] The processing steps of the multi-point optical flow tracking specifically include:

[0046] (1) Determine the expression of multi-point optical flow tracking:

[0047]

[0048] where I u represents the gray derivative of the pixel point in the u direction, I v represents the gray derivative of the pixel point in the v direction, L nm represents the distance between the m-th feature point and the n-th feature point on the line segment, θ represents the angle between the line segment and the horizontal direction of the image, a and b represent the displacement changes of the line segment in the u and v directions, c represents the angle change amount between the line segment and the horizontal axis in the images I 1 and I 2 and the horizontal axis, It Represents the gray derivative of the pixel gray value with respect to time t;

[0049] (2) Use the Gauss-Newton method to perform iterative solution to calculate the optimal values of (a, b, c). After obtaining the parameters (a, b, c), calculate the image I according to the following expression 1 The positions of the points on the current straight line segment to be tracked in the image I 2 to obtain the tracking straight line segment of the current straight line segment to be tracked;

[0050] u″ m = u m + a + L nm c sinθ

[0051] v″ m = v m + b - L nm c cosθ

[0052] where u″ m , v″ m represent the position coordinates of the m-th point on the straight line segment in the image I 2 , u m , v m represent the position coordinates of the m-th point on the straight line segment in the image I 1 .

[0053] Compared with the prior art, the present invention has at least the following beneficial effects:

[0054] (1) The method for fast tracking and fusion of image straight line segments based on multi-point optical flow provided by the present invention has the advantages of small computational complexity, fast operation speed, perfect theoretical basis, strong algorithm interpretability, etc. Compared with the commonly used descriptor method, this method does not require additional calculation of descriptors, greatly reducing the computational complexity. At the same time, it effectively avoids the problem of easy false matching of the descriptor method on straight line segments with repeated textures.

[0055] (2) The present invention can further fuse the straight line segments directly extracted by EDLines to obtain more accurate and continuous long straight line segments. This improvement provides more reliable feature support for the straight line segment tracking in the subsequent SLAM system, helping to improve the overall positioning and mapping accuracy.

[0056] (3) The technical solution of the present invention has strong interpretability. Different from the traditional optical flow method, the traditional method only relies on the two endpoints of the straight line segment, which easily leads to drift and inaccurate tracking results. However, the present invention ensures higher tracking accuracy through the strategy of multi-point optical flow. The theoretical advantages and the transparency of the algorithm make this method have higher reliability and operability in practical applications. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The accompanying drawings are only for the purpose of showing specific embodiments and are not considered to be a limitation of the present invention.

[0058] Figure 1 It is a flowchart of the method for fast tracking and fusion of image straight line segments based on multi-point optical flow provided by the present invention.

[0059] Figure 2 It is a schematic diagram of the set of points selected based on the gradient for the straight line segment provided by the present invention.

[0060] Figure 3 It is a flowchart of the straight line segment fusion method provided by the present invention.

[0061] Figure 4 It is a schematic diagram of the construction of the image pyramid provided by the present invention.

[0062] Figure 5 It is a schematic diagram of the straight line segment tracking provided by the present invention.

[0063] Figure 6 It is a schematic diagram of the result of straight line segment extraction and fusion tracking provided by the present invention.

[0064] Figure 7 It is a schematic diagram of the comparison of the straight line segment tracking effect with other methods provided by the present invention. Specific Embodiments

[0065] In order to more clearly understand the above objects, features and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, without conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein. Therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0066] The present invention discloses a method for fast tracking and fusion of image straight line segments based on multi-point optical flow, aiming to achieve efficient extraction, fusion and tracking of straight line segment features between continuously captured images, so as to quickly and accurately complete straight line segment recognition and tracking.

[0067] The method provided by the present invention requires a series of continuously captured images as input, and calculates the histogram of the gray value distribution based on the grayscale images. The chi-square distance between consecutive images needs to be less than a set threshold to meet the requirements of continuous shooting. After preprocessing the images by histogram equalization, straight line segments are extracted from the current image, and the original straight line segments are fused and screened. To obtain more accurate straight line segment tracking results, the method provided by the present invention constructs an image pyramid, performs layer-by-layer tracking and adopts multi-point optical flow tracking to reduce the drift error. Compared with the traditional descriptor method or the two-point based optical flow method, the method provided by the present invention has a small computational amount, high tracking accuracy and small drift error.

[0068] To illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention will be described in detail through a specific embodiment as follows, as Figure 1 shown, a fast tracking and fusion method for image straight line segments based on multi-point optical flow is disclosed, and the specific implementation steps are as follows:

[0069] Step S1: Obtain an image sequence arranged in chronological order, verify whether the image sequence is continuously captured, and obtain the continuously captured image sequence.

[0070] In this step, an image sequence arranged in chronological order captured in a laboratory environment is used. First, image data in different formats are uniformly represented in the form of grayscale images.

[0071] Calculate the histogram of the gray value distribution based on the grayscale image, calculate the chi-square distance between the histograms of two consecutive images before and after, and determine that the image parallax is small and meets the condition of continuous shooting when the distance is less than the preset chi-square distance threshold. The image sequence that meets the condition of continuous shooting is used as the continuously captured image sequence.

[0072] Through the above steps, it can be ensured that the input images are obtained from continuous shooting, and a reliable data basis is provided for subsequent processing. Calculate data such as the gradient value and direction of all images to prepare for straight line segment extraction.

[0073] Step S2: Preprocess all the images in the continuously captured image sequence and calculate the gradient to obtain the preprocessed image sequence and the gradient of each image.

[0074] In this step, histogram equalization processing is performed on all the grayscale images in the continuously captured image sequence to reduce the influence of light on the images, enhance the contrast of the images and highlight the details of the images, and finally obtain the preprocessed image sequence.

[0075] Based on the image after histogram equalization in the steps, use the Sobel operator to calculate the gradient of the image. The specific steps include: use the Sobel operator to calculate the horizontal gradient value and vertical gradient value of each pixel in the image after histogram equalization, and calculate the gradient direction and amplitude of each pixel based on the horizontal gradient and vertical gradient.

[0076] Step S3: Process the images in the preprocessed image sequence one by one in sequence order. For the currently processed image:

[0077] Extract multiple line segments from the currently processed image, and fuse and filter the multiple line segments according to the gradient of the currently processed image and the distance between the multiple line segments to obtain the set of line segments to be tracked in the currently processed image;

[0078] For each line segment to be tracked in the set of line segments to be tracked: Based on multiple points on the line segment to be tracked and the next image of the currently processed image, use the multi-point optical flow method to obtain the corresponding tracked line segment of the line segment to be tracked in the next image; finally, obtain the set of tracked line segments of the currently processed image.

[0079] In this step, traverse each image in the preprocessed image sequence in chronological order and process each image.

[0080] First, obtain the tracked line segment features of each image according to the information of the currently processed image and the next image. The specific steps are as follows.

[0081] (1) Apply the EDLines algorithm to the currently processed image to extract multiple line segments. The steps for EDLines to extract line segments in the image are as follows:

[0082] Image filtering and smoothing processing. First, perform Gaussian filtering on the image to suppress noise and achieve smoothing processing; in the smoothed image, calculate the gradient intensity and direction of each pixel point, usually using operators such as Sobel for this calculation; according to the calculated gradient information, identify the set of pixel points called "anchor points". Anchor points refer to those pixel points corresponding to the maximum value generated by the gradient operator, and these points are usually candidate pixel points for the edge part. Anchor points correspond to the pixel points that generate the maximum value of the gradient operator, and these pixel points are also likely to become pixel points of the edge part. By connecting the anchor points, the edges of the image are depicted. The specific process is to start from an anchor point, use the gradient intensity and direction of its neighboring pixels, and search for the next anchor point along the direction of the maximum gradient to complete the connection.

[0083] Line segment extraction and fitting. Traverse the pixel grid in sequence, and apply the least squares line fitting method to fit each line segment until the fitting error exceeds a specific threshold (such as one pixel). After outputting the extracted line segments, the algorithm will continue to iteratively process the remaining pixels on the pixel chain until all pixels are fitted.

[0084] Detection of spurious line segments. Based on the Helmholtz criterion, multiple straight line segments are obtained. At the same time, short straight line segments are usually filtered out because they may be misdetected; while longer straight line segments are usually retained because of the significant deviation from the background, thus ensuring the reliability and accuracy of the extraction results.

[0085] The present invention filters the lengths of multiple straight line segments extracted by the EDLines algorithm, and screens out multiple straight line segments with lengths greater than the minimum length. The specific steps are as follows: Assume that the width of the picture is cols and the height is rows, then the minimum length len is determined according to the width and height of the picture min , to ensure that the extracted straight line segments have a certain degree of significance and reliability. The expression is:

[0086] len min = min(rows, cols) × 0.125

[0087] where min(·) represents calculating the minimum value.

[0088] (2) After completing the length filtering, fuse the filtered multiple straight line segments. Set a merge mask and a mask mask for each straight line segment. In the image, the range where the distance to the straight line segment is less than 5 pixels is determined as the merge mask for screening adjacent straight line segments; the range where the distance to the straight line segment is less than 20 pixels is determined as the mask mask for screening evenly distributed straight line segments in the current image.

[0089] For each of the filtered multiple straight line segments, according to the gradient of the currently processed image calculated in step S2, obtain feature points with obvious gradient changes in regions on this straight line segment. As Figure 2 shown, first evenly divide the straight line segment into 7 regions, discard the regions near both ends of the straight line segment, and retain the middle 5 regions to ensure that the selected feature points are more representative and stable. In the retained middle 5 regions, select 3 points with the largest gradient values at fixed intervals, and then add the two endpoints of the straight line segment to form a set of 5 points {(u i , v i )}, i ∈ {0, 1, 2, 3, 4}. Where u i , v iDenote the horizontal and vertical coordinates of the \(i\)-th point, where \(i = 0\) and \(i = 4\) correspond to the starting point and the ending point of the line respectively. The starting point represents the point closest to the lower left corner of the image, and the ending point represents the point closest to the upper right corner of the image.

[0090] Determine whether the line segment needs to be merged according to the positional relationship between the 5 feature points of the line segment and the merge mask of other line segments. If at least one of the 5 feature points is within the merge mask range of other line segments (i.e., within 5 pixels around the line segment), it indicates that this line segment is very close to other line segments. At this time, the current line segment needs to be merged with the line segment to which the merge mask belongs. During the merging process, the least squares method is used to determine the unified line after merging. Assume that the functional expressions of the two extracted line segments are respectively line segment 1: \(a\) 1 u + b 1 v + c 1 = 0, line segment 2: \(a\) 2 u + b 2 v + c 2 = 0, where \(a\) 1 , b 1 , c 1 are the parameters of line segment 1, \(a\) 2 , b 2 , c 2 are the parameters of line segment 2, and \(x, y\) are the horizontal and vertical coordinates of the pixel points in the image.

[0091] The error function \(E\) of the least squares method is defined as the square of the perpendicular distance from the line segment to the unified line \(a'u + b'v + c' = 0\), and the expression is:

[0092]

[0093] where \((u\) 0 , v 0 ) and \((u\) 4 , v 4 ) are the coordinates of the two endpoints of the line segment respectively, \(|\cdot|\) represents calculating the absolute value, \(a', b', c'\) are the parameters of the unified line, and \(u, v\) are the horizontal and vertical coordinates of the pixel points in the image.

[0094] In order to find the parameters \(a', b'\) and \(c'\) of the unified line, it is necessary to minimize the error function \(E\). The present invention realizes this by taking the partial derivatives of \(E\) with respect to \(a', b'\) and \(c'\) and setting the partial derivatives to 0. The expression is:

[0095]

[0096] Add the constraint \(a'\) 2 + b' 2Let it equal to 1 to ensure the uniqueness of the parameters, solve the above system of equations, obtain the parameters a′, b′, and c′ of the unified straight line, thereby determining the unified straight line. At this time, project the endpoints of line segment 1 and the endpoints of line segment 2 onto a′x + b′y + c′ = 0, select the two farthest endpoints as the endpoints of the fused line segment, and determine the mask mask and merge mask of the fused line segment.

[0097] (3) After completion of the fusion, filter multiple line segments. Determine whether the point p i =(u i , v i ), i ∈ {0, 1, 2, 3, 4} on the line segment is within the mask range of other line segments. If at least one point is within the mask, it indicates that the distribution of the line segment is uneven, discard the line segment, and complete the screening of multiple line segments in the currently processed image in the above manner.

[0098] (4) After confirming that all points on the remaining line segments are not within the merge mask and mask ranges, detect the number of currently valid line segments. If the number of valid line segments is less than the preset valid line segment number threshold, reduce the detection parameters of the EDLines algorithm, and return to step (1) until the number of valid line segments is greater than the valid line segment number threshold. Use the set of finally formed valid line segments as the set of line segments to be tracked in the currently processed image. Figure 3 Shows the processing process of line segment screening and fusion in the above steps.

[0099] After obtaining the set of line segments to be tracked in each image, the present invention uses the multi-point optical flow method to obtain the corresponding line segments to be tracked in the next image based on multiple points on the line segments to be tracked and the next image of the currently processed image; finally obtain the set of line segments to be tracked in the next image, and the processing process is as follows:

[0100] (1) As Figure 4 shown, build an image pyramid for the currently processed image I 1 and the next image I 2 of the currently processed image. Build image pyramids for I 1 and I 2The image is set as the bottom layer of the pyramid, with the resolution remaining at 1.0 and without any scaling. Subsequently, images with different resolutions are generated by downsampling layer by layer to form an image pyramid. Specifically, an image pyramid constructed according to the ratios of 1.0, 0.5, 0.25, and 0.125 means that the resolution of each layer of the image is 100%, 50%, 25%, and 12.5% of the original image respectively. In this structure, the top layer has a resolution of 0.125, the smallest size and resolution, and shows the least amount of detail, thus providing a good foundation for subsequent optical flow tracking at multiple levels of resolution.

[0101] The downsampling process uses a Gaussian filter to smooth the image to reduce the aliasing effect introduced during downsampling. After smoothing, the image is downsampled. During the construction of the image pyramid, the 0.5 layer is obtained by sampling every other pixel from the 1.0 layer, and the 0.25 layer is sampled every other pixel from the 0.5 layer. The pyramid layer processed first is the top layer L 3 (a total of four layers: 0, 1, 2, 3), that is, starting from the pyramid layer with a resolution of 0.125, which is the lowest, and the set p 1 of feature points on all valid line segments in the image I i =(u i ,v i ), i ∈ {0, 1, 2, 3, 4} is scaled to the top layer L 3 of the image pyramid.

[0102] (2) Perform multi-point optical flow tracking for line segments layer by layer. First, start tracking from the top layer L 3 of the image pyramid, calculate the optical flow using the resolution of this layer to obtain the tracking positions of points on the line segment. By going layer by layer downwards (i.e., from L 3 to L 0 ), the image details are gradually restored to ensure the accuracy and robustness of the optical flow tracking. When tracking each layer, first scale the tracking positions of points on the line segment of the previous layer to the current layer, then calculate the optical flow at the resolution of this layer, update the tracking positions of points on the line segment, and finally achieve accurate line segment matching and tracking in the entire image pyramid.

[0103] The tracking principle of the single-point optical flow method is as follows: For a single point, the optical flow method performs tracking by minimizing the photometric error. Assume that at time t, the position of a point is (u, v), and its gray value is I(u, v, t). After a time dt, that is, at time t + dt, the point moves to a new position (u + du, v + dv). Based on this, according to the gray value invariance assumption (i.e., within a small range of motion, the image gray value remains unchanged), the following expression can be obtained:

[0104] I(u + du, v + dv, t + dt) = I(u, v, t)

[0105] Performing a Taylor expansion on this equation and neglecting the higher-order terms gives the expression:

[0106]

[0107] where I u represents the gray-scale derivative of this pixel point in the u direction, and I v represents the gray-scale derivative of this pixel point in the v direction; I t represents the gray-scale derivative of the pixel point gray-scale value with respect to time t, represents the pixel movement speed of the pixel point in the u direction, represents the pixel movement speed of the pixel point in the v direction.

[0108] As Figure 5 shown, for the multi-point optical flow tracking of a straight line segment, assuming that in the image I 1 the multiple points included on the straight line segment are p i =(u i , v i ), i ∈ {0, 1, 2, 3, 4}, and the positions of these points in the image I 2 are {p″ i =(u″ i , v″ i )}, i ∈ {0, 1, 2, 3, 4}. In this case, the position change of the straight line segment in the image I 2 relative to the straight line segment in I 1 is solved. The position change can be described using the parameters (a, b, c), and these parameters define the transformation of the position of the straight line segment in the image i 2 relative to i 1 . a and b represent the displacement changes of the straight line segment in the u and v directions, and c represents the angular change amount of the angle between the straight line segment and the horizontal axis in the images I 1 and I 2 . The angle θ represents the angle between the straight line segment and the horizontal direction of the image, and L nm represents the distance between the points p m and p n on the straight line segment. The expression for the multi-point optical flow tracking of the straight line segment is:

[0109]

[0110] In multi-point optical flow tracking of a straight line segment, it is assumed that each point on the straight line segment satisfies the optical flow tracking equation. There are a total of 5 points, and the unknowns to be solved are 3, namely (a, b, c). For this reason, the Gauss-Newton method is used for iterative solution to calculate the optimal values of (a, b, c). The Gauss-Newton method updates (a, b, c) by minimizing the cost function and gradually approaches the optimal solution. When the cost function fails to converge and reaches the maximum number of iterations, it is considered that the current straight line segment cannot be effectively tracked, so the straight line segment is discarded.

[0111] (3) After obtaining the parameters (a, b, c), calculate the points on the current straight line segment to be tracked in the image I according to the following expressions 1 in the image I 2 to obtain the tracking straight line segment of the current straight line segment to be tracked.

[0112] u″ m = u m + a + L nm c sinθ

[0113] v″ m = v m + b - L nm c cosθ

[0114] where u″ m , v″ m represent the position coordinates of the m-th point on the straight line segment in the image I 2 , u m , v m represent the position coordinates of the m-th point on the straight line segment in the image I 1 .

[0115] Repeat the straight line segment extraction and tracking process until all images in the preprocessed image sequence are processed.

[0116] Step S4: Use the set of tracking straight line segments of each image in the preprocessed image sequence as the result of straight line segment fast tracking and fusion.

[0117] After all image processing is completed, use the set of tracking straight line segments of each image as the result of straight line segment fast tracking and fusion.

[0118] In some embodiments, each straight line segment in the set of tracking straight line segments is marked in the captured image to show the straight line segment extraction result, as Figure 6 shown, Figure 6 the left is the straight line segment extraction result, while Figure 6 the right is the straight line segment fusion result, where adjacent straight line segments in the horizontal frame are fused and filtered, and the straight lines are homogenized.

[0119] Figure 7Table 1 shows the comparison between the method of the present invention and the conventional straight-line segment extraction and tracking methods in multiple data. MY is the method of the present invention, Edline+LK uses EDLine straight-line extraction and LK optical flow tracking, and LSD+LBD uses LSD straight-line segment extraction and LBD line feature descriptor. Table 1 shows the time consumed. The present invention reduces the time consumed by up to 80% compared with LSD+LBD. Although it takes about 10 ms more time than Edline+LK, Figure 7 in the extraction results of Figure 7 , MY shows better straight-line segment extraction and tracking results. In the horizontal frame, MY successfully fuses the straight lines repeatedly extracted at the edges. In the vertical frame, MY removes the densely distributed straight-line segments and fuses them.

[0120] Table 1

[0121]

[0122] From the above results, it can be seen that the straight-line segment fusion and tracking method based on multi-point optical flow provided by the present invention can effectively extract, fuse and track straight-line segments in a series of continuously captured images, showing strong adaptability, being able to cope with different lighting conditions and repeated texture environments, and ensuring reliability in complex scenarios. This method has a small amount of calculation and a fast operation speed, making the processing process more efficient and suitable for real-time applications. At the same time, the algorithm has a solid theoretical basis, is easy to understand and apply, overcomes the deficiencies of the existing descriptor methods and the traditional two-endpoint optical flow tracking methods, and avoids the possible false matching and drift phenomena in the processing process. Therefore, the straight-line segment fusion and tracking method based on multi-point optical flow of the present invention can quickly, accurately extract, fuse and track straight-line segments in continuously captured images, providing a solid foundation for subsequent visual processing and applications.

[0123] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by those skilled in the art within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for fast tracking and fusing straight line segments of an image based on multi-point optical flow, characterized in that: The following steps are involved: Step S1, using a camera to obtain a sequence of images arranged in chronological order, verifying whether the image sequence is continuously shot, and obtaining a sequence of images shot continuously; Step S2, preprocessing all images in the continuously shot image sequence and calculating gradients to obtain a preprocessed image sequence and gradients of each image; Step S3, processing the images in the pre-processed image sequence one by one in sequence order, for the current processed image: Extracting a plurality of straight line segments from the current processing image, fusing and screening the plurality of straight line segments according to the gradient of the current processing image and the distances between the plurality of straight line segments, and obtaining a set of straight line segments to be tracked of the current processing image; For each straight line segment to be tracked in the set of straight line segments to be tracked: based on a plurality of points on the straight line segment to be tracked and an image next to the currently processed image, a multi-point optical flow method is used to obtain a tracking straight line segment corresponding to the straight line segment to be tracked in the next image; Finally, the set of tracked straight line segments of the current processed image is obtained; Step S4: taking the set of tracked straight line segments of each image in the pre-processed image sequence as a result of fast tracking and fusion of straight line segments.

2. The method for fast tracking and fusing straight line segments of an image based on multi-point optical flow according to claim 1, characterized in that: In step S1, the specific steps of verifying whether the image sequence is continuously shot are as follows: for the images in the image sequence, the chi-square distance of the gray value distribution histogram of two consecutive images is calculated, and when the distance is less than a preset chi-square distance threshold, it is determined that the image disparity meets the continuous shooting condition; and the image sequence that meets the continuous shooting condition is used as the continuously shot image sequence.

3. The method for fast tracking and fusing straight line segments of an image based on multi-point optical flow according to claim 2, characterized in that: The step S2 specifically includes: Step S2-1, performing histogram equalization processing on all grayscale images in the continuously shot image sequence to obtain the pre-processed image sequence; Step S2-2, calculating the gradient of each image in the preprocessed image sequence, including: using the Sobel operator to calculate the horizontal gradient value and the vertical gradient value of each pixel in the image, and calculating the gradient direction and amplitude of each pixel based on the horizontal gradient and the vertical gradient.

4. The method for fast tracking and fusing straight line segments of an image based on multi-point optical flow according to claim 3, characterized in that: The step S3 specifically includes: Step S3-1, setting the first image in the pre-processed image sequence as the current processed image; Step S3-2, applying the EDLines algorithm to the current processed image and filtering the result of the EDLines algorithm to extract a plurality of straight line segments; setting a merge mask, a mask and a plurality of feature points for each of the plurality of straight line segments, and fusing the plurality of straight line segments based on the merge mask and the plurality of feature points; Step S3-3, filtering the fused multiple straight line segments based on the mask to obtain multiple valid straight line segments; Step S3-4, if the number of the plurality of valid straight line segments is less than a preset threshold value of the number of valid straight line segments, then reduce the detection parameters of the EDLines algorithm and return to step S3-2 until the number of the valid straight line segments is greater than the threshold value of the number of valid straight line segments; the set of valid straight line segments finally formed is used as the set of straight line segments to be tracked of the current processed image; Step S3-5, respectively establishing an image pyramid of the current processed image and an image next to the current processed image; determining an initial amount of top-level optical flow tracking based on a set of straight line segments to be tracked of the current processed image; Step S3-6, performing multi-point optical flow tracking of straight line segments layer by layer based on the top-level optical flow tracking initial quantity, obtaining the tracking straight line segments corresponding to the straight line segments to be tracked in the current processing image in the next image; and finally obtaining the tracking straight line segment set of the previous processing image; Step S3-7, setting the next image of the currently processed image as the currently processed image, and returning to step S3-2 until the processing of all images in the pre-processed image sequence is completed.

5. The method for fast tracking and fusing straight line segments of an image based on multi-point optical flow according to claim 4, characterized in that: In step S3-2, the step of filtering the result of the EDLines algorithm specifically includes: determining the minimum length len according to the width cols and height rows of the image min , the expression is: len min =min(rows,cols)×0.125 Wherein, min(·) indicates the minimum value to be calculated; multiple straight line segments extracted by the EDLines algorithm are filtered by length, and the straight line segments with a length greater than the minimum length len are selected. min Multiple straight line segments of ; In step S3-2, the steps of setting a merge mask, a mask and a plurality of feature points specifically include: All ranges whose distances to the straight line segment are less than 5 pixels are determined as the merge mask; all ranges whose distances to the straight line segment are less than 20 pixels are determined as the mask; The straight line segment is evenly divided into 7 areas, and the areas close to the two ends of the straight line segment are removed to obtain the middle 5 areas. The gradient of each point in the middle 5 areas is obtained based on the gradient of the current processed image. The 3 points with the largest gradient values ​​are determined based on the gradient of each point in the middle 5 areas. Then the two endpoints of the straight line segment are added to finally form a set of 5 points as the multiple feature points of the straight line segment.

6. The method for fast tracking and fusing straight line segments of an image based on multi-point optical flow according to claim 5, characterized in that: In step S3-2, the step of fusing the plurality of straight line segments based on the merge mask and the plurality of feature points specifically includes: If at least one of the five feature points of a straight line segment is within the merge mask range of other straight line segments, it is determined that fusion is required, and the least squares method is used to determine the unified straight line after fusion, and the following equations are solved: a′ 2 +b′ 2 =1 Where E is the error function of the least squares method, (u0, v0) and (u4, v4) are the coordinates of the two endpoints of the straight line segment, and a′, b′, c′ are the parameters of the equation of the unified straight line; Solve the above equations to obtain the parameters a′, b′ and c′ of the unified straight line to determine the unified straight line; project the endpoints of the straight line segments to be fused onto the unified straight line, select the two farthest endpoints as the endpoints of the fused straight line segment, and obtain the fused straight line segment.

7. The method for fast tracking and fusing straight line segments of an image based on multi-point optical flow according to claim 6, characterized in that: The step S3-3 specifically includes: For the fused multiple straight line segments, determine whether the feature points on the straight line segment are within the mask range of other straight line segments. If at least one feature point on the straight line segment is within the mask range of other straight line segments, remove the straight line segment, and finally obtain multiple valid straight line segments.

8. The method for fast tracking and fusing straight line segments of an image based on multi-point optical flow according to claim 7, characterized in that: The step S3-5 specifically includes: For the current processed image I1 and the next image I2 of the current processed image, set the images I1 and I2 as the bottom layer of the pyramid, and construct a four-layer image pyramid L3, L2, L1, L0 with a resolution of 100%, 50%, 25% and 12.5% ​​of the original image from the bottom layer to the top layer respectively; The set p of feature points on all valid straight line segments in image I1 i =(u i ,v i ), i∈{0,1,2,3,4} is scaled to the top layer L3 of the image pyramid as the initial amount of the top layer optical flow tracking, where u i ,v i Represents the horizontal and vertical coordinates of the i-th point.

9. The method for fast tracking and fusing straight line segments of an image based on multi-point optical flow according to claim 8, characterized in that: The step S3-6 specifically includes: Starting from the top layer L3 of the image pyramid, when tracking each layer, first scale the tracking position of the point on the straight line segment of the previous layer to the current layer, then perform multi-point optical flow tracking at the resolution of the layer, update the tracking position of the point on the straight line segment of the layer, and finally obtain multiple tracked straight line segments of the bottom layer, and obtain the tracked straight line segment set of the pre-processed image; The processing steps of the multi-point optical flow tracking specifically include: (1) Determine the expression for multi-point optical flow tracking: Among them, I u Represents the grayscale derivative of the pixel in the u direction, I v Represents the grayscale derivative of the pixel in the v direction, L nm represents the distance between the mth feature point and the nth feature point on the straight line segment, θ represents the angle between the straight line segment and the horizontal direction of the image, a and b represent the displacement changes of the straight line segment in the u and v directions, c represents the angle change between the straight line segment and the horizontal axis in image I1 and image I2, I t Represents the grayscale derivative of the pixel grayscale value over time t; (2) The Gauss-Newton method is used to iteratively calculate the optimal value of (a, b, c). After obtaining the parameters (a, b, c), the position of each point on the current straight line segment to be tracked in image I1 in image I2 is calculated according to the following expression to obtain the tracking straight line segment of the current straight line segment to be tracked; the m =the m +a+L nm csinθ v″ m =v m +b-L nm ccosθ Among them, u″ m ,v″ m Represents the position coordinates of the mth point on the straight line segment in image I2, u m ,v m Represents the position coordinates of the mth point on the straight line segment in image I1.

Images