A line segment matching method and matching system

Abstract

The present invention discloses a line segment matching method, comprising: obtaining a first line segment set of a current image and a second line segment set of a reference image; constructing a first neighborhood set; constructing a second neighborhood set; selecting a line segment to be matched from the first line segment set, and using the first neighborhood to obtain the neighborhood of the line segment to be matched; obtaining a first candidate line segment set of the line segment to be matched according to the neighborhood of the line segment to be matched and the second neighborhood set; performing significant matching or matching based on affine similarity according to the category of the line segment to be matched using the first candidate line segment set to obtain a candidate line segment set of the line segment to be matched; obtaining a candidate line segment set of each line segment in the first line segment set; constructing a score matrix using the candidate line segment set of each line segment in the first line segment set; and obtaining matching line segments of the first line segment set and the second line segment set by calculating the score matrix. The present invention also discloses a line segment matching system.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and image processing, and in particular to a line segment matching method and a matching system. Background Art

[0002] Feature matching is a basic step in solving many computer vision problems, among which point features and line features are the two most commonly used features. In the past few years, matching schemes based on point features have been widely studied and applied. They can quickly obtain the point feature matching relationship between images, while line features in images are often difficult to quickly and accurately match due to the view transformation between images. However, line features contain more structural information than point features, so using line features for image matching plays an important role in 3D modeling, robot navigation and other fields.

[0003] However, the line segment matching method in the prior art has many problems, such as low robustness, high matching cost, long matching time, low matching accuracy, etc. Summary of the invention

[0004] In view of this, the present invention provides a line segment matching method and a matching system, in order to at least partially solve at least one of the above-mentioned technical problems.

[0005] According to one aspect of the present invention, a line segment matching method is provided, comprising:

[0006] Obtain a first line segment set of a current image and a second line segment set of a reference image;

[0007] Obtaining a neighborhood of each line segment in the first line segment set, constructing a first neighborhood set, and classifying the line segments in the first line segment set using the first neighborhood set;

[0008] Obtain the neighborhood of each line segment in the second line segment set, and construct a second neighborhood set;

[0009] Selecting a line segment to be matched from the first line segment set, and using the first neighborhood to obtain a neighborhood of the line segment to be matched;

[0010] Obtain a first candidate line segment set for the line segment to be matched according to the neighborhood and the second neighborhood set of the line segment to be matched;

[0011] According to the category of the line segment to be matched, the first candidate line segment set is used to perform saliency matching or matching based on affine similarity to obtain a candidate line segment set to be matched;

[0012] Obtain a candidate line segment set for each line segment in the first line segment set;

[0013] Using the candidate line segment set of each line segment in the first line segment set, construct a score matrix;

[0014] By calculating the score matrix, matching line segments of the first line segment set and the second line segment set are obtained.

[0015] According to an embodiment of the present invention, the obtaining of the first line segment set of the current image and the second line segment set of the reference image includes:

[0016] Processing the current image using a straight line detection and segmentation algorithm to obtain a first line segment set;

[0017] Processing the reference image using a line detection and segmentation algorithm to obtain a second line segment set;

[0018] The neighborhood is a rectangular area with the midpoint of the line segment as the center point. The length of the rectangular area is expressed by formula (1), and the width is expressed by formula (2):

[0019] l=α*len(1), w=β*len(2),

[0020] Wherein, α is the preset length ratio coefficient, β is the preset width ratio coefficient, and len represents the length of the line segment;

[0021] The classifying the line segments in the first line segment set by using the first neighborhood set includes:

[0022] Selecting a line segment to be processed from the first line segment set;

[0023] Get the neighborhood of the line segment to be processed;

[0024] According to the number of feature points in the neighborhood of the line segment to be processed, the line segment to be processed is classified into a significant line segment and a weak texture line segment;

[0025] The first line segment set is traversed, and each line segment in the first line segment set is classified.

[0026] According to an embodiment of the present invention, obtaining a first candidate line segment set of the line segment to be matched according to the neighborhood and the second neighborhood set of the line segment to be matched includes:

[0027] Selecting a reference line segment from the second line segment set, and obtaining a neighborhood of the reference line segment using the second neighborhood set;

[0028] Get the neighborhood of the line segment to be matched;

[0029] Using the scale-invariant feature detection algorithm and the nearest neighbor feature matching algorithm, the number of feature points that match each other in the neighborhood of the line segment to be matched and the neighborhood of the reference line segment is obtained;

[0030] When the number of mutually matching feature points is greater than a preset feature point threshold, the reference line segment is added to the first candidate line segment set;

[0031] Traverse the second line segment set to obtain a first candidate line segment set of line segments to be matched;

[0032] The preset feature point threshold is expressed by formula (3):

[0033] V p =s*min(N1,N2) (3),

[0034] Among them, V p represents the preset feature point threshold, s represents the preset proportional coefficient, N1 is the number of matched feature points in the neighborhood of the line segment to be matched, and N2 is the number of matched feature points in the neighborhood of the reference line segment.

[0035] According to an embodiment of the present invention, obtaining a set of candidate line segments to be matched includes:

[0036] When the line segment to be matched is a significant line segment, performing significant matching using the first candidate line segment set;

[0037] Among them, significant matches include:

[0038] Using the neighborhood of the line segment to be matched, construct the homography matrix of the line segment to be matched;

[0039] According to the homography matrix, obtaining a first mapping point and a second mapping point of a first endpoint and a second endpoint of the line segment to be matched on the reference image;

[0040] According to the first mapping point and the second mapping point, obtaining a mapping line segment;

[0041] Calculate the distance between the mapped line segment and each candidate line segment in the first candidate line segment set;

[0042] The candidate line segments whose distance values ​​are less than the preset distance threshold are added to the candidate line segment set.

[0043] According to an embodiment of the present invention, the above mapping point is expressed by formula (4):

[0044] (X′ j ) i =H(X j ) i (4)

[0045] Among them, X′ j represents the jth mapping point, X j represents the jth endpoint, H represents the homography matrix;

[0046] The distance from the mapped line segment to the candidate line segment is expressed by formula (5):

[0047] d=(d1+d2+d3+d4) / 4 (5),

[0048] Among them, d1 represents the distance from the first mapping point of the mapped line segment to the candidate line segment, d2 represents the distance from the second mapping point of the mapped line segment to the candidate line segment, d3 represents the distance from the first endpoint of the candidate line segment to the mapped line segment, and d4 represents the distance from the second endpoint of the candidate line segment to the mapped line segment.

[0049] According to an embodiment of the present invention, when the saliency matching fails, matching is performed on the line segments to be matched based on affine similarity;

[0050] Among them, matching based on affine similarity includes:

[0051] Selecting a reference line segment from the first candidate line segment set, and acquiring a neighborhood of the reference line segment according to the reference line segment neighborhood set;

[0052] Obtain a set of feature point pairs that match each other in the neighborhood of the line segment to be matched and the neighborhood of the reference line segment;

[0053] Select a feature point pair from the feature point pair set, calculate the affine similarity of the remaining feature point pairs, and obtain multiple affine similarity values;

[0054] Select the median value from multiple affine similarity values ​​as the neighborhood similarity value of the feature point pair;

[0055] Traverse the feature point pair set, obtain the neighborhood similarity value of each feature point pair in the feature point pair set, and use the maximum neighborhood similarity value as the affine similarity value between the line segment to be matched and the reference line segment;

[0056] Traverse the first candidate line segment set and obtain the affine similarity value between each reference line segment and the line segment to be matched;

[0057] According to the similarity threshold, reference line segments in the first candidate line segment set whose affine similarity values ​​are greater than the similarity threshold are added to the candidate line segment set.

[0058] According to an embodiment of the present invention, obtaining a set of candidate line segments to be matched includes:

[0059] When the line segment to be matched is a weak texture line segment, the line segment to be matched is matched based on affine similarity;

[0060] Among them, matching based on affine similarity includes:

[0061] Select a reference line segment from the first candidate line segment set, and calculate the angle difference between the line segment to be matched and the reference line segment;

[0062] According to the angle difference, the reference line segment and the feature points in the reference line segment neighborhood are rotated;

[0063] Performing spatial encoding and screening on the feature points in the neighborhood of the rotated reference line segment to obtain the feature points in the neighborhood of the screened reference line segment;

[0064] Using the feature points in the neighborhood of the line segment to be matched and the feature points in the neighborhood of the filtered reference line segment, the affine similarity value of the line segment to be matched and the reference line segment is calculated;

[0065] The first candidate line segment set is traversed, and reference line segments whose affine similarity values ​​are greater than the similarity threshold are added to the candidate line segment set.

[0066] According to an embodiment of the present invention, constructing the score matrix includes:

[0067] Determine the number of rows and columns of the score matrix according to the number of line segments in the first line segment set and the number of line segments in the second line segment set;

[0068] Initialize the elements in the score matrix to 0;

[0069] Using the candidate line segment set of each line segment in the first line segment set to assign a value to each row element of the score matrix, the construction of the score matrix is ​​completed;

[0070] The value of the element F(i, j) of the score matrix represents the matching value between the i-th line segment of the first line segment set and the j-th line segment of the second line segment set.

[0071] According to an embodiment of the present invention, the value of the element F(i, j) of the above score matrix is ​​obtained in the following manner:

[0072] When the i-th line segment of the second line segment set does not belong to the candidate line segment set of the i-th line segment of the first line segment set, the value of the element F(i, j) of the score matrix is ​​the initial value 0;

[0073] When the j-th line segment of the second line segment set belongs to the candidate set of the i-th line segment of the first line segment set and the candidate set of the i-th line segment of the first line segment set is obtained by saliency matching, the value of the element F(i, j) of the score matrix is ​​expressed by formula (6):

[0074] F(i, j) = 1 + 1 / d (6),

[0075] Wherein, d represents the distance between the j-th line segment of the second line segment set and the i-th line segment of the first line segment set, wherein d is smaller than a preset distance threshold;

[0076] When the j-th line segment of the second line segment set belongs to the candidate set of the i-th line segment of the first line segment set and the candidate set of the i-th line segment of the first line segment set is obtained by matching based on affine similarity, the value of the element F(i, j) of the score matrix is ​​expressed by formula (7):

[0077]

[0078] in, represents the i-th line segment of the first line segment set, represents the j-th line segment of the second line segment set, and SIM represents the affine similarity value between the j-th line segment of the second line segment set and the i-th line segment of the first line segment set.

[0079] According to an embodiment of the present invention, obtaining matching line segments of the first line segment set and the second line segment set by calculating the score matrix includes:

[0080] Traverse the score matrix and obtain the value of the element F(i, j) of the score matrix;

[0081] When the value of the element F(i, j) of the score matrix is ​​simultaneously the maximum value of the i-th row and the j-th column, the i-th line segment of the first line segment set and the j-th line segment of the second line segment set match each other.

[0082] According to another aspect of the present invention, there is provided a line segment matching system, comprising:

[0083] A first acquisition module, used to acquire a first line segment set of a current image and a second line segment set of a reference image;

[0084] A first construction module, used for obtaining the neighborhood of each line segment in the first line segment set, constructing a first neighborhood set, and classifying the line segments in the first line segment set by using the first neighborhood set;

[0085] A second construction module is used to obtain the neighborhood of each line segment in the second line segment set and construct a second neighborhood set;

[0086] A second acquisition module is used to select a line segment to be matched from the first line segment set, and obtain a neighborhood of the line segment to be matched using the first neighborhood;

[0087] A third acquisition module, used for acquiring a first candidate line segment set of the line segment to be matched according to the neighborhood and the second neighborhood set of the line segment to be matched;

[0088] A matching module, configured to perform saliency matching or affine similarity-based matching using the first candidate line segment set according to the category of the line segment to be matched, to obtain a candidate line segment set for the line segment to be matched;

[0089] A fourth acquisition module, used to acquire a candidate line segment set for each line segment in the first line segment set;

[0090] A third construction module is used to construct a score matrix using the candidate line segment set of each line segment in the first line segment set;

[0091] The fifth acquisition module is used to acquire matching line segments of the first line segment set and the second line segment set by calculating a score matrix.

[0092] The line segment matching method provided by the present invention fully utilizes the neighborhood features of the line segments to classify the line segments, and performs corresponding matching according to the categories of the line segments. It can improve the robustness and accuracy of line segment matching while increasing the line segment matching speed; at the same time, the line segment matching system provided by the present invention has a wide range of application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 A flowchart of a line segment matching method according to an embodiment of the present invention is schematically shown;

[0094] Figure 2 is a schematic process diagram of line segment matching according to an embodiment of the present invention;

[0095] Figure 3 is a neighborhood diagram of a line segment according to an embodiment of the present invention;

[0096] Figure 4 A flowchart of classifying line segments in a first line segment set according to an embodiment of the present invention is schematically shown;

[0097] Figure 5 A flowchart of obtaining a first candidate line segment set for line segments to be matched according to an embodiment of the present invention is schematically shown;

[0098] Figure 6 Schematically shows a flow chart of saliency matching according to an embodiment of the present invention;

[0099] Figure 7 A flowchart of matching affine similarity of significant line segments according to an embodiment of the present invention is schematically shown;

[0100] Figure 8 is a schematic diagram of the construction of an affine invariant according to an embodiment of the present invention;

[0101] Fig. 9 is a flow chart of matching affine similarity of weak texture line segments according to an embodiment of the present invention;

[0102] Fig.10 Schematically shows a flow chart of constructing a score matrix according to an embodiment of the present invention;

[0103] Fig.11 The structure diagram of a line segment matching system according to an embodiment of the present invention is schematically shown. DETAILED DESCRIPTION

[0104] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0105] Figure 1 The flowchart of a line segment matching method according to an embodiment of the present invention is schematically shown.

[0106] like Figure 1 As shown, operations S110 to S190 are included.

[0107] In operation S110, a first line segment set of a current image and a second line segment set of a reference image are acquired;

[0108] Current Image I C and reference image I R are two images for matching each other, wherein the i-th element of the line segment set represents the i-th line segment, for example, the first element of the first line segment set represents the first line segment of the first line segment set.

[0109] In operation S120, a neighborhood of each line segment in the first line segment set is obtained, a first neighborhood set is constructed, and the line segments in the first line segment set are classified using the first neighborhood set;

[0110] The line segments in the first line segment set are classified according to the number of feature points in the neighborhood. Through the above classification process, each line segment in the first line segment set is either a significant line segment or a weak texture line segment.

[0111] In operation S130, a neighborhood of each line segment in the second line segment set is obtained to construct a second neighborhood set;

[0112] In operation S140, a line segment to be matched is selected from the first line segment set, and a neighborhood of the line segment to be matched is obtained using the first neighborhood;

[0113] In operation S150, a first candidate line segment set of the line segment to be matched is obtained according to the neighborhood and the second neighborhood set of the line segment to be matched;

[0114] The line segments in the first candidate line segment set are from the second line segment set and are marked with element positions of the second line segment set, such as the elements of the first candidate line segment set. represents the i-th line segment in the first candidate line segment set and the j-th line segment in the second line segment set.

[0115] In operation S160, according to the category of the line segment to be matched, a saliency matching or a matching based on affine similarity is performed using the first candidate line segment set to obtain a candidate line segment set to be matched;

[0116] The line segments in the candidate line segment set are from the first candidate line segment set, and the element position marks of the second line segment set are retained.

[0117] In operation S170, a candidate line segment set for each line segment in the first line segment set is obtained;

[0118] Operations S140 to S160 are iterated to obtain a candidate line segment set for each line segment in the first line segment set.

[0119] In operation S180, a score matrix is ​​constructed using a candidate line segment set for each line segment in the first line segment set;

[0120] In operation S190 , matching line segments of the first line segment set and the second line segment set are obtained by calculating a score matrix.

[0121] Figure 2 is a schematic process diagram of line segment matching according to an embodiment of the present invention, such as Figure 2 As shown, for the two images to be matched, the corresponding line segment sets are extracted respectively and the corresponding line segment neighborhoods are constructed respectively. The statistical constraints are used to obtain the initial candidate line segment set of the line segments to be matched, and different matching strategies are adopted according to the types of the line segments to be matched, that is, significant line segments or weak texture line segments. When the line segments to be matched are significant line segments, significant line segment matching based on the local homography matrix is ​​performed. If the matching is successful, a score matrix is ​​constructed. If the matching is unsuccessful or the line segments to be matched are weak texture line segments, line segment matching based on affine similarity is performed. After constructing the score matrix, the best matching result of the line segments to be matched, that is, the best matching line segments, can be obtained by analyzing the score matrix.

[0122] Through the above line segment matching method, the feature points in the neighborhood of the line segment are fully utilized to classify and match the line segments, which greatly improves the matching efficiency of the line segments and improves the matching accuracy of the line segments.

[0123] According to an embodiment of the present invention, the obtaining of the first line segment set of the current image and the second line segment set of the reference image includes:

[0124] Processing the current image using a straight line detection and segmentation algorithm to obtain a first line segment set;

[0125] Processing the reference image using a line detection and segmentation algorithm to obtain a second line segment set;

[0126] By extracting the line segments of the two images to be matched (the current image and the reference image) through the above-mentioned line segmentation algorithm (Line Segment Detector, LSD), the corresponding line segment set can be obtained, such as L c = {l c1 , lc2 ,…,l cn} and L R = {l r1 , l r2 ,…,l rm}, where n and m represent the total number of extracted line segments.

[0127] Figure 3 4 is a neighborhood diagram of a line segment according to an embodiment of the present invention.

[0128] Combine the following Figure 3 Further explanation of the neighborhood of the line segment is given, such as Figure 3 As shown in the figure, the neighborhood is a rectangular area with the midpoint of the line segment as the center point. The length of the rectangular area is expressed by formula (1), and the width is expressed by formula (2):

[0129] l=α*len(1), w=β*len (2),

[0130] Wherein, α is the preset length ratio coefficient, β is the preset width ratio coefficient, and len represents the length of the line segment;

[0131] Figure 4 A flowchart for classifying line segments in a first line segment set according to an embodiment of the present invention is schematically shown.

[0132] like Figure 4 As shown, operation S410 to operation S440 are included.

[0133] In operation S410, a line segment to be processed is selected from a first line segment set;

[0134] In operation S420, a neighborhood of the line segment to be processed is obtained;

[0135] In operation S430, the line segment to be processed is classified into a significant line segment and a weak texture line segment according to the number of feature points in the neighborhood of the line segment to be processed;

[0136] In operation S440 , the first line segment set is traversed, and each line segment in the first line segment set is classified.

[0137] By using the above method, line segments of two images to be matched (current image and reference image) are extracted, and the extracted line segments are matched with each other. Meanwhile, the above line segment matching method can also be applied to image matching.

[0138] Figure 5 The flowchart for obtaining a first candidate line segment set for line segments to be matched according to an embodiment of the present invention is schematically shown.

[0139] like Figure 5 As shown, operation S510 to operation S550 are included.

[0140] In operation S510, a reference line segment is selected from the second line segment set, and a neighborhood of the reference line segment is acquired using the second neighborhood set;

[0141] In operation S520, a neighborhood of the line segment to be matched is obtained;

[0142] In operation S530, a scale-invariant feature detection algorithm and a nearest neighbor feature matching algorithm are used to obtain the number of feature points that match each other in the neighborhood of the line segment to be matched and the neighborhood of the reference line segment;

[0143] Traverse the image L C With L R The line segment pairs in (l ci , l rj ), statistical line segment l c With line segment l r The number of feature points that match each other in the neighborhood of , where i and j are the numbers of the line segments in the line segment set, i.e. l ci is the i-th line segment in the first line segment set; the scale-invariant feature detection algorithm is a feature extraction algorithm, and the nearest neighbor feature matching algorithm is a feature matching algorithm.

[0144] In operation S540, when the number of mutually matching feature points is greater than a preset feature point threshold, the reference line segment is added to the first candidate line segment set;

[0145] In operation S550, the second line segment set is traversed to obtain a first candidate line segment set of line segments to be matched;

[0146] The preset feature point threshold is expressed by formula (3):

[0147] V p =s*min(N1,N2) (3),

[0148] Among them, V p represents the preset feature point threshold, s represents the preset proportional coefficient, N1 is the number of matched feature points in the neighborhood of the line segment to be matched, and N2 is the number of matched feature points in the neighborhood of the reference line segment.

[0149] When p>V p , then it is considered that the two line segments may be matched, and line segment l rj is line segment l ci Candidate matching line segments, then the line segment l rj Add to the first candidate line segment set.

[0150] From the smoothness of the motion, it can be seen that if a pair of line segments are matched, most of the feature points in their neighborhood should also be matched. Therefore, the present invention proposes to determine the candidate line segments using the number of matching feature points in the line segment neighborhood.

[0151] The above method aims to determine candidate line segments through statistical constraints, and determining reasonable candidate line segments can effectively improve the matching speed.

[0152] According to an embodiment of the present invention, obtaining a set of candidate line segments to be matched includes:

[0153] When the line segment to be matched is a significant line segment, performing significant matching using the first candidate line segment set;

[0154] in, Figure 6 A flowchart of saliency matching according to an embodiment of the present invention is schematically shown. Figure 6 As shown, operations S610 to S650 are included.

[0155] In operation S610, a homography matrix of the line segment to be matched is constructed using a neighborhood of the line segment to be matched;

[0156] In operation S620, a first mapping point and a second mapping point of a first endpoint and a second endpoint of a line segment to be matched on a reference image are acquired according to the homography matrix;

[0157] In operation S630, a mapping line segment is acquired according to the first mapping point and the second mapping point;

[0158] In operation S640, a distance between the mapped line segment and each candidate line segment in the first candidate line segment set is calculated;

[0159] In operation S650, candidate line segments whose distance values ​​are less than a preset distance threshold are added to a candidate line segment set.

[0160] When the line segment to be matched is a significant line segment, since there are many features in the neighborhood of the significant line segment, it can be used to calculate the local homography matrix and then perform line segment matching. For the line segment that successfully restores the local homography matrix, the candidate matching line segment can be obtained by projecting the homography matrix between images.

[0161] According to an embodiment of the present invention, the above mapping point is expressed by formula (4):

[0162] (X′ j ) i =H(X j ) i (4)

[0163] Among them, X′ j represents the jth mapping point, X j represents the jth endpoint, H represents the homography matrix;

[0164] The distance from the mapped line segment to the candidate line segment is expressed by formula (5):

[0165] d=(d1+d2+d3+d4) / 4 (5),

[0166] Among them, d1 represents the distance from the first mapping point of the mapped line segment to the candidate line segment, d2 represents the distance from the second mapping point of the mapped line segment to the candidate line segment, d3 represents the distance from the first endpoint of the candidate line segment to the mapped line segment, and d4 represents the distance from the second endpoint of the candidate line segment to the mapped line segment.

[0167] The saliency matching process of the saliency line segments is performed by selecting the line segment to be matched l in the first line segment set. ci And obtain the successfully matched feature points in its neighborhood, calculate the homography matrix H, and according to the homography matrix, transform the line segment l ci The two endpoints X1, X2 are mapped to image I R In the above equation, we get two mapping points X′1 and X′2 of the mapped line segment, and then get the mapped line segment l′ ci , after obtaining the mapping line segment, calculate the mapping line segment l′ ci and the candidate line segment l in the first candidate line segment set rj The distances d1, d2, d3, and d4 between the endpoints and the other line segment are calculated, and the distance d from the mapping line segment to the candidate line segment is obtained by averaging the above distances. When the distance satisfies d<λ, where λ represents the pre-set distance threshold, l rj Add to the candidate line segment set.

[0168] According to an embodiment of the present invention, when the saliency matching fails, matching is performed on the line segments to be matched based on affine similarity;

[0169] Figure 7 A flowchart of matching affine similarity of significant line segments according to an embodiment of the present invention is schematically shown. Figure 7 As shown, operation S710 to operation S770 are included.

[0170] In operation S710, a reference line segment is selected from the first candidate line segment set, and a neighborhood of the reference line segment is acquired according to the reference line segment neighborhood set;

[0171] In operation S720, a set of mutually matching feature point pairs in a neighborhood of the line segment to be matched and a neighborhood of the reference line segment is obtained;

[0172] In operation S730, a feature point pair is selected from the feature point pair set, and the affine similarities of the remaining feature point pairs are calculated to obtain a plurality of affine similarity values;

[0173] In operation S740, a median value is selected from the plurality of affine similarity values ​​as a neighborhood similarity value of the feature point pair;

[0174] In operation S750, the feature point pair set is traversed to obtain the neighborhood similarity value of each feature point pair in the feature point pair set, and the maximum neighborhood similarity value is used as the affine similarity value of the line segment to be matched and the reference line segment;

[0175] In operation S760, the first candidate line segment set is traversed to obtain an affine similarity value between each reference line segment and the line segment to be matched;

[0176] In operation S770, according to the similarity threshold, reference line segments in the first candidate line segment set whose affine similarity values ​​are greater than the similarity threshold are added to the candidate line segment set.

[0177] Since most of the salient line segments that successfully restore the local homography matrix have found matching line segments, for the salient line segments that fail to restore the local homography matrix, it is necessary to calculate the affine invariant between the line points to measure the similarity between images and then determine the matching line segments.

[0178] Figure 8 is a schematic diagram of the construction of an affine invariant according to an embodiment of the present invention, such as Figure 8 As shown, the principle of affine invariant is that for spatial points P1, P2 and spatial line L, its image L C and I R The projections are X1, X2, l c , Y1, Y2, l r , then they are matched respectively, and the distance ratio between the matching feature points and the matching straight lines can be described by the affine invariant, as shown in the following formula: Among them, for the image L C with I R A pair of candidate matching line segments is obtained, and the matching point pairs in its neighborhood are obtained. Where h represents the number of matching feature points; any pair of matching points (x k ,y k ) as the base point, and construct an affine invariant with the remaining h-1 pairs of matching points to obtain h-1 affine similarities. The median value is selected as the neighborhood similarity under this base point, that is, Sim(x k ,y k )=median{affsim(x i ,y i , x k ,y k , l r , l c )}; Each pair of matching feature points in the neighborhood is used as the base point to calculate the neighborhood similarity, and the maximum value of all similarities is taken as the similarity between the line segment neighborhoods, that is, SIML(l r , l cp )=max{Sim(xk ,y k )}, k∈[1,h]; the maximum similarity between the left and right neighbors of the two line segments is taken as the similarity between the two line segments, that is, SIM(l r , l c )=max(SIMR(l r , l c ), SIML(l r , l c )); if SIM(l r , l c )>0.95, then the line segment l r With its candidate line segment l c is the correctly matched line segment.

[0179] According to an embodiment of the present invention, obtaining a set of candidate line segments to be matched includes:

[0180] When the line segment to be matched is a weak texture line segment, the line segment to be matched is matched based on affine similarity;

[0181] Fig. 9 The figure is a flow chart of matching the affine similarity of weak texture line segments according to an embodiment of the present invention.

[0182] like Fig. 9 As shown, operations S910 to S950 are included.

[0183] In operation S910, a reference line segment is selected from the first candidate line segment set, and an angle difference between the line segment to be matched and the reference line segment is calculated;

[0184] In operation S920, the reference line segment and the feature points in the neighborhood of the reference line segment are rotated according to the angle difference;

[0185] In operation S930, spatial encoding and filtering are performed on the feature points in the neighborhood of the rotated reference line segment to obtain the feature points in the neighborhood of the filtered reference line segment;

[0186] In operation S940, the affine similarity value between the line segment to be matched and the reference line segment is calculated using the feature points in the neighborhood of the line segment to be matched and the feature points in the neighborhood of the filtered reference line segment;

[0187] In operation S950, the first candidate line segment set is traversed, and reference line segments whose affine similarity values ​​are greater than a similarity threshold are added to the candidate line segment set.

[0188] The above matching process is for the line segment to be matched is a weak texture line segment, and the angle θ of the line segment is calculated by calculating the gradient of the line segment; for image I C with I R A pair of candidate matching segments (l c , lr ), according to the candidate line segment l rp The difference in angle Δθ = θ c -θ r , the line segment l c The feature points in its neighborhood rotate Δθ around the endpoint of the line segment; r The feature points in the neighborhood and the rotated feature points are encoded in spatial position respectively, and invalid matching point pairs are eliminated according to the topological relationship to obtain topologically consistent feature point pairs; l is calculated based on the selected matching points r With l c The similarity of r With l c The similarity between c , l r )=SIM(l c , l′ r ); if SIM(l c , l r )>0.95, then the line segment l r With its candidate line segment l c is the correctly matched line segment.

[0189] Fig.10 The flowchart of constructing a score matrix according to an embodiment of the present invention is schematically shown.

[0190] like Fig.10 As shown, operation S1010 to operation S1030 are included.

[0191] In operation S1010, the number of rows and columns of the score matrix is ​​determined according to the number of line segments of the first line segment set and the number of line segments of the second line segment set;

[0192] In operation S1020, the elements in the score matrix are initialized to 0;

[0193] In operation S1030, a candidate line segment set of each line segment in the first line segment set is used to assign a value to each row element of the score matrix, thereby completing the construction of the score matrix;

[0194] The value of the element F(i, j) of the score matrix represents the matching value between the i-th line segment of the first line segment set and the i-th line segment of the second line segment set.

[0195] According to an embodiment of the present invention, the value of the element F(i, j) of the above score matrix is ​​obtained in the following manner:

[0196] When the i-th line segment of the second line segment set does not belong to the candidate line segment set of the i-th line segment of the first line segment set, the value of the element F(i, j) of the score matrix is ​​the initial value 0;

[0197] When the i-th line segment of the second line segment set belongs to the candidate set of the i-th line segment of the first line segment set and the candidate set of the i-th line segment of the first line segment set is obtained by saliency matching, the value of the element F(i, j) of the score matrix is ​​expressed by formula (6):

[0198] F(i, j) = 1 + 1 / d (6),

[0199] Wherein, d represents the distance between the ith line segment of the second line segment set and the ith line segment of the first line segment set, wherein d is smaller than a preset distance threshold;

[0200] For example, for a line segment that is successfully restored by restoring the local homography matrix Find the line segment closest to it If the distance between the two satisfies d<λ, then F(s, t)=1+1 / d, otherwise the line segment will find the best match based on the line point affine invariant constraint.

[0201] When the j-th line segment of the second line segment set belongs to the candidate set of the i-th line segment of the first line segment set and the candidate set of the i-th line segment of the first line segment set is obtained by matching based on affine similarity, the value of the element F(i, j) of the score matrix is ​​expressed by formula (7):

[0202]

[0203] in, represents the i-th line segment of the first line segment set, represents the j-th line segment of the second line segment set, and SIM represents the affine similarity value between the j-th line segment of the second line segment set and the i-th line segment of the first line segment set.

[0204] For example, for the matching line segments obtained by the line point affine invariant constraint and If the similarity between two line segments satisfies Then there is Otherwise the two line segments do not match.

[0205] According to an embodiment of the present invention, obtaining the matching line segment of each line segment in the first line segment set and the second line segment set by calculating the score matrix includes:

[0206] Traverse the score matrix and obtain the value of the element F(i, j) of the score matrix;

[0207] When the value of the element F(i, j) of the score matrix is ​​simultaneously the maximum value of the i-th row and the j-th column, the i-th line segment of the first line segment set and the j-th line segment of the second line segment set match each other.

[0208] Traverse the score matrix, if F(i, j) is the maximum value of both the i-th row and the j-th column, then it is considered and is the best matching line segment.

[0209] The above process of constructing a score matrix and obtaining the best matching line segment to be matched by calculating the score matrix is ​​aimed at solving the preliminary matching results between segments. However, after satisfying the statistical constraints and the similarity requirements between the line segments, there is a one-to-many or many-to-one problem in the matching line segments in the two images.

[0210] Fig.11 The structure diagram of a line segment matching system according to an embodiment of the present invention is schematically shown.

[0211] like Fig.11 As shown, the line segment matching system 1100 includes a first acquisition module 1110 , a first construction module 1120 , a second construction module 1130 , a second acquisition module 1140 , a third acquisition module 1150 , a matching module 1160 , a fourth acquisition module 1170 , a third construction module 1180 and a fifth acquisition module 1190 .

[0212] A first acquisition module 1110, configured to acquire a first line segment set of a current image and a second line segment set of a reference image;

[0213] A first construction module 1120, configured to obtain a neighborhood of each line segment in the first line segment set, construct a first neighborhood set, and classify the line segments in the first line segment set using the first neighborhood set;

[0214] A second construction module 1130 is used to obtain the neighborhood of each line segment in the second line segment set and construct a second neighborhood set;

[0215] A second acquisition module 1140 is used to select a line segment to be matched from the first line segment set, and obtain a neighborhood of the line segment to be matched using the first neighborhood;

[0216] A third acquisition module 1150 is used to acquire a first candidate line segment set of the line segment to be matched according to the neighborhood and the second neighborhood set of the line segment to be matched;

[0217] A matching module 1160 is used to perform saliency matching or affine similarity-based matching using the first candidate line segment set according to the category of the line segment to be matched, so as to obtain a candidate line segment set to be matched;

[0218] A fourth acquisition module 1170 is used to acquire a candidate line segment set for each line segment in the first line segment set;

[0219] A third construction module 1180 is used to construct a score matrix using the candidate line segment set of each line segment in the first line segment set;

[0220] The fifth acquisition module 1190 is used to acquire matching line segments of the first line segment set and the second line segment set by calculating a score matrix.

[0221] The above line segment matching system can be applied in 3D reconstruction. For example, in the process of 3D reconstruction through images, the robust line segment matching method can improve the accuracy of reconstruction and restore the structural information of the 3D scene well, such as the spatial structure of the transmission tower. It can also be applied to robot simultaneous localization and mapping (SLAM). For example, visual SLAM technology generally calculates the pose by matching feature points, but it is difficult to get enough feature points in some weak texture areas. At this time, the line segment feature can be used to make up for this problem. The robust line segment matching method can help optimize the robot pose and achieve better positioning effect.

[0222] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A line segment matching method, comprising: Obtain a first line segment set of a current image and a second line segment set of a reference image; Acquire a neighborhood of each line segment in the first line segment set, construct a first neighborhood set, and classify the line segments in the first line segment set using the first neighborhood set; Obtaining the neighborhood of each line segment in the second line segment set, and constructing a second neighborhood set; Selecting a line segment to be matched from the first line segment set, and using the first neighborhood to obtain a neighborhood of the line segment to be matched; Acquire a first candidate line segment set for the line segment to be matched according to the neighborhood of the line segment to be matched and the second neighborhood set; According to the category of the line segment to be matched, using the first candidate line segment set to perform saliency matching or matching based on affine similarity to obtain the candidate line segment set to be matched; Obtain a candidate line segment set for each line segment in the first line segment set; constructing a score matrix using a candidate line segment set for each line segment in the first line segment set; Obtaining matching line segments of the first line segment set and the second line segment set by calculating the score matrix; Wherein, obtaining the candidate line segment set to be matched comprises: When the line segment to be matched is a significant line segment, the first candidate line segment set is used to perform significant matching; when the line segment to be matched is a weak texture line segment, the line segment to be matched is matched based on affine similarity; Wherein, when the saliency matching fails, matching is performed on the line segments to be matched based on affine similarity; Wherein, according to the neighborhood of the to-be-matched line segment and the second neighborhood set, obtaining a first candidate line segment set of the to-be-matched line segment comprises: Selecting a reference line segment from the second line segment set, and acquiring a neighborhood of the reference line segment using the second neighborhood set; Acquire the neighborhood of the line segment to be matched; and acquire the number of mutually matching feature points in the neighborhood of the line segment to be matched and the neighborhood of the reference line segment using a scale-invariant feature detection algorithm and a nearest neighbor feature matching algorithm; When the number of mutually matching feature points is greater than a preset feature point threshold, the reference line segment is added to the first candidate line segment set; the second line segment set is traversed to obtain the first candidate line segment set of the line segment to be matched; The preset feature point threshold is expressed by formula (3): In p =s*min(N1,N2) (3), Among them, V p represents the preset feature point threshold, s represents the preset proportional coefficient, N1 is the number of feature points matched in the neighborhood of the line segment to be matched, and N2 is the number of feature points matched in the neighborhood of the reference line segment; Wherein, constructing the scoring matrix includes: Determine the number of rows and columns of the score matrix according to the number of line segments in the first line segment set and the number of line segments in the second line segment set; initialize the elements in the score matrix to 0; Assigning a value to each row element of the score matrix using a candidate line segment set of each line segment in the first line segment set, thereby completing the construction of the score matrix; The value of the element F(i, j) of the score matrix represents the matching value between the i-th line segment of the first line segment set and the j-th line segment of the second line segment set.

2. The method according to claim 1, wherein: Acquiring a first line segment set of a current image and a second line segment set of a reference image includes: Processing the current image using a line detection and segmentation algorithm to obtain the first line segment set; Processing the reference image using a line detection and segmentation algorithm to obtain the second line segment set; The neighborhood is a rectangular area with the midpoint of the line segment as the center point. The length of the rectangular area is expressed by formula (1), and the width is expressed by formula (2): l=α*len(1), w=β*len(2), Wherein, α is a preset length ratio coefficient, β is a preset width ratio coefficient, and len represents the length of the line segment; The classifying the line segments in the first line segment set by using the first neighborhood set includes: Selecting a line segment to be processed from the first line segment set; Obtaining the neighborhood of the line segment to be processed; Classifying the line segment to be processed into a significant line segment and a weak texture line segment according to the number of feature points in the neighborhood of the line segment to be processed; The first line segment set is traversed, and each line segment in the first line segment set is classified.

3. The method according to claim 1, wherein: The significant matches include: Constructing a homography matrix of the line segment to be matched using the neighborhood of the line segment to be matched; According to the homography matrix, obtaining a first mapping point and a second mapping point of a first endpoint and a second endpoint of the to-be-matched line segment on the reference image; Acquire the mapping line segment according to the first mapping point and the second mapping point; Calculating the distance between the mapped line segment and each candidate line segment in the first candidate line segment set; The candidate line segments whose distance values ​​are less than a preset distance threshold are added to the candidate line segment set.

4. The method according to claim 3, wherein: The mapping point is expressed by formula (4): (X′ j ) i =H(X j ) i (4), Among them, X′ j represents the jth mapping point, X j represents the jth endpoint, H represents the homography matrix; The distance from the mapped line segment to the candidate line segment is expressed by formula (5): d=(d1+d2+d3+d4) / 4 (5), Among them, d1 represents the distance from the first mapping point of the mapped line segment to the candidate line segment, d2 represents the distance from the second mapping point of the mapped line segment to the candidate line segment, d3 represents the distance from the first endpoint of the candidate line segment to the mapped line segment, and d4 represents the distance from the second endpoint of the candidate line segment to the mapped line segment.

5. The method according to claim 3, wherein: The matching based on affine similarity includes: Selecting a reference line segment from the first candidate line segment set, and acquiring a neighborhood of the reference line segment according to the reference line segment neighborhood set; Acquire a set of mutually matching feature point pairs in the neighborhood of the line segment to be matched and the neighborhood of the reference line segment; Selecting a feature point pair from the feature point pair set, calculating the affine similarity of the remaining feature point pairs, and obtaining a plurality of affine similarity values; Selecting a median value from the multiple affine similarity values ​​as a neighborhood similarity value of the feature point pair; Traversing the feature point pair set, obtaining a neighborhood similarity value of each feature point pair in the feature point pair set, and taking the maximum neighborhood similarity value as the affine similarity value between the line segment to be matched and the reference line segment; Traversing the first candidate line segment set, obtaining an affine similarity value between each reference line segment and the line segment to be matched; According to a similarity threshold, reference line segments in the first candidate line segment set whose affine similarity values ​​are greater than the similarity threshold are added to the candidate line segment set.

6. The method according to claim 1, wherein: The matching based on affine similarity includes: Select a reference line segment from the first candidate line segment set, and calculate the angle difference between the line segment to be matched and the reference line segment; Rotating the reference line segment and the feature points in the neighborhood of the reference line segment according to the angle difference; Performing spatial encoding and screening on the feature points in the neighborhood of the rotated reference line segment to obtain the feature points in the neighborhood of the screened reference line segment; Calculating an affine similarity value between the line segment to be matched and the reference line segment by using the feature points in the neighborhood of the line segment to be matched and the feature points in the neighborhood of the filtered reference line segment; The first candidate line segment set is traversed, and reference line segments whose affine similarity values ​​are greater than a similarity threshold are added to the candidate line segment set.

7. The method according to claim 1, wherein the value of the element F(i, j) of the score matrix is ​​obtained by: When the j-th line segment of the second line segment set does not belong to the candidate line segment set of the i-th line segment of the first line segment set, the value of the element F(i, j) of the score matrix is ​​an initial value of 0; When the j-th line segment of the second line segment set belongs to the candidate set of the i-th line segment of the first line segment set and the candidate set of the i-th line segment of the first line segment set is obtained by saliency matching, the value of the element F(i, j) of the score matrix is ​​expressed by formula (6): F(i, j) = 1 + 1 / d (6), in, d represents the distance between the j-th line segment of the second line segment set and the i-th line segment of the first line segment set, wherein d is smaller than a preset distance threshold; When the jth line segment of the second line segment set belongs to the candidate set of the ith line segment of the first line segment set and the candidate set of the ith line segment of the first line segment set is obtained by matching based on affine similarity, the value of the element F(i, j) of the score matrix is ​​expressed by formula (7): in, represents the i-th line segment of the first line segment set, represents the j-th line segment of the second line segment set, and SIM represents the affine similarity value between the j-th line segment of the second line segment set and the i-th line segment of the first line segment set.

8. The method according to claim 1, wherein: Obtaining matching line segments of the first line segment set and the second line segment set by calculating the score matrix includes: Traversing the score matrix to obtain the value of the element F(i, j) of the score matrix; When the value of the element F(i, j) of the score matrix is ​​simultaneously the maximum value of the i-th row and the j-th column, the i-th line segment of the first line segment set and the j-th line segment of the second line segment set match each other.

Images