A Line Feature Matching Method Based on Fourier Representation and Posterior Probability Estimation

Abstract

The present invention discloses a line feature matching method based on Fourier representation and posterior probability estimation, including: detecting and extracting line features on an image sequence pair, calculating binary descriptors; obtaining initial line feature matching pairs; introducing a compact Fourier series to represent the mapping relationship satisfied between adjacent images; establishing a prior maximum likelihood estimation; iteratively obtaining the maximum a posteriori estimation parameters satisfied by the line feature matching pairs; establishing a uniform distribution probability model; screening the image line feature matching pairs that satisfy the probability threshold; searching for the remaining feature matching pairs, and iteratively completing and realizing the accurate and robust matching of the image line features. The present invention solves the problem of accurate and robust matching of image line features only through basic line feature descriptors, represents the mapping relationship between images through Fourier, combines the geometric relationship satisfied by the line feature matching pairs to establish a likelihood estimation model, and realizes the accurate and efficient matching of line features based on the iterative convergence of the EM algorithm.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and particularly to a line feature matching method based on Fourier representation and posterior probability estimation. Background Art

[0002] Image line features are a basic visual perception feature in machine vision in addition to point features. Like image point features, line features are widely used in photogrammetry, pose estimation in computer vision, 3D reconstruction, SLAM, and the field of industrial robots. Accurately and efficiently matching line features in images is a key link in a robot's understanding and perception of scene structure information.

[0003] However, most of the current existing image line feature matching is based on descriptor bidirectional matching and descriptor similarity constraints, or jointly uses the point feature RANSAC algorithm to calculate the homography matrix or fundamental matrix between images to achieve accurate matching and rejection of line features. The matching process is cumbersome with a large amount of calculation or depends on point feature information, and it is easy to cause "over-rejection" of line feature matching pairs. Summary of the Invention

[0004] The purpose of the present invention is to provide a line feature matching method based on Fourier representation and posterior probability estimation. This method represents the correspondence between two images based on Fourier series, without particularly relying on line feature descriptors for "hard distance screening", nor relying on the assistance of point features. This method has important application value in the fields of image processing, pose estimation, and SLAM.

[0005] The technical solution to achieve the purpose of the present invention is as follows: A line feature matching method based on Fourier representation and posterior probability estimation, comprising the following steps:

[0006] Step 1: The camera sensor collects a color image sequence in the actual scene, calls the line feature detection interface LSD algorithm in OpenCV to detect and extract the line features in the image, and then calls the OpenCV line feature descriptor interface to calculate and extract the binary descriptors of the line features;

[0007] Step 2: Based on the line feature binary descriptors, call the OpenCV line feature matching interface to perform initial matching of the line features to obtain the initial matching pair results; the initial matching of the line features refers to completely using the line feature binary descriptors and directly obtaining the line feature matching pairs using the OpenCV matching interface;

[0008] Step 3: Use the two-dimensional coordinate information of the line feature endpoints to calculate the image two-dimensional line feature expression equation; the mapping relationship satisfied between the corresponding line feature matching pairs on the image is represented by a compact Fourier series;

[0009] Step 4: Calculate the distances corresponding to the endpoints of the initial line feature matching pairs, and perform normalization data processing between [0, 1]; calculate the midpoint coordinates corresponding to the initial line feature matching pairs, and perform normalization data processing between [0, 1]. Based on the initial line feature matching pairs, establish a maximum likelihood a posteriori estimation model based on the distances from the line feature endpoints to the corresponding line matching pairs. Use the EM algorithm to iteratively solve the a posteriori probability estimation parameters, and perform inverse normalization data processing. Calculate the distance error corresponding to the line feature matching pairs, update the parameter variables in the a posteriori probability estimation model, and calculate the a posteriori probability of the endpoint distance error and the distance error of the midpoint coordinates of the line feature matching pairs

[0010] Step 5: Use the inverse-normalized line feature endpoint coordinates in Step 4 to calculate the angle between the line feature matching pairs:

[0011]

[0012] where (a, b, c) and (a′, b′, c′) are unit vectors, respectively representing the coefficients of the straight line equations in the image coordinate system of the line feature matching pairs; and regard this line feature angle result as a uniform distribution, and calculate the probability corresponding to each pair of line features With the ones in Step 4 and Calculate the final probability of the line feature matching pairs according to the probabilities of the three Finally, screen out the line feature matching pairs by the set threshold;

[0013] Step 6: Use the feature matching pairs obtained in Step 5 to search for the remaining line features, and perform line feature matching using the original binary feature descriptors. At the same time, remove the feature matching pairs in Step 5, and search for and obtain potential feature matching pairs. Normalize the potential feature matching pairs in this step, and jointly substitute them into the a posteriori probability estimation model established by the feature matching pairs in Step 5, and continuously perform iterative calculations until the number of feature matching pairs no longer increases and converges

[0014] A computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above method

[0015] A computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements the steps of the above method

[0016] Compared with the prior art, the significant advantages of the present invention are: the present invention does not rely on the image basic matrix or homography matrix, and only solves the accuracy and robustness problems of image line feature matching through basic line feature binary descriptors. The mapping relationship between the images to be matched is represented by Fourier, and a likelihood estimation model is established in combination with the geometric relationship satisfied by the line feature matching. Based on the EM algorithm, the line feature is accurately and efficiently matched through continuous progressive iterative convergence. This method has strong applicability to the field of image processing pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of a specific implementation scheme of the present invention.

[0018] Figure 2 This is an example diagram of extracting detected line features on an image and obtaining an initial line feature matching image based on a line feature descriptor according to the present invention.

[0019] Figure 3 The figure is an example of a line feature matching pair obtained based on the method of the present invention. DETAILED DESCRIPTION

[0020] The present invention proposes a feature matching process method based on binary descriptors of image line features and Bayesian a posteriori probability estimation. The method respectively includes the following steps: collecting a sequence of ordinary optical or aerial photography digital images, detecting and extracting line features on the image sequence pair based on the OpenCV (Open Source Computer Vision Library) open-source computer vision library using the LSD algorithm, and calculating the binary descriptors corresponding to the line features; comparing the similarity of the line feature descriptors based on the Hamming Distance to obtain the initial line feature matching pairs; introducing a compact Fourier series to characterize the mapping relationship satisfied between adjacent images; using the distances from the endpoints of the matching line features to the corresponding matching line features and the distance index between the midpoints of the matching line features to establish a prior maximum likelihood estimation respectively; using the EM (Expectation-Maximum) data mining algorithm to iteratively obtain the maximum a posteriori estimation parameters satisfied by the line feature matching pairs; during the iterative solution process of the line feature matching pairs, establishing a uniform distribution probability model according to the included angle between the corresponding line feature matching pairs after mapping; through the union statistical analysis of the maximum a posteriori estimation probabilities of three constraint variables, screening the image line feature matching pairs that satisfy the probability threshold by setting a certain threshold; finally, searching for the remaining feature matching pairs and using the above matching steps to continue the iteration to complete and realize the accurate and robust matching of the image line features. The present invention does not rely on the fundamental matrix or the homography matrix, and only uses the basic line feature descriptors to solve the problem of accurate and robust matching of image line features. By using the Fourier series to characterize the mapping relationship between images, combining the geometric relationship satisfied by the line feature matching pairs to establish a likelihood estimation model, and realizing the accurate and efficient matching of line features based on the iterative convergence of the EM algorithm, this method has strong applicability to the field of image processing pose estimation.

[0021] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Combined with Figure 1 , a line feature matching method based on Fourier representation and posterior probability estimation, first obtains the initial line feature matching pairs by means of binary line feature descriptors, and then comprehensively considers the geometric constraint relationship corresponding to the line feature matching, and iteratively solves the maximum a posteriori probability estimation corresponding to each feature matching pair. The method includes the following steps:

[0023] Step 1: The camera sensor collects a sequence of color images in the actual scene, calls the line feature detection interface LSD algorithm in OpenCV to detect and extract the line features in the images, and then calls the OpenCV line feature descriptor interface to calculate and extract the binary descriptors of the line features.

[0024] Step 2: Based on the line feature binary descriptor, call the OpenCV line feature matching interface to perform the initialization matching of line features, and obtain the initial matching pair results;

[0025] The initialization matching of line features means that completely using the line feature binary descriptor, directly obtaining the line feature matching pairs by using the OpenCV matching interface, rather than relying on other visual features to calculate the homography matrix or fundamental matrix to complete.

[0026] Step 3: Use the two-dimensional coordinate information of the line feature endpoints to calculate the two-dimensional line feature expression equation of the image (ax + by + c = O); in addition, the mapping relationship satisfied between the corresponding line feature matching pairs on the image is characterized by a compact Fourier series.

[0027] The compact Fourier representation means that the mapping relationship f(X) between the corresponding matching pairs of the image is approximately represented by the multiple regression of the Fourier series:

[0028]

[0029] where in formula (1), T represents the number of terms in the compact Fourier series, a n represents the coefficient corresponding to the term of the Fourier series, φ(X) is the cosine component in the Fourier series, and X is the set of endpoint coordinate vectors of the line feature.

[0030] Step 4: Calculate the distances corresponding to the endpoints of the initial line feature matching pairs, and perform [0, 1] normalized data processing; calculate the midpoint coordinates corresponding to the initial line feature matching pairs, and perform [0, 1] normalized data processing. Based on the initial line feature matching pairs, establish a maximum likelihood a posteriori estimation model based on the distance from the line feature endpoints to the corresponding line matching pairs, use the EM algorithm to iteratively solve the a posteriori probability estimation parameters, and perform anti-normalized data processing, calculate the distance error corresponding to the line feature matching pairs, update the parameter variables in the a posteriori probability estimation model, and calculate the a posteriori probability of the endpoint distance error and the distance error of the midpoint coordinates of the line feature matching pairs

[0031] The normalized data processing means that first calculate the distances between the two endpoints of the initialized line feature matching pairs (d s , d e ), first calculate the mean value of the endpoint distances, and calculate the scaling coefficient, so as to normalize the line feature endpoint distances in the [0, 1] interval; for the line feature midpoint distances, the normalization process is still to calculate the mean value from the coordinates of the two endpoints of the line segment, and also calculate the scaling coefficient, and then normalize the line feature endpoint distance values in the [0, 1] interval.

[0032] The maximum likelihood estimation model established based on the distance from the endpoint of the line feature to the corresponding line matching pair is to calculate the maximum likelihood estimation model of feature matching using the initial initialized line feature matching pairs and corresponding parameters:

[0033]

[0034] Among them, θ = {f, σ, γ} represents the corresponding unknown variables in the corresponding probability distribution, N is the total number of matching pairs, and z n is a hidden variable, z n = 1 indicates that this matching pair is a correct match, otherwise it is an abnormal match, γ is its corresponding probability, and the range is [0, 1], σ represents the standard deviation of the endpoint distance of the line feature matching pair, 1 / a is the uniform distribution under the abnormal line feature matching pair, f(*) is the Fourier representation coefficient of the line feature matching pair, x n represents the coordinates of the two endpoints of the line feature in the current image, y n represents the coordinates of the corresponding endpoints of the initial matching image, X represents the set of endpoint coordinate vectors of the line feature in the current image, Y represents the set of endpoint coordinate vectors of the line feature in the current image, and from the distance constraint from the point to the line segment, a mixed probability model can be obtained:

[0035]

[0036] Among them, D represents the data dimension.

[0037] The anti-normalization data processing refers to reverse-solving and calculating the endpoint coordinate information of the image line feature after mapping for the data between [0, 1] that was normalized before. Again, using the endpoint coordinate information of the mapped line, the line feature distance is calculated. Since after Fourier compact representation, the original endpoint coordinates X of the line feature become f(X), and its range is [0, 1]. At this time, in order to calculate the distance between the endpoints of the line feature after mapping, this value needs to be anti-normalized to obtain the endpoint coordinates of the line feature after Fourier representation, and according to the previous line feature equation, the endpoint coordinates of all line features are calculated;

[0038] Step 5: Use the anti-normalized feature point coordinates in Step 4 to calculate the angle Line ang between the line feature matching pairs, and the calculation formula is shown in (4):

[0039]

[0040] Among them, (a, b, c) and (a′, b′, c′) are unit vectors, which respectively represent the coefficients of the straight line equation on the image coordinate system of the line feature matching pair; and regard the result of this line feature angle as a uniform distribution, and calculate the probability corresponding to each pair of line features Having the in Step 4 Calculate the final probability of the line feature matching pair according to the probabilities of the three Finally, line feature matching pairs are screened out by the set threshold;

[0041] The included angle between the line feature matching pairs refers to, based on the initial matching result, after Fourier characterization, calculating the line feature coordinates through inverse normalization to obtain the characterized line feature coordinate quantities, then obtaining the corresponding line features through inverse normalization, calculating the straight line included angle from the endpoints of the line features obtained by inverse normalization and the corresponding line features, with the included angle ranging from (0° to 45°), uniformly distributed in this range, and calculating the angle probability of the corresponding line feature matching pairs;

[0042] The final probability mainly refers to the three probabilities calculated and obtained in steps 4 and 5. Since the posterior probability constrained by each line feature matching pair is at least greater than 0.9, and the direct multiplication of the three posterior probabilities is approximately 0.73, the threshold here is set to 0.7, so as to screen out the required line feature matching pairs;

[0043] Step 6: Use the feature matching pairs obtained in step 5 to search for the remaining line features, perform line feature matching using the original binary feature descriptor, and at the same time remove the feature matching pairs in step 5, search for and obtain potential feature matching pairs, perform normalization processing on the potential feature matching pairs in this step, combine and substitute them into the posterior probability estimation model established by the feature matching pairs in step 5, and continuously perform iterative calculations until the number of feature matching pairs no longer increases and converges.

[0044] The potential feature matching pairs refer to that the feature matching obtained in step 5 is based on the initial line feature matching pairs, and some correct matching pairs may be screened out. Therefore, it is necessary to first screen out the line feature matching pairs in step 5, then perform feature matching again based on the line feature descriptor, and screen the remaining feature matching pairs according to the threshold distance of the feature descriptor, so as to search for and obtain potential line feature matching pairs.

[0045] Embodiment

[0046] The following briefly introduces the specific implementation process of the present invention in conjunction with the accompanying drawings:

[0047] (1) In this embodiment, image pairs in different situations (conventional indoor scenes, rotated images) are selected, line features are respectively detected and extracted, the corresponding line feature descriptors are calculated, line feature matching is performed according to the Hamming distance of the feature descriptors, and initial line feature matching pairs are obtained;

[0048] (2) Select the initial line feature matching pairs, calculate the straight line equations of the image line features (ax + by + c = 0) according to the endpoint coordinates of the line features, and calculate the corresponding distances of the line segment endpoints according to the endpoint coordinates of the line features (d s , d e) and perform a normalization operation on it to the interval [0, 1];

[0049] (3) Initialize a in formula (1) n The coefficient obtains the Fourier representation f between the image mapping relationships, and calculates the initial variance in the probability distribution of formula (3);

[0050] (4) Substitute into the probability estimation model formula (3) to iteratively solve the Fourier representation coefficient relationship, and calculate the probability values P corresponding to the mathematical relationships such as the endpoint distance, the distance between the midpoints of the line features, and the angle i , calculate the total probability value, continuously iteratively update the Fourier representation coefficient f, and screen and obtain the line feature matching pairs;

[0051] (5) Remove the abnormal line feature matching pairs obtained in step 4 from the initial matching pairs, search for the remaining potential line feature matching pairs, and add them to the posterior probability estimation model in step 4, and iterate until the number of feature matching pairs no longer increases, and finally obtain the final line feature matching pairs, as Figures 2 - 3 shown, Figure 2 In the two figures in, the number of detected line features is 70 and 63 respectively. The present invention can obtain 52 line feature matching pairs, and the matching accuracy rate is 100%; Figure 3 In the scene, the number of detected line features in both figures is 34. The present invention can obtain 26 line feature matching pairs, and the matching accuracy rate is 100%.

Claims

1. A line feature matching method based on Fourier representation and posterior probability estimation, characterized in that: The steps include: Step 1: The camera sensor collects a color image sequence in the actual scene, calls the line feature detection interface LSD algorithm in OpenCV to detect and extract line features in the image, and then calls the OpenCV line feature descriptor interface to calculate and extract the binary descriptor of the line feature; Step 2: Based on the line feature binary descriptor, call the OpenCV line feature matching interface to perform initial matching of line features to obtain initial matching pair results; the line feature initial matching refers to completely using the line feature binary descriptor and using the OpenCV matching interface to directly obtain line feature matching pairs; Step 3: Calculate the two-dimensional line feature expression equation of the image using the two-dimensional coordinate information of the line feature endpoints; the mapping relationship satisfied between the corresponding line feature matching pairs on the image is represented by a compact Fourier series; Step 4: Calculate the distance between the initial line feature matching endpoints and perform normalized data processing between [0, 1]; Calculate the midpoint coordinates of the initial line feature matching pair, and perform [0,1] normalized data processing. Based on the initial line feature matching pair, establish a maximum likelihood posterior estimation model based on the distance from the line feature endpoint to the corresponding line matching pair. Use the EM algorithm to iteratively solve the posterior probability estimation parameters, and perform denormalized data processing. Calculate the distance error corresponding to the line feature matching pair, update the parameter variables in the posterior probability estimation model, and calculate the posterior probability of the endpoint distance error. And the coordinate distance error of the midpoint of line feature matching Step 5: Use the line feature endpoint coordinates denormalized in step 4 to calculate the angle between the matching pairs of line features: Where (a, b, c) and (a', b', c') are unit vectors, representing the coefficients of the line equation on the image coordinate system of the line feature matching pair respectively; and the line feature angle result is regarded as a uniform distribution, and the probability corresponding to each pair of line features is calculated will have the and Calculate the final probability of the line feature matching pair based on the three probabilities The line feature matching pairs are finally screened out by the set threshold; Step 6: Use the feature matching pairs obtained in step 5 to search for the remaining line features, and use the original binary feature descriptors to perform line feature matching. At the same time, remove the feature matching pairs in step 5, and search for potential feature matching pairs. Normalize the potential feature matching pairs in this step, combine and substitute them into the posterior probability estimation model established by the feature matching pairs in step 5, and continuously iterate the calculation until the number of feature matching pairs no longer increases and converges.

2. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The characterization using a compact Fourier series refers to using Fourier series multinomial regression to approximate the mapping relationship f(X) between corresponding matching pairs of images: In formula (1), T represents the number of terms in the compact Fourier series, a n It represents the coefficient corresponding to the Fourier series term, φ(X) is the cosine component in the Fourier series, and X is the set of endpoint coordinate vectors of the line feature.

3. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The normalized data processing refers to first calculating the distance between the two endpoints of the initialization line feature matching pair (d s ,d e ), first calculate the mean of the endpoint distances and calculate the scaling factor, so as to normalize the line feature endpoint distances to the interval [0,1]; for the line feature midpoint distance, the normalization process is still to calculate the mean of the coordinates of the two end points of the line segment, and similarly calculate the scaling factor, and then normalize the line feature endpoint distance value to the interval [0,1].

4. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The maximum likelihood estimation model is established based on the distance from the line feature endpoint to the corresponding line matching pair, and the feature matching maximum likelihood estimation model is calculated using the initialization line feature matching pair and the corresponding parameters: Among them, θ = {f, σ, γ} represents the corresponding unknown variable in the corresponding probability distribution, N is the total number of matching pairs, z n is a hidden variable, z n =1 indicates that the matching pair is a correct match, otherwise it is an abnormal match, γ is its corresponding probability, ranging from [0,1], σ represents the standard deviation of the end point distance of the line feature matching pair, 1 / a is the uniform distribution under the abnormal line feature matching pair, f(*) is the Fourier characterization coefficient of the line feature matching pair, x n Represents the coordinates of the two endpoints of the current image line feature, y n represents the endpoint coordinates corresponding to the initial matching image, X represents the current image line feature endpoint coordinate vector set, Y represents the current image line feature endpoint coordinate vector set, and the mixed probability model can be obtained by the distance constraint from the point to the line segment: Where D represents the data dimension.

5. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The denormalized data processing refers to the reverse solution of the previously normalized [0,1] data to calculate the image line feature endpoint coordinate information after mapping, and the mapped line endpoint coordinate information is used again to calculate the line feature distance; since the original line feature endpoint coordinate X becomes f(X) after Fourier compact representation, its range is [0,1], at this time, in order to calculate the distance of the line feature endpoint after mapping, it is necessary to denormalize this value to obtain the line feature endpoint coordinate after Fourier representation, and calculate all line feature endpoint coordinates according to the previous line feature equation.

6. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The angle between the line feature matching pairs refers to the calculation of the line feature coordinates based on the initial matching results, after Fourier characterization, the line feature coordinates are obtained by denormalization, and then the corresponding line features are obtained by denormalization. The straight line angle is calculated from the line feature endpoints obtained by denormalization and the corresponding line features. The angle ranges from 0° to 45°, and this is used as a uniform distribution to calculate the corresponding line feature matching pair angle probability.

7. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The final probability Mainly the three probabilities calculated in step 4 and step 5. Since the posterior probability of each line feature matching pair is constrained to be at least greater than 0.9, the final total threshold is set to 0.7 to screen out the required line feature matching pairs.

8. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The potential feature matching pair means that the feature matching obtained in step 5 is based on the initial line feature matching pair. First, the line feature matching pair in step 5 is screened out, and then feature matching is performed again based on the line feature descriptor, and the remaining feature matching pairs are screened according to the feature descriptor threshold distance, so as to search and obtain potential line feature matching pairs.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images