Line feature matching method based on Fourier representation and posterior probability estimation

Abstract

The invention discloses a line feature matching method based on Fourier representation and posterior probability estimation, and the method comprises the steps: detecting and extracting line features on an image sequence pair, and calculating a binary descriptor; obtaining an initial line feature matching pair; a compact Fourier series is introduced to represent a mapping relation satisfied between adjacent images; prior maximum likelihood estimation is established; iteratively solving a maximum post-test estimation parameter satisfied by the line feature matching pair; establishing a uniform distribution probability model; screening image line feature matching pairs meeting a probability threshold value; and searching residual feature matching pairs, completing iteration and realizing accurate and steady matching of image line features. According to the method, the problem of accurate and robust matching of the image line features is solved only through the basic line feature descriptors, the mapping relation between the images is represented through Fourier, the likelihood estimation model is established in combination with the geometrical relation met by line feature matching, and accurate and efficient matching of the line features is achieved based on EM algorithm iterative convergence.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a line feature matching method based on Fourier characterization and posterior probability estimation. Background Art

[0002] Image line features are a basic visual perception feature in machine vision in addition to point features. Like image point features, line features are widely used in photogrammetry, posture estimation in computer vision, 3D reconstruction, SLAM and industrial robotics. Accurately and efficiently matching line features on images is a key link in robots' understanding and perception of scene structure information.

[0003] However, most of the existing image line feature matching is based on descriptor bidirectional matching and descriptor similarity constraints, or the point feature RANSAC algorithm is used to calculate the homography matrix or basic matrix between images to achieve accurate matching and elimination of line features. The matching process is cumbersome and computationally intensive or relies on point feature information, and it is easy to cause "over-elimination" of line feature matching pairs. Summary of the invention

[0004] The purpose of the present invention is to provide a line feature matching method based on Fourier characterization and posterior probability estimation. The method characterizes the correspondence between two images based on the Fourier series, does not need to rely on line feature descriptors for "hard distance screening", and does not need to rely on the assistance of point features. The method has important application value in the fields of image processing, pose estimation and SLAM.

[0005] The technical solution to achieve the purpose of the present invention is: a line feature matching method based on Fourier characterization and posterior probability estimation, comprising the following steps:

[0006] Step 1: The camera sensor collects a color image sequence in the actual scene, calls the line feature detection interface LSD algorithm in OpenCV to detect and extract line features in the image, and then calls the OpenCV line feature descriptor interface to calculate and extract the binary descriptor of the line feature;

[0007] Step 2: Based on the line feature binary descriptor, call the OpenCV line feature matching interface to perform initial matching of line features to obtain initial matching pair results; the line feature initial matching refers to completely using the line feature binary descriptor and using the OpenCV matching interface to directly obtain line feature matching pairs;

[0008] Step 3: Calculate the two-dimensional line feature expression equation of the image using the two-dimensional coordinate information of the line feature endpoints; the mapping relationship satisfied between the corresponding line feature matching pairs on the image is represented by a compact Fourier series;

[0009] Step 4: Calculate the distances corresponding to the endpoints of the initial line feature matching pair, and perform normalized data processing between [0, 1]; calculate the midpoint coordinates corresponding to the initial line feature matching pair, and perform normalized data processing between [0, 1]; establish a maximum likelihood posterior estimation model based on the distances from the line feature endpoints to the corresponding line matching pairs based on the initial line feature matching pairs, use the EM algorithm to iteratively solve the posterior probability estimation parameters, perform denormalized data processing, calculate the distance error corresponding to the line feature matching pair, update the parameter variables in the posterior probability estimation model, and calculate the posterior probability of the endpoint distance error. And the coordinate distance error of the midpoint of line feature matching

[0010] Step 5: Use the line feature endpoint coordinates denormalized in step 4 to calculate the angle between the matching pairs of line features:

[0011]

[0012] Where (a, b, c) and (a′, b′, c′) are unit vectors, representing the coefficients of the line equation on the image coordinate system of the line feature matching pair respectively; and the line feature angle result is regarded as a uniform distribution, and the probability corresponding to each pair of line features is calculated will have the and Calculate the final probability of the line feature matching pair based on the three probabilities The line feature matching pairs are finally screened out by the set threshold;

[0013] Step 6: Use the feature matching pairs obtained in step 5 to search for the remaining line features, and use the original binary feature descriptors to perform line feature matching. At the same time, remove the feature matching pairs in step 5, and search for potential feature matching pairs. Normalize the potential feature matching pairs in this step, combine and substitute them into the posterior probability estimation model established by the feature matching pairs in step 5, and continuously iterate the calculation until the number of feature matching pairs no longer increases and converges.

[0014] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0015] A computer-readable storage medium stores a computer program, which implements the steps of the above method when executed by a processor.

[0016] Compared with the prior art, the significant advantages of the present invention are: the present invention does not rely on the image basic matrix or homography matrix, and only solves the accuracy and robustness problems of image line feature matching through basic line feature binary descriptors. The mapping relationship between the images to be matched is represented by Fourier, and a likelihood estimation model is established in combination with the geometric relationship satisfied by the line feature matching. Based on the EM algorithm, the line feature is accurately and efficiently matched through continuous progressive iterative convergence. This method has strong applicability to the field of image processing pose estimation. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a schematic diagram of a specific implementation scheme of the present invention.

[0018] Figure 2 This is an example diagram of extracting detected line features on an image and obtaining an initial line feature matching image based on a line feature descriptor according to the present invention.

[0019] Figure 3 The figure is an example of a line feature matching pair obtained based on the method of the present invention. DETAILED DESCRIPTION

[0020] The present invention proposes a feature matching process method based on image line feature binary descriptor and Bayesian posterior probability estimation, which comprises the following steps: collecting ordinary optical or aerial photography digital image sequences, based on OpenCV (Open Source Computer Vision) The paper uses the open-source computer vision library of the ImageNet Library to detect and extract line features on image sequence pairs, and calculates the binary descriptors corresponding to the line features. The paper compares the similarity of line feature descriptors based on the Hamming distance to obtain the initial line feature matching pairs. The paper introduces a compact Fourier series to characterize the mapping relationship between adjacent images. The paper uses the distance from the end point of the matching pair line feature to the corresponding matching line feature and the distance between the midpoints of the matching pair line features to establish the prior maximum likelihood estimation. The paper uses the EM (Expectation-Maximum) data mining algorithm to iteratively obtain the maximum a posteriori estimation parameters satisfied by the line feature matching pairs. During the iterative solution of the line feature matching pairs, the paper establishes a uniform distribution probability model according to the angle between the corresponding line feature matching pairs after mapping. The paper selects the image line feature matching pairs that meet the probability threshold through the statistical analysis of the union of the maximum a posteriori estimation probabilities of the three constraint variables and by setting a certain threshold. Finally, the paper searches for the remaining feature matching pairs, and uses the above matching steps to continue the iteration to complete and achieve accurate and robust matching of the image line features. The present invention does not rely on the basic matrix or the homography matrix, but solves the problem of accurate and robust matching of image line features only through the basic line feature descriptor. The mapping relationship between images is characterized by Fourier, and a likelihood estimation model is established in combination with the geometric relationship satisfied by the line feature matching. Based on the iterative convergence of the EM algorithm, accurate and efficient matching of line features is achieved. This method has strong applicability to the field of image processing pose estimation.

[0021] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.

[0022] Combination Figure 1 A line feature matching method based on Fourier representation and posterior probability estimation is firstly used to obtain an initial line feature matching pair by means of a binary line feature descriptor, and then the geometric constraint relationship corresponding to the line feature matching is integrated to iteratively solve the maximum posterior probability estimation corresponding to each feature matching pair. The method comprises the following steps:

[0023] Step 1: The camera sensor collects a color image sequence in the actual scene, calls the line feature detection interface LSD algorithm in OpenCV to detect and extract line features in the image, and then calls the OpenCV line feature descriptor interface to calculate and extract the binary descriptor of the line feature;

[0024] Step 2: Based on the line feature binary descriptor, call the OpenCV line feature matching interface to perform initial matching of line features to obtain the initial matching pair results;

[0025] The line feature initialization matching refers to completely using the line feature binary descriptor and using the OpenCV matching interface to directly obtain the line feature matching pairs, instead of using other visual features to calculate the homography matrix or the basic matrix.

[0026] Step 3: Use the two-dimensional coordinate information of the line feature endpoints to calculate the two-dimensional line feature expression equation of the image (ax+by+c=O); in addition, the mapping relationship between the corresponding line feature matching pairs on the image is represented by a compact Fourier series.

[0027] The compact Fourier representation refers to using Fourier series multinomial regression to approximate the mapping relationship f(X) between corresponding matching pairs of images:

[0028]

[0029] In formula (1), T represents the number of terms in the compact Fourier series, a n It represents the coefficient corresponding to the Fourier series term, φ(X) is the cosine component in the Fourier series, and X is the set of endpoint coordinate vectors of the line feature.

[0030] Step 4: Calculate the distances corresponding to the endpoints of the initial line feature matching pair, and perform [O, 1] normalized data processing; calculate the midpoint coordinates corresponding to the initial line feature matching pair, and perform [O, 1] normalized data processing, establish a maximum likelihood posterior estimation model based on the distances from the line feature endpoints to the corresponding line matching pairs based on the initial line feature matching pair, use the EM algorithm to iteratively solve the posterior probability estimation parameters, perform denormalized data processing, calculate the distance error corresponding to the line feature matching pair, update the parameter variables in the posterior probability estimation model, and calculate the posterior probability of the endpoint distance error. And the coordinate distance error of the midpoint of line feature matching

[0031] The normalized data processing refers to first calculating the distance between the two endpoints of the initialization line feature matching pair (d s , d e ), first calculate the mean of the endpoint distances, and calculate the scaling factor, so as to normalize the line feature endpoint distances to the interval [O, 1]; for the line feature midpoint distance, the normalization process is still to calculate the mean of the coordinates of the two end points of the line segment, and similarly calculate the scaling factor, and then normalize the line feature endpoint distance value to the interval [O, 1].

[0032] The maximum likelihood estimation model is established based on the distance from the line feature endpoint to the corresponding line matching pair, and the feature matching maximum likelihood estimation model is calculated using the initialization line feature matching pair and the corresponding parameters:

[0033]

[0034] Where θ = {f, σ, γ} represents the corresponding unknown variable in the corresponding probability distribution, N is the total number of matching pairs, z n is a hidden variable, z n =1 indicates that the matching pair is a correct match, otherwise it is an abnormal match, γ is its corresponding probability, the range is [O, 1], σ represents the standard deviation of the end point distance of the line feature matching pair, 1 / a is the uniform distribution under the abnormal line feature matching pair, f(*) is the Fourier characterization coefficient of the line feature matching pair, x n Represents the coordinates of the two endpoints of the current image line feature, y n represents the endpoint coordinates corresponding to the initial matching image, X represents the current image line feature endpoint coordinate vector set, Y represents the current image line feature endpoint coordinate vector set, and the mixed probability model can be obtained by the distance constraint from the point to the line segment:

[0035]

[0036] Where D represents the data dimension.

[0037] The denormalized data processing refers to the reverse solution of the previously normalized [O, 1] data to calculate the image line feature endpoint coordinate information after mapping, and the line feature distance is calculated again using the mapped line endpoint coordinate information. Since the original line feature endpoint coordinate X becomes f(X) after Fourier compact representation, its range is [O, 1]. At this time, in order to calculate the distance of the line feature endpoint after mapping, it is necessary to denormalize this value to obtain the line feature endpoint coordinate after Fourier representation, and calculate all line feature endpoint coordinates according to the previous line feature equation;

[0038] Step 5: Use the denormalized feature point coordinates in step 4 to calculate the angle Line between the line feature matching pairs ang , the calculation formula is shown in (4):

[0039]

[0040] Where (a, b, c) and (a′, b′, c′) are unit vectors, representing the coefficients of the line equation on the image coordinate system of the line feature matching pair respectively; and the line feature angle result is regarded as a uniform distribution, and the probability corresponding to each pair of line features is calculated will have the and Calculate the final probability of the line feature matching pair based on the three probabilities The line feature matching pairs are finally screened out by the set threshold;

[0041] The angle between the line feature matching pairs refers to the calculation of the line feature coordinates by denormalization based on the initial matching results after Fourier characterization, obtaining the line feature coordinates after characterization, and then obtaining the corresponding line features by denormalization. The straight line angle is calculated from the line feature endpoints obtained by denormalization and the corresponding line features. The value range of the angle is (0°~45°), which is uniformly distributed to calculate the corresponding line feature matching pair angle probability;

[0042] The final probability Mainly the three probabilities calculated in step 4 and step 5. Since the posterior probability of each line feature matching pair is at least greater than 0.9, the direct multiplication of the three posterior probabilities is about 0.73, so we set the threshold here to 0.7 to filter out the required line feature matching pairs;

[0043] Step 6: Use the feature matching pairs obtained in step 5 to search for the remaining line features, and use the original binary feature descriptors to perform line feature matching. At the same time, remove the feature matching pairs in step 5, and search for potential feature matching pairs. Normalize the potential feature matching pairs in this step, combine and substitute them into the posterior probability estimation model established by the feature matching pairs in step 5, and continuously iterate the calculation until the number of feature matching pairs no longer increases and converges.

[0044] The potential feature matching pairs refer to the feature matching obtained in step 5, which is based on the initial line feature matching pairs. Some correct matching pairs may be screened and filtered. Therefore, it is necessary to first screen out the line feature matching pairs in step 5, and then perform feature matching again based on the line feature descriptor, and screen the remaining feature matching pairs according to the feature descriptor threshold distance, so as to search for potential line feature matching pairs.

[0045] Example

[0046] The specific implementation process of the present invention is briefly described below in conjunction with the accompanying drawings:

[0047] (1) This embodiment selects image pairs in different situations (conventional indoor scenes, rotated images), detects and extracts line features respectively, calculates corresponding line feature descriptors, performs line feature matching based on the Hamming distance of the feature descriptors, and obtains initial matching pairs of line features;

[0048] (2) Select the initial line feature matching pair, calculate the image line feature straight line equation (ax+by+c=0) according to the line feature endpoint coordinates, and calculate the corresponding distance of the line segment endpoints (d s ,d e), and normalize it to the interval [0,1];

[0049] (3) Initialize a in formula (1) n The coefficients are used to obtain the Fourier representation f between the image mapping relationship, and the initial variance in the probability distribution of formula (3) is calculated;

[0050] (4) Substitute into the probability estimation model formula (3) and iteratively solve the Fourier representation coefficient relationship to calculate the probability value P corresponding to the mathematical relationship such as endpoint distance, line feature midpoint distance and angle. i , calculate the total probability value, continuously iteratively update the Fourier representation coefficient f, screen and obtain line feature matching pairs;

[0051] (5) Remove the abnormal line feature matching pairs obtained in step 4 from the initial matching pairs, search for the remaining potential line feature matching pairs, and add them to the posterior probability estimation model in step 4, iterate until the number of feature matching pairs no longer increases, and finally obtain the final line feature matching pairs, as shown in Figure 2-Figure 3 As shown, Figure 2 The number of line feature detections in the two images is 70 and 63 respectively. The present invention can obtain 52 pairs of line feature matching pairs, and the matching accuracy is 100%; Figure 3 The number of line feature detections in the two images in the scene is 34. The present invention can obtain 26 pairs of line feature matching pairs, and the matching accuracy rate is 100%.

Claims

1. A line feature matching method based on Fourier representation and posterior probability estimation, characterized in that: The steps include: Step 1: The camera sensor collects a color image sequence in the actual scene, calls the line feature detection interface LSD algorithm in OpenCV to detect and extract line features in the image, and then calls the OpenCV line feature descriptor interface to calculate and extract the binary descriptor of the line feature; Step 2: Based on the line feature binary descriptor, call the OpenCV line feature matching interface to perform initial matching of line features to obtain initial matching pair results; the line feature initial matching refers to completely using the line feature binary descriptor and using the OpenCV matching interface to directly obtain line feature matching pairs; Step 3: Calculate the two-dimensional line feature expression equation of the image using the two-dimensional coordinate information of the line feature endpoints; the mapping relationship satisfied between the corresponding line feature matching pairs on the image is represented by a compact Fourier series; Step 4: Calculate the distance between the initial line feature matching endpoints and perform normalized data processing between [0, 1]; Calculate the midpoint coordinates of the initial line feature matching pair, and perform [0,1] normalized data processing. Based on the initial line feature matching pair, establish a maximum likelihood posterior estimation model based on the distance from the line feature endpoint to the corresponding line matching pair. Use the EM algorithm to iteratively solve the posterior probability estimation parameters, and perform denormalized data processing. Calculate the distance error corresponding to the line feature matching pair, update the parameter variables in the posterior probability estimation model, and calculate the posterior probability of the endpoint distance error. And the coordinate distance error of the midpoint of line feature matching Step 5: Use the line feature endpoint coordinates denormalized in step 4 to calculate the angle between the matching pairs of line features: Where (a, b, c) and (a', b', c') are unit vectors, representing the coefficients of the line equation on the image coordinate system of the line feature matching pair respectively; and the line feature angle result is regarded as a uniform distribution, and the probability corresponding to each pair of line features is calculated will have the and Calculate the final probability of the line feature matching pair based on the three probabilities The line feature matching pairs are finally screened out by the set threshold; Step 6: Use the feature matching pairs obtained in step 5 to search for the remaining line features, and use the original binary feature descriptors to perform line feature matching. At the same time, remove the feature matching pairs in step 5, and search for potential feature matching pairs. Normalize the potential feature matching pairs in this step, combine and substitute them into the posterior probability estimation model established by the feature matching pairs in step 5, and continuously iterate the calculation until the number of feature matching pairs no longer increases and converges.

2. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The characterization using a compact Fourier series refers to using Fourier series multinomial regression to approximate the mapping relationship f(X) between corresponding matching pairs of images: In formula (1), T represents the number of terms in the compact Fourier series, a n It represents the coefficient corresponding to the Fourier series term, φ(X) is the cosine component in the Fourier series, and X is the set of endpoint coordinate vectors of the line feature.

3. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The normalized data processing refers to first calculating the distance between the two endpoints of the initialization line feature matching pair (d s ,d e ), first calculate the mean of the endpoint distances and calculate the scaling factor, so as to normalize the line feature endpoint distances to the interval [0,1]; for the line feature midpoint distance, the normalization process is still to calculate the mean of the coordinates of the two end points of the line segment, and similarly calculate the scaling factor, and then normalize the line feature endpoint distance value to the interval [0,1].

4. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The maximum likelihood estimation model is established based on the distance from the line feature endpoint to the corresponding line matching pair, and the feature matching maximum likelihood estimation model is calculated using the initialization line feature matching pair and the corresponding parameters: Among them, θ = {f, σ, γ} represents the corresponding unknown variable in the corresponding probability distribution, N is the total number of matching pairs, z n is a hidden variable, z n =1 indicates that the matching pair is a correct match, otherwise it is an abnormal match, γ is its corresponding probability, ranging from [0,1], σ represents the standard deviation of the end point distance of the line feature matching pair, 1 / a is the uniform distribution under the abnormal line feature matching pair, f(*) is the Fourier characterization coefficient of the line feature matching pair, x n Represents the coordinates of the two endpoints of the current image line feature, y n represents the endpoint coordinates corresponding to the initial matching image, X represents the current image line feature endpoint coordinate vector set, Y represents the current image line feature endpoint coordinate vector set, and the mixed probability model can be obtained by the distance constraint from the point to the line segment: Where D represents the data dimension.

5. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The denormalized data processing refers to the reverse solution of the previously normalized [0,1] data to calculate the image line feature endpoint coordinate information after mapping, and the mapped line endpoint coordinate information is used again to calculate the line feature distance; since the original line feature endpoint coordinate X becomes f(X) after Fourier compact representation, its range is [0,1], at this time, in order to calculate the distance of the line feature endpoint after mapping, it is necessary to denormalize this value to obtain the line feature endpoint coordinate after Fourier representation, and calculate all line feature endpoint coordinates according to the previous line feature equation.

6. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The angle between the line feature matching pairs refers to the calculation of the line feature coordinates based on the initial matching results, after Fourier characterization, the line feature coordinates are obtained by denormalization, and then the corresponding line features are obtained by denormalization. The straight line angle is calculated from the line feature endpoints obtained by denormalization and the corresponding line features. The angle ranges from 0° to 45°, and this is used as a uniform distribution to calculate the corresponding line feature matching pair angle probability.

7. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The final probability Mainly the three probabilities calculated in step 4 and step 5. Since the posterior probability of each line feature matching pair is constrained to be at least greater than 0.9, the final total threshold is set to 0.7 to screen out the required line feature matching pairs.

8. The line feature matching method based on Fourier characterization and posterior probability estimation according to claim 1, characterized in that: The potential feature matching pair means that the feature matching obtained in step 5 is based on the initial line feature matching pair. First, the line feature matching pair in step 5 is screened out, and then feature matching is performed again based on the line feature descriptor, and the remaining feature matching pairs are screened according to the feature descriptor threshold distance, so as to search and obtain potential line feature matching pairs.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images