Dual-aircraft Relative Positioning Method Based on Visual Feature Point Matching

Abstract

The present invention discloses a method for dual-aircraft relative positioning based on visual feature point matching, which includes obtaining a sequence of dual-aircraft image frames and the relative pose at the corresponding moment of each frame of image, and constructing a database of image relative pose sequences; obtaining real-time dual-aircraft images, extracting feature points, and determining the feature points of the real-time dual-aircraft images; performing feature point matching between the real-time dual-aircraft images and the database of image relative pose sequences, determining the matching feature point pairs, and eliminating the mismatched point pairs; determining the translation and rotation vectors between two frames of images according to the matching feature point pairs, and determining a frame of image in the database of image relative pose sequences with the smallest difference from the real-time dual-aircraft images according to the translation and rotation vectors; using the relative pose of the dual aircraft corresponding to a frame of image determined in the database of image relative pose sequences as the dual-aircraft pose at the current moment of the real-time dual-aircraft images. The present invention can improve the adaptability, intelligence, and concealment of dual-aircraft pose estimation.

Description

Technical Field

[0001] The present invention relates to the technical field of machine vision, and particularly relates to a method for dual-aircraft relative positioning in the air based on visual feature point matching. Background Art

[0002] With the increasing complexity of the scale of modern air missions and the flight time in the air, air refueling technology plays an increasingly important role. To ensure the safety and efficiency of this mission, it is necessary to obtain the accurate relative pose between the tanker and the receiver in a timely manner, avoiding the situation where refueling cannot be carried out due to the excessive distance between the two, or collisions occurring due to the too-close distance between the two. Although the Global Navigation Satellite System (GNSS) can provide drift-free absolute positioning information, in the context of modern warfare, the reception of these active information is facing more and more challenges. Moreover, in complex terrain and electromagnetic environments, satellite positioning signals are vulnerable to interference, making it difficult to achieve robust and accurate positioning. Although the inertial navigation system does not rely on external information and has high autonomy, the positioning information it provides is relative to the starting position, and there is cumulative drift during long-term operation. Summary of the Invention

[0003] In view of the above deficiencies in the prior art, the present invention provides a method for dual-aircraft relative positioning in the air based on visual feature point matching.

[0004] In order to achieve the above-mentioned invention purpose, the technical solution adopted by the present invention is as follows:

[0005] A method for dual-aircraft relative positioning in the air based on visual feature point matching, comprising the following steps:

[0006] S1. Obtain the image frame sequences of the two aircraft and the relative pose at the corresponding moment of each frame of the image, and construct an image relative pose sequence database;

[0007] S2. Obtain real-time dual-aircraft images, perform feature point extraction, and determine the feature points of the real-time dual-aircraft images;

[0008] S3. Perform feature point matching between the real-time dual-aircraft images and the image relative pose sequence database, determine the matching feature point pairs, and eliminate the mis-matched point pairs;

[0009] S4. Determine the translation and rotation vectors between two frames of images according to the matching feature point pairs, and determine the frame of image in the image relative pose sequence database with the smallest difference from the real-time dual-aircraft images according to the translation and rotation vectors;

[0010] S5. Use the relative pose of the two aircraft corresponding to a frame of image determined in the image relative pose sequence database as the relative pose of the two aircraft at the current moment of the real-time dual-aircraft images.

[0011] Further, in step S2, feature points are extracted from the real-time dual-camera images to determine the feature points of the real-time dual-camera images, which specifically includes the following steps:

[0012] S21. Determine key points according to the change of pixel gray values in the real-time dual-camera images;

[0013] S22. Construct an image pyramid based on the real-time dual-camera images, and extract key points at each layer of the image pyramid to construct a feature descriptor of the key points, so as to obtain the feature points of the real-time dual-camera images.

[0014] Further, step S21 specifically includes the following steps:

[0015] S211. Select a pixel point in the real-time dual-camera images as a to-be-determined key point in turn, and obtain the pixel gray value of the to-be-determined key point;

[0016] S212. Set a pixel gray value threshold for the to-be-determined key point, and compare the pixel gray value of the to-be-determined key point with the gray values of all pixel points within a set radius centered on the to-be-determined key point;

[0017] If there are a continuous set number of pixel gray values greater than the sum of the pixel gray value of the to-be-determined key point and the pixel gray value threshold, or less than the difference between the pixel gray value of the to-be-determined key point and the pixel gray value threshold, then the to-be-determined key point is determined as a key point;

[0018] S213. Traverse all pixel points in the real-time dual-camera images to obtain all key points in the real-time dual-camera images.

[0019] Further, in step S3, the real-time dual-camera images are matched with the image relative pose sequence database to determine the matching feature point pairs, which specifically includes the following steps:

[0020] S31. Randomly select multiple pairs of matching feature point pairs and calculate the homography matrix;

[0021] S32. According to the calculated homography matrix, project the feature points with matching relationships in the first frame image to the second frame image and calculate the reprojection coordinates;

[0022] S33. Calculate the distance between the reprojection coordinates and the coordinates of the already-matched feature points;

[0023] S34. Determine whether the calculated distance is less than the set distance threshold. If so, it is determined as a correct matching feature point pair; otherwise, it is determined as an incorrect matching feature point pair, and record the number of correct matching feature point pairs;

[0024] S35. Repeat steps S31 to S34 according to the set number of times, compare the number of correctly matched feature point pairs counted after multiple cycles, and take the case with the largest number of correctly matched feature point pairs as the final result.

[0025] Further, the specific steps of removing mismatched point pairs from the matched feature point pairs in step S3 are as follows:

[0026] S36. Select the smallest data set from which a model can be estimated;

[0027] S37. Calculate the data model using the smallest data set;

[0028] S38. Substitute all the data into the data model and calculate the number of inliers;

[0029] S39. Compare the number of inliers of the current model and the optimal model obtained previously, and record the model parameters and the number of inliers with the largest number of inliers;

[0030] S310. Repeat steps S36 to S39 until the maximum number of iterations is reached or the number of inliers of the current model reaches the set number threshold.

[0031] Further, the specific steps of determining the translation and rotation vectors between two frames of images according to the matched feature point pairs in step S4 are as follows:

[0032] S41. Set the sum of the squares of the Euclidean distances between the matched feature point pairs as the objective function, expressed as:

[0033]

[0034] where p i , p′ i are the feature point coordinates of the i-th pair of matched feature point pairs, p, p′ are the centroid coordinates of the i-th pair of matched feature point pairs, R is the rotation vector, t is the translation vector, and n is the number of pairs of matched feature point pairs;

[0035] S42. Construct a relationship matrix of the matched feature point pairs according to the translation and rotation vectors between two frames of images, expressed as:

[0036]

[0037] where T is the transpose symbol;

[0038] S43. Perform singular value decomposition on the relationship matrix of the matched feature point pairs to obtain the rotation vector, expressed as:

[0039] R = UV T

[0040] where UV is the diagonal matrix of the value decomposition;

[0041] S44. Calculate the translation vector based on the rotation vector, expressed as:

[0042] t = p - Rp'.

[0043] Furthermore, in step S4, to determine the frame of image in the image relative pose sequence database with the smallest difference from the real-time dual-camera images based on the translation and rotation vectors, the specific steps are as follows:

[0044] S45. Construct a rotation matrix difference matrix based on the rotation vector, expressed as:

[0045] E = R - I 2×2

[0046] where R is the rotation vector and I 2×2 is the identity matrix;

[0047] S46. Calculate the L1 norm of the rotation matrix difference matrix, expressed as:

[0048] m1 = norm1{E}

[0049] where norm1 represents the L1 norm function;

[0050] S47. Construct a translation matrix difference matrix based on the translation vector, expressed as:

[0051]

[0052] where t is the translation vector, is the zero vector

[0053] S48. Calculate the L1 norm of the translation matrix difference matrix, expressed as:

[0054]

[0055] S49. Calculate the difference between two frames of images based on the L1 norm of the rotation matrix difference matrix and the L1 norm of the translation matrix difference matrix, expressed as:

[0056] m = s·m1 + (1 - s)·m2

[0057] where s is the scaling factor.

[0058] The present invention has the following beneficial effects:

[0059] The present invention uses a visual feature point matching method to achieve relative positioning of two aircraft flying before and after, and has the advantages of low cost, simple equipment, large amount of information, strong autonomy, etc., and can well meet the requirements of relative pose estimation between two aircraft, so as to improve the adaptability, intelligence and concealment of dual-aircraft pose estimation. Description of the Drawings

[0060] Figure 1 Schematic diagram of the dual - aircraft relative positioning method based on visual feature point matching in the present invention;

[0061] Figure 2 Effect diagram of feature point extraction in the present invention;

[0062] Figure 3 Comparison diagram of feature point matching effects in the present invention;

[0063] Figure 4 Effect diagram of the dual - aircraft relative positioning experiment based on visual feature point matching in the present invention. Specific implementation manners

[0064] The following describes the specific implementation manners of the present invention to facilitate those skilled in the art of the present technology to understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation manners. For those of ordinary skill in the art of the present technology, as long as various changes are within the spirit and scope of the present invention defined and determined by the appended claims, these changes are obvious, and all inventions and creations using the concept of the present invention are within the scope of protection.

[0065] As Figure 1 shown, an embodiment of the present invention provides a dual - aircraft relative positioning method based on visual feature point matching, including the following steps S1 to S5:

[0066] S1. Obtain the dual - aircraft image frame sequence and the relative pose corresponding to each frame of the image at the corresponding moment, and construct an image relative pose sequence database;

[0067] In an alternative embodiment of the present invention, in this embodiment, a camera fixedly connected to the rear aircraft is used to obtain an image sequence of the front aircraft under the camera of the rear aircraft at a certain frequency during the movement of the dual - aircraft. At the same time, an instrument with relatively high precision can be used to record the relative pose between the front aircraft and the rear aircraft when each frame of the image is taken. Finally, an image - relative pose sequence will be obtained, and then this sequence can be encapsulated into a database for later query. The frequency of image acquisition is set according to the accuracy requirements. Due to the continuity of space, it is impossible to collect all visual scenes. When the required accuracy is relatively high, the sampling frequency should be as high as possible. When the accuracy requirement is not high, the sampling frequency can be appropriately reduced.

[0068] S2. Obtain real - time dual - aircraft images, perform feature point extraction, and determine the feature points of the real - time dual - aircraft images;

[0069] In an alternative embodiment of the present invention, in this embodiment, feature point extraction is performed on the real - time dual - aircraft images to determine the feature points of the real - time dual - aircraft images, which specifically includes the following steps:

[0070] S21. Determine key points based on the change in pixel gray values in the real-time dual-camera images;

[0071] S22. Construct an image pyramid from the real-time dual-camera images, and extract key points at each layer of the image pyramid to construct feature descriptors of the key points, obtaining the feature points of the real-time dual-camera images.

[0072] Among them, step S21 specifically includes the following steps:

[0073] S211. Sequentially select a pixel point in the real-time dual-camera images as a to-be-determined key point, and obtain the pixel gray value of the to-be-determined key point;

[0074] S212. Set the pixel gray threshold of the to-be-determined key point, and compare the pixel gray value of the to-be-determined key point with the gray values of all pixel points within a set radius centered on the to-be-determined key point;

[0075] If there are a continuous set number of pixel points whose gray values are greater than the sum of the pixel gray value of the to-be-determined key point and the pixel gray threshold, or less than the difference between the pixel gray value of the to-be-determined key point and the pixel gray threshold, then the to-be-determined key point is determined as a key point;

[0076] S213. Traverse all pixel points in the real-time dual-camera images to obtain all key points in the real-time dual-camera images.

[0077] Specifically, in this embodiment, an anterior monocular camera fixedly installed directly in front of the flight direction of the rear machine is used to obtain image information. The original image information can be a single-channel grayscale image or a three-channel RGB image, with a resolution set to 800 * 450 pixels and an image frame rate set to 30 frames per second.

[0078] ORB feature points are composed of FAST corner points and BRIEF descriptors. First, the FAST corner points are used to determine the pixel points in the image that are significantly different from the surrounding pixels as key points, and then the BRIEF descriptors of each key point are calculated to determine the ORB feature points.

[0079] The FAST corner points determine key points by comparing the change in pixel gray values in the image. If there is a pixel point with a significantly larger gray value in a region with a relatively small gray value, then this pixel point has obvious features in this region and can be used as a feature point. This algorithm compares the relationship between the gray value of the central pixel in a certain region and the gray values of the surrounding pixels. If there is an obvious difference between the gray values of the surrounding pixels and the central pixel, then it can be determined as a FAST corner point. The calculation process is as follows:

[0080] 1) Select a pixel point in the image as a to-be-determined key point p, and its pixel value is I p ;

[0081] 2) Set the pixel threshold for determining FAST corner points, such as T p = 20% × I p ;

[0082] 3) Compare the pixel value of the center point p with the pixel values of all pixels on the circumference with a radius of r. If there are N consecutive pixels whose pixel values are greater than I p + T p or less than I p - T p , then the point p can be determined as a FAST corner point. N is generally selected as 12, corresponding to the FAST-12 corner point;

[0083] 4) Traverse each pixel point in the image and repeat the above steps to obtain all FAST corner points that meet the conditions in the image.

[0084] FAST corner points do not have scale invariance and rotation invariance. ORB feature points add scale and rotation invariance on the basis of FAST corner points. By constructing an image "pyramid" and extracting FAST corner points on each layer, the FAST corner points detected in multiple layers of images have scale invariance. Therefore, ORB feature points only retain the FAST corner points with scale invariance. The scale invariance of ORB features depends on obtaining the image pyramid. The number of layers of the pyramid is set to L, and the scale factor is s. The pyramid will perform downsampling of the source image at different levels. The higher the number of pyramid layers, the smaller the area of the image and the fewer the number of feature points extracted, as Figure 2 shown.

[0085] Divide the feature points evenly according to the area of each layer of the image. Let the width of the image in the 0th layer be W, the length be L, and the scaling factor be s. The total area of the pyramid is:

[0086]

[0087] The number of feature points per unit area is:

[0088]

[0089] Derive the number of feature points allocated in the nth layer as:

[0090]

[0091] Each extracted feature point has a corresponding layer number of the image pyramid. When the camera moves in the three-dimensional world, the same world point may be extracted by different pyramid layer numbers in different images. The image pyramid ensures that even if the camera moves, the same feature point can always be recognized by the camera and feature matching will occur between different levels of the pyramid. The matching effect is as Figure 3 shown.

[0092] Regarding rotational invariance, the ORB feature points propose a vector from the FAST corner points to the centroid of the surrounding area (the center weighted by the grayscale values of the image patch) as the direction vector of the ORB feature points. The direction of the feature points can be obtained from the direction vector, thus solving the problem of rotational invariance. The steps to calculate the direction vector are as follows:

[0093] 1) Select a rectangular area centered on the FAST corner points and calculate the image moments of the rectangular area. The calculation formula is as follows:

[0094]

[0095] 2) Calculate the centroid of the rectangular area using the following formula:

[0096]

[0097] 3) Connect the FAST corner points and the centroid to obtain the direction vector:

[0098]

[0099] Since FAST corner points tend to appear concentrated, a large number of FAST corner points are very likely to appear near a certain FAST corner point. Therefore, it is necessary to use the non-maximum suppression algorithm to retain the FAST corner point with the largest corner response value within a certain range and remove other points.

[0100] In this embodiment, the number of ORB feature points detected nfeatures = 500; the scale factor of the pyramid size scaleFactor = 1.2f; the number of pyramid levels nlevels = 8; the edge threshold edgeThreshold = 31; the level where the original image is located firstLevel = 0, that is, the original image is placed at the bottom of the pyramid; the number of pixel points WTA_K = 2 required for generating each descriptor; the evaluation method of the key points when detecting key points is the Harris category; the size of the area around the key points when generating descriptors patchsize = 31; the threshold of the pixel value difference when calculating FAST corner points fastThreshold = 20.

[0101] According to the above set values, the following can be obtained:

[0102] The pixel threshold of the FAST corner points is:

[0103]

[0104] The effective pixel value range is:

[0105] [0, I p -T p ∪ [I p +Tp , 255] = [0, 0.8I p ∪ [1.2I p , 255]

[0106] If among the pixel values of all pixels in the surrounding area with a radius of 31 around the pixel value at the center point p, there are 12 consecutive pixels whose pixel values are not in the invalid pixel value area, then the center point p is a FAST corner point.

[0107] According to the scaling factor So the total area of the pyramid is:

[0108] S = L × W × [s 2 0 + L × W × [s 2 1 +... + L × W × [s 2 n-1 = 3.0957·L·W

[0109] The number of feature points per unit area is:

[0110]

[0111] So the number of feature points allocated in the α-th layer is:

[0112]

[0113] S3. Match the feature points of the real-time dual-camera images with the image relative pose sequence database to determine the matching feature point pairs and eliminate the mismatched point pairs;

[0114] In an alternative embodiment of the present invention, in step S3, matching the feature points of the real-time dual-camera images with the image relative pose sequence database to determine the matching feature point pairs specifically includes the following steps:

[0115] S31. Randomly select multiple pairs of matching feature point pairs and calculate the homography matrix;

[0116] S32. According to the calculated homography matrix, project the feature points with matching relationships in the first frame image onto the second frame image and calculate the reprojection coordinates;

[0117] S33. Calculate the distance between the reprojection coordinates and the coordinates of the already matched feature points;

[0118] S34. Determine whether the calculated distance is less than the set distance threshold. If so, it is determined as a correct matching feature point pair; otherwise, it is determined as an incorrect matching feature point pair, and record the number of correct matching feature point pairs;

[0119] ​​​S35. Repeat steps S31 to S34 according to the set number of times, compare the number of correctly matched feature point pairs counted after multiple loops, and take the case with the largest number of correctly matched feature point pairs as the final result.

[0120] The steps of removing mis-matched point pairs from the matched feature point pairs in step S3 specifically include the following steps:

[0121] S36. Select the smallest data set from which a model can be estimated;

[0122] S37. Calculate the data model using the smallest data set;

[0123] S38. Substitute all the data into the data model and calculate the number of inliers;

[0124] S39. Compare the number of inliers of the current model and the optimal model obtained previously, and record the model parameters and the number of inliers with the largest number of inliers;

[0125] S310. Repeat steps S36 to S39 until the maximum number of iterations is reached or the number of inliers of the current model reaches the set number threshold.

[0126] Specifically, in this embodiment, first, the difference in the number of feature points between the image to be matched and the database graphics is judged. This step is mainly for rough matching to improve the image matching efficiency. If feature matching, mis-matching removal, similarity calculation and other steps are performed for each database image with the image to be matched, the running efficiency of the program will be greatly reduced. Therefore, before performing feature point matching, the number of feature points of the image to be matched and the database image can be roughly compared. If the number difference is too large, the subsequent steps are not executed and the similarity is directly set to zero; otherwise, the subsequent steps are executed.

[0127] Feature point matching is to find the same feature point of the same object in different images. Since each feature point has a descriptor that marks its unique identity and characteristics, feature point matching is actually to find two feature points with similar descriptors in two images.

[0128] There is an important parameter in FLANN (Fast_Library_for_Approximate_Nearest_Neighbors) matching: indexParams, which is used to configure the algorithm to be used. In the experiment, the priority search k-means tree algorithm with higher accuracy was selected.

[0129] When using the FLANN method for matching, the descriptor needs to be of type CV_32F. Therefore, the descriptor variable of the ORB feature point needs to be type-converted before feature point matching can be achieved.

[0130] When creating a FLANN matcher, the random K-means method with higher precision is selected. The level of the convenience tree is set to 5, and the search value is set to 50. In this case, the computational amount is relatively small, and the computational speed and matching accuracy can be guaranteed.

[0131] The RANSAC (RAndom SAmple Consensus) algorithm is an iterative algorithm for correctly estimating the parameters of a mathematical model from a set of data containing "outliers". "Outliers" generally refer to the noise in the data, such as mis-matches in matching and outliers in the estimated curve. Therefore, RANSAC is also an "outlier" detection algorithm. RANSAC estimates the model by repeatedly selecting data sets and iterating until a relatively good model is estimated. The specific implementation steps can be divided into the following five steps:

[0132] 1) Select the smallest data set that can estimate the model; (for line fitting, it is two points, and for calculating the Homography matrix, it is 4 points)

[0133] 2) Use this data set to calculate the data model;

[0134] 3) Substitute all the data into this model and calculate the number of inliers; (accumulate the data that fits the model deduced in the current iteration within a certain error range)

[0135] 4) Compare the number of "inliers" of the current model and the best model deduced previously, and record the model parameters and the number of inliers with the largest number of inliers;

[0136] 5) Repeat steps 1-4 until the iteration ends or the current model is good enough (the number of inliers is greater than a certain number). The calculation process of the iteration number k is as follows:

[0137] 1) Assume that the proportion of inliers in the data is:

[0138]

[0139] 2) When calculating the model using N points, the situation where at least one outlier is selected from the selected points is:

[0140] 1 - t N

[0141] 3) When the iteration number is k, the probability of sampling the correct N points to calculate the correct model is:

[0142] P = 1 - (1 - t n ) k

[0143] 4) Calculate the iteration number:

[0144]

[0145] In the feature point matching scheme, the optimization implementation of the RANSAC algorithm can be summarized into the following three steps:

[0146] 1) Randomly select four pairs of matching pairs and calculate the homography matrix;

[0147] 2) According to the calculated homography matrix, project the feature points with matching relationships in the first frame image onto the second frame image and calculate the reprojection coordinates. Then calculate the distance between the reprojection coordinates and the coordinates of the already matched feature points. If it is less than the set threshold, then they are considered correct matching point pairs; otherwise, they are regarded as incorrect matches, and the number of correct matching point pairs is recorded. Here, the least squares method is used to calculate the homography matrix;

[0148] 3) Replace the matching pairs to calculate the homography matrix, repeat the above two steps, compare the number of correct matching point pairs statistically after multiple loops, take the case with the largest number of correct matching point pairs as the final result, eliminate the incorrect matches, and output the correct matching pairs, thereby realizing the screening of feature point matching.

[0149] Before executing the RANSAC algorithm, it is first necessary to optimize the matching feature points using the descriptor distance. When the threshold is too low, the optimization effect is not obvious. When the threshold is too high, although some incorrect matches can be removed, some correct matching point pairs will also be lost. Therefore, it is particularly important to select a suitable threshold. The optimization steps designed in this paper are as follows:

[0150] 1) Calculate the Hamming distance between all matching point pairs, and record the maximum value max_dist and the minimum value min_dist;

[0151] 2) Take 0.4 times the maximum distance max_dist as the optimal matching result for screening. If the distance of the matching point pair is less than this threshold, it is considered a correct matching pair;

[0152] 3) Eliminate all incorrect matching pairs.

[0153] 4) Execute the RANSAC algorithm. First, randomly select 4 pairs of feature points, calculate the homography matrix between two frames of images, and then calculate the distance between the reprojection coordinates of the feature points in the first frame of image in the second frame of image according to the homography matrix. If it is less than a certain threshold, it is considered a correct matching point pair; otherwise, it is regarded as an incorrect match, and record the number of correct matching point pairs. Then repeat the above two steps, compare the number of correct matching point pairs counted after multiple loops, and take the case with the largest number of correct matching point pairs as the final result. Here, we use the least squares method to calculate the homography matrix, set the maximum number of iterations of RANSAC to 3000, and the confidence interval to 0.995.

[0154] S4. Determine the translation and rotation vectors between two frames of images according to the matching feature point pairs, and determine the frame of image in the image relative pose sequence database with the smallest difference from the real-time dual-camera images according to the translation and rotation vectors;

[0155] In an optional embodiment of the present invention, the step of determining the translation and rotation vectors between two frames of images according to the matching feature point pairs in step S4 specifically includes the following steps:

[0156] S41. Set the sum of the squares of the Euclidean distances between the matching feature point pairs as the objective function, expressed as:

[0157]

[0158] where p i , p′ i are the feature point coordinates of the i-th pair of matching feature point pairs, p, p′ are the centroid coordinates of the i-th pair of matching feature point pairs, R is the rotation vector, t is the translation vector, and n is the number of pairs of matching feature point pairs;

[0159] S42. Construct a matching feature point pair relationship matrix according to the translation and rotation vectors between two frames of images, expressed as:

[0160]

[0161] where T is the transpose symbol;

[0162] S43. Perform singular value decomposition on the matching feature point pair relationship matrix to obtain the rotation vector, expressed as:

[0163] R = UV T

[0164] where UV is the diagonal matrix of the value decomposition;

[0165] S44. Calculate the translation vector according to the rotation vector, expressed as:

[0166] t = p - Rp'.

[0167] In step S4, the frame of image in the image relative pose sequence database with the smallest difference from the real-time dual-camera image is determined according to the translation and rotation vectors, which specifically includes the following steps:

[0168] S45. Construct a rotation matrix difference matrix according to the rotation vector, expressed as:

[0169] E = R - I 2×2

[0170] where R is the rotation vector and I 2×2 is the identity matrix;

[0171] S46. Calculate the L1 norm of the rotation matrix difference matrix, expressed as:

[0172] m1 = norm1{E}

[0173] where norm1 represents the L1 norm function;

[0174] S47. Construct a translation matrix difference matrix according to the translation vector, expressed as:

[0175]

[0176] where t is the translation vector, is the zero vector

[0177] S48. Calculate the L1 norm of the translation matrix difference matrix, expressed as:

[0178]

[0179] S49. Calculate the difference between two frames of images according to the L1 norm of the rotation matrix difference matrix and the L1 norm of the translation matrix difference matrix, expressed as:

[0180] m = s·m1 + (1 - s)·m2

[0181] where s is the scale factor.

[0182] Specifically, ICP (Iterative closet point) is an iterative rigid matching algorithm. It finds the nearest point s in the point set S for each point m in the point set M i and then obtains the transformation relationship T by the least squares method. j

[0183] The ICP algorithm is sensitive to the initial positions of the point sets to be paired. When the initial position of point set M is close to point set S, the registration effect will be better. In addition, the ICP algorithm changes the closest point pairs in each calculation iteration, so it is actually difficult to prove that the ICP algorithm can accurately converge to the local optimal value. However, due to its intuitive understanding and easy implementation, it is still the most commonly used point set registration algorithm at present. When calculating the transformation T, the SVD method can be used, and the process is as follows:

[0184] 1) First, define the error of the i-th pair of points as:

[0185] e i =p i -(Rp i '+t)

[0186] 2) Based on the above formula, a least squares problem can be constructed to obtain R and t that minimize the sum of squared errors:

[0187]

[0188] 3) Define the centroids of the two sets of points:

[0189]

[0190]

[0191] 4) Calculate the centroid positions p and p’ of the two sets of points, and then calculate the centroid-removed coordinates of each point:

[0192] q i =p i -p

[0193] q’ i =p’ i -p’

[0194] Thus, we can obtain:

[0195]

[0196] So the entire optimization problem is transformed into:

[0197]

[0198] 5) Define the matrix:

[0199]

[0200] 6) Perform singular value decomposition on W to obtain:

[0201] W=UΣV T

[0202] Among them, Σ is a diagonal matrix composed of singular values, and the diagonal elements are arranged from large to small. U and V are diagonal matrices. When W is full rank:

[0203] R = UV T

[0204] 7) Calculate the translation vector t according to the R obtained in the previous step:

[0205] t * = p - Rp'

[0206] After extracting the feature points from the images, a point set will be corresponding. After the feature point matching of two images, the corresponding relationship between the points in the two point sets is established. Then, the pose change between the two point sets can be calculated. Assuming the rotation vector is R and the translation vector is t, the sum of the squares of the Euclidean distances between all point pairs is defined as the objective function:

[0207]

[0208] where A i and B i represent the feature points (correctly matched points) of images A and B respectively. For the 2D-2D ICP algorithm, R is a second-order square matrix and T is a two-dimensional vector.

[0209] The rotation and translation vectors between two frames of images have been obtained in the previous step. Next, the similarity between these two images can be constructed based on these two vectors. According to empirical knowledge, when the rotation matrix between two frames of images is close to the identity matrix and the translation vector is close to the zero vector, the two images can be considered as the same image. If the deviation is larger, it indicates that the difference between the two images is greater. According to this idea, the difference between the two images can be constructed (similarity and difference are relative). The specific calculation steps are as follows:

[0210] 1) Construct the rotation matrix difference matrix E, and then calculate the L1 norm m1:

[0211]

[0212]

[0213] 2) Construct the translation vector difference m2, also in the form of L1 norm:

[0214]

[0215]

[0216] 3) Find a suitable scaling factor s, and combine m1 and m2 to obtain the difference between the two frames of images:

[0217] m = s·m1+(1 - s)·m2

[0218] In the previous step, the translation vector t and the rotation vector R between two frames of images have been obtained. Thus, two similarities can be constructed based on these two vectors and combined according to a certain scale factor. During the experiment, the scale factor s is continuously adjusted. Finally, it is found that when s = 0.15, the matching effect is optimal. At this time, the difference expression is as follows:

[0219] m = s·m1+(1 - s)·m2 = 0.15m1+0.85m2 = 0.15norm1{R * -I}+0.85norm1{t *}

[0220] S5. According to the relative pose of the two cameras corresponding to a frame of image in the determined image relative pose sequence database, it is used as the relative pose of the two cameras at the current moment of the real-time two-camera images.

[0221] In an alternative embodiment of the present invention, after the above steps, the differences between the image to be matched and all the data images can be calculated. Next, it is necessary to find the database image with the smallest difference and output the corresponding relative pose of the two cameras as the relative pose of the two cameras at the corresponding moment of the image to be matched; as Figure 4 shown, it is the experimental effect diagram of the relative positioning in the air of two cameras based on visual feature point matching in the present invention.

[0222] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, and the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate means for implementing the functions specified in one Figure 1 process or multiple processes and / or blocks Figure 1 block or multiple blocks.

[0223] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device, and the instruction device implements the functions specified in one Figure 1 process or multiple processes and / or blocks Figure 1 block or multiple blocks.

[0224] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus, so that a series of operation steps are performed on the computer or other programmable apparatus to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable apparatus provide steps for implementing the functions specified in one process or a plurality of processes and / or blocks Figure 1 one process or a plurality of processes and / or blocks Figure 1 or steps for implementing the functions specified in a block or a plurality of blocks.

[0225] Specific embodiments are used in the present invention to illustrate the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

[0226] Those of ordinary skill in the art will realize that the embodiments described herein are for helping the reader understand the principles of the present invention, and it should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those of ordinary skill in the art can make various other specific deformations and combinations that do not depart from the essence of the present invention based on the technical revelations disclosed in the present invention, and these deformations and combinations are still within the protection scope of the present invention.

Claims

1. A dual - aircraft air relative positioning method based on visual feature point matching, characterized in that, It includes the following steps: S1. Obtain the binocular image frame sequence and the relative pose at the corresponding moment of each frame of image, and construct an image relative pose sequence database; S2. Obtain the real-time binocular image, extract feature points, and determine the feature points of the real-time binocular image; S3. Match the feature points of the real-time binocular image with the image relative pose sequence database, determine the matching feature point pairs, and eliminate the mis-matching point pairs; S4. Determine the translation and rotation vectors between two frames of images according to the matching feature point pairs, and determine the frame of image in the image relative pose sequence database with the smallest difference from the real-time binocular image according to the translation and rotation vectors; S5. Use the relative binocular pose corresponding to a frame of image determined in the image relative pose sequence database as the current binocular pose of the real-time binocular image.

2. The dual - aircraft air relative positioning method based on visual feature point matching according to claim 1, characterized in that, In step S2, when extracting the feature points of the real-time binocular image and determining the feature points of the real-time binocular image, it specifically includes the following steps: S21. Determine the key points according to the change of pixel gray value in the real-time binocular image; S22. Construct an image pyramid according to the real-time binocular image, and extract key points on each layer of the image pyramid to construct the feature descriptors of the key points, so as to obtain the feature points of the real-time binocular image.

3. The dual - aircraft air relative positioning method based on visual feature point matching according to claim 2, characterized in that, Step S21 specifically includes the following steps: S211. Select a pixel point in the real-time binocular image as a to-be-determined key point in turn, and obtain the pixel gray value of the to-be-determined key point; S212. Set the pixel gray value threshold of the to-be-determined key point, and compare the pixel gray value of the to-be-determined key point with the gray values of all pixel points within the set radius centered on the to-be-determined key point; If there are a continuous set number of pixel points whose gray values are greater than the sum of the pixel gray value of the to-be-determined key point and the pixel gray value threshold, or less than the difference between the pixel gray value of the to-be-determined key point and the pixel gray value threshold, then it is determined that the to-be-determined key point is a key point; S213. Traverse all pixel points in the real-time binocular image to obtain all key points in the real-time binocular image.

4. The dual - aircraft air relative positioning method based on visual feature point matching according to claim 1, characterized in that, In step S3, when matching the feature points of the real-time binocular image with the image relative pose sequence database to determine the matching feature point pairs, it specifically includes the following steps: S31. Randomly select multiple pairs of matching feature point pairs and calculate the homography matrix; S32. According to the calculated homography matrix, project the feature points with matching relationships in the first frame of image to the second frame of image and calculate the reprojection coordinates; S33. Calculate the distance between the reprojection coordinates and the coordinates of the already matched feature points; S34. Determine whether the calculated distance is less than the set distance threshold. If so, it is determined as a correct matching feature point pair, otherwise it is determined as a wrong matching feature point pair, and record the number of correct matching feature point pairs; S35. Repeat steps S31 to S34 according to the set number of times, compare the number of correct matching feature point pairs statistically after multiple loops, and take the situation with the largest number of correct matching feature point pairs as the final result.

5. The dual - aircraft air relative positioning method based on visual feature point matching according to claim 4, characterized in that, In step S3, eliminating the mis-matching point pairs from the matching feature point pairs specifically includes the following steps: S36. Select the smallest data set that can estimate the model; S37. Calculate the data model using the smallest data set; S38. Bring all the data into the data model and calculate the number of inliers; S39. Compare the number of inliers of the current model and the optimal model obtained previously, and record the model parameters and the number of inliers with the maximum number of inliers; S310. Repeat steps S36 to S39 until the maximum number of iterations is reached or the number of inliers of the current model reaches the set number threshold.

6. The dual - aircraft air relative positioning method based on visual feature point matching according to claim 1, characterized in that, In step S4, determining the translation and rotation vectors between two frames of images according to the matching feature point pairs specifically includes the following steps: S41. Set the sum of the squares of the Euclidean distances between the matching feature point pairs as the objective function, expressed as: where p i , p i ′ are the feature point coordinates of the i-th pair of matching feature points, p, p′ are the centroid coordinates of the i-th pair of matching feature points, R is the rotation vector, t is the translation vector, and n is the number of pairs of matching feature points; S42. Construct a relationship matrix of the matching feature point pairs according to the translation and rotation vectors between two frames of images, expressed as: where, T is the transpose symbol; S43. Perform singular value decomposition on the relationship matrix of the matching feature point pairs to obtain the rotation vector, expressed as: R = UV T where, UV is the diagonal matrix of the value decomposition; S44. Calculate the translation vector according to the rotation vector, expressed as: t = p - Rp'.

7. The method for dual - aircraft relative positioning in the air based on visual feature point matching according to claim 1, characterized in that, In step S4, determining the frame of image in the image relative pose sequence database with the smallest difference from the real-time binocular images according to the translation and rotation vectors specifically includes the following steps: S45. Construct a difference matrix of rotation matrices according to the rotation vector, expressed as: E = R - I 2×2 where R is a rotation vector and I 2×2 is the identity matrix; S46. Calculate the L1 norm of the difference matrix of rotation matrices, expressed as: m1 = norm1{E} where, norm1 represents the L1 norm function; S47. Construct a difference matrix of translation matrices according to the translation vector, expressed as: where t is the translation vector, is the zero vector S48. Calculate the L1 norm of the difference matrix of translation matrices, expressed as: S49. Calculate the difference between two frames of images according to the L1 norm of the difference matrix of rotation matrices and the L1 norm of the difference matrix of translation matrices, expressed as: m = s·m1 + (1 - s)·m2 where, s is the scaling factor.

Images