An aerial vehicle image matching positioning method based on pose guidance candidate region

Abstract

The application discloses an image matching positioning method of an aircraft based on a pose-guided candidate region, and comprises the following steps: 1. obtaining pixel coordinates of an angle point of a field of view of the aircraft mapped onto a reference image based on a pose of the aircraft; 2. obtaining a candidate region image block from the reference image; 3. performing feature extraction on a real-time image of the aircraft and the candidate region image block by using an XFeat lightweight network; 4. obtaining a mutual nearest neighbor matching pair set; 5. obtaining a homographic transformation matrix from the real-time image to the candidate region image block and an optimized matching point pair by using a PROSAC algorithm based on the mutual nearest neighbor matching pair set; and 6. realizing positioning of the aircraft based on the homographic transformation matrix. The method is simple in steps, the candidate region image block containing the real-time image of the aircraft is obtained from the reference image based on the pose of the aircraft, the search range of scene matching is reduced, the calculation efficiency and matching robustness are improved, and the positioning accuracy is further improved.

Description

Technical Field

[0001] This invention belongs to the field of aircraft positioning technology, and in particular relates to an image matching and positioning method for aircraft based on pose-guided candidate regions. Background Technology

[0002] In application, aircraft matching navigation technology typically requires pre-selecting regions with rich texture and small elevation differences as matching areas and preparing a reference map. As the aircraft flies through this region, it matches the real-time image with the reference map to correct its flight status. This approach results in excessive preparation time for the aircraft and limits its online flight decision-making. In some cases, it may even fail to find a sufficiently suitable matching area. Low-assertivity scene matching navigation pre-binds the entire area the aircraft might fly through as a reference map. During flight, the aircraft performs scene matching "anytime, anywhere" based on the status information provided by inertial navigation. This method eliminates the need for reference map preparation, preprocessing, and binding, compressing the preparation process, reducing the risk of human error, and adapting to scenarios where the aircraft makes online decisions that deviate significantly from the standard trajectory. However, this approach faces two main challenges: first, the aircraft needs to search and complete matching within a larger area, increasing computational load and matching difficulty; second, the matching features exhibit low robustness in scenarios with weak texture or occlusion.

[0003] Therefore, a well-designed image matching and localization method for aircraft based on pose-guided candidate regions is needed. Based on the aircraft's pose, candidate region image patches containing the real-time image of the aircraft are obtained from the reference image. The real-time image and candidate region image patches are then used to extract key point features and perform feature matching through the XFeat lightweight network and the mutual nearest neighbor matching method. This reduces the search range of scene matching, improves computational efficiency and matching robustness, and thus improves localization accuracy. Summary of the Invention

[0004] The technical problem to be solved by this invention is to address the shortcomings of the prior art by providing an image matching and localization method for aircraft based on pose-guided candidate regions. The method has simple steps: based on the aircraft's pose, candidate region image blocks containing the real-time image of the aircraft are obtained from the reference image; and the real-time image and candidate region image blocks are used to extract key point features and perform feature matching through the XFeat lightweight network and the nearest neighbor matching method. This reduces the search range for scene matching, improves computational efficiency and matching robustness, and thus improves localization accuracy.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is: an image matching and localization method for aircraft based on pose-guided candidate regions, the method comprising the following steps:

[0006] Step 1: Obtain the pixel coordinates of the corner points of the aircraft's field of view mapped onto the reference map based on the aircraft's pose;

[0007] Step 2: Based on the pixel coordinates of the corner points mapped to the reference image, obtain candidate region image patches from the reference image;

[0008] Step 3: Use the XFeat lightweight network to extract features from the real-time image of the aircraft and candidate region image blocks to obtain descriptor feature maps and pixel coordinates of the final key points; wherein, the XFeat lightweight network includes a descriptor generation network branch and a key point detection network branch;

[0009] Step 4: Obtain the set of mutual nearest neighbor pairs:

[0010] Based on the pixel coordinates and descriptor feature maps of the final key points, the mutual nearest neighbor matching method is used to match the descriptor feature vectors of the final key points in the real-time image and the candidate region image blocks to obtain a set of mutual nearest neighbor matching pairs.

[0011] Step 5: Based on the set of mutual nearest neighbor matching pairs, the PROSAC algorithm is used to obtain the homography transformation matrix from the real-time image to the candidate region image patch and the optimized matching point pairs;

[0012] Step 6: Based on the homography transformation matrix, project the center pixel coordinates of the real-time image onto the candidate region image block, and then convert them into geodetic latitude and longitude coordinates to achieve the positioning of the aircraft.

[0013] The above-mentioned image matching and localization method for aircraft based on pose-guided candidate regions, further, step one, is as follows:

[0014] Step 101, according to The horizontal field of view of the camera on the aircraft was obtained. and vertical field of view ;in, This indicates the camera's focal length, expressed in meters. Indicates pixel size, in meters per pixel; The image resolution is represented by W (width) and H (height), with units of pixels.

[0015] Step 102, according to The expanded horizontal field of view is obtained. and the expanded vertical field of view ;in, Indicates the expansion factor. The value range is 1.02 to 1.05;

[0016] Step 103, according to , , and This yields the direction vectors of the four corner points (top right, top left, bottom left, and bottom right) in the camera coordinate system. Where i is a positive integer, and i can take the value 1, 2, 3 or 4;

[0017] Step 104, according to This yields the direction vector of the i-th corner point transformed into the navigation coordinate system. ;in, This represents the rotation matrix from the camera coordinate system to the body coordinate system. This represents the rotation matrix from the navigation coordinate system to the body coordinate system;

[0018] Step 105: Transform the i-th corner point into the direction vector in the navigation coordinate system. After unfolding, you get ;in, Indicates the position of the i-th corner point in the navigation coordinate system Axial components, Indicates the position of the i-th corner point in the navigation coordinate system Axial components, Indicates the position of the i-th corner point in the navigation coordinate system Axial components;

[0019] Step 106, according to The scaling factor for the i-th corner point is obtained; where the ground plane is represented in the navigation coordinate system as... =-h, where h is the aircraft altitude;

[0020] Step 107: Transform the direction vector of the i-th corner point into the navigation coordinate system, and obtain the i-th ground point intersecting the ground plane through perspective projection. Then, based on... To obtain the coordinates of the i-th ground point in the navigation coordinate system. ;in, Indicates the position of the i-th ground point in the navigation coordinate system Axial components, Indicates the position of the i-th ground point in the navigation coordinate system Axial components; T indicates transpose;

[0021] Step 108, according to Obtain the latitude and longitude of the i-th corner point. ;in, The average radius of the Earth Pi The latitude and longitude of the spacecraft's center of mass;

[0022] Step 109, according to The geographic coordinates of the i-th corner point are mapped to the pixel coordinates of the base map. ;in, This represents the latitude and longitude coordinates of the upper right corner of the reference map. This represents the latitude and longitude coordinates of the lower left corner of the reference map. This indicates the width and height pixel dimensions of the baseline image.

[0023] The above-mentioned image matching and localization method for aircraft based on pose-guided candidate regions, further, step two, is as follows:

[0024] Step 201: Based on the pixel coordinates of the four corner points mapped to the reference image, according to... ,get The enclosed candidate area; among which, This represents the floor function. This represents the floor function. Represents the minimum x-coordinate. Indicates the maximum x-coordinate. Represents the minimum y-coordinate. Indicates the maximum ordinate;

[0025] Step 202, based on The enclosed candidate region is cropped from the reference image to obtain candidate region image blocks.

[0026] The above-mentioned image matching and localization method for aircraft based on pose-guided candidate regions, further, step three, is as follows:

[0027] Step 301: Input the real-time graph into the descriptor generation network branch of the XFeat lightweight network to obtain the descriptor feature map and reliability heatmap;

[0028] Step 302: Divide the real-time image into 8×8 grid regions, input each grid region into the keypoint detection network branch of the XFeat lightweight network for processing, and output a 65-dimensional vector; where the first 64 dimensions represent probability values ​​and the 65th dimension represents the probability of no keypoints.

[0029] Step 303: Reshape the first 64-dimensional vector into a local keypoint heatmap; and stitch together each local keypoint heatmap to form a global keypoint heatmap;

[0030] Step 304: For the global key point heatmap, set the probability value of the grid area with no key point probability greater than 0.5 to zero to obtain the filtered global key point heatmap;

[0031] Step 305: Use the Non-Maximum Suppression (NMS) method to evaluate the filtered global key point heatmap and obtain the key point grid coordinates; wherein, the size of the NMS window is 5×5;

[0032] Step 306: Obtain the reliability value corresponding to the grid coordinates of the key point from the reliability heatmap; and obtain the score corresponding to the key point according to score=w1×a′+w2×b′; where w1 represents the first weight, a′ is the probability value corresponding to the grid coordinates of the key point, w2 represents the second weight, and b′ is the reliability value corresponding to the grid coordinates of the key point; the values ​​of w1 and w2 are 0 to 1, and w1+w2=1;

[0033] Step 307: Sort the scores corresponding to the key points in descending order, and take the first K key points as the final key points. This will give you the descriptor feature map corresponding to the real-time image and the pixel coordinates of the final key points; where K is a positive integer.

[0034] Step 308: Input the candidate region image block and process it according to the methods in steps 301 to 307 to obtain the descriptor feature map and the pixel coordinates of the final key point corresponding to the candidate region image block.

[0035] The above-mentioned image matching and localization method for aircraft based on pose-guided candidate regions, further, step four, is as follows:

[0036] Step 401: Map the pixel coordinates of the final key points corresponding to the real-time image onto the descriptor feature map to obtain the mapped coordinates of the final key points;

[0037] Step 402: On the descriptor feature map, obtain the 4-neighbor coordinates that surround the mapped coordinates of the final key point, and obtain the Euclidean distances between the left pixel coordinate, right pixel coordinate, upper pixel coordinate, lower pixel coordinate and the mapped coordinates of the final key point in the 4-neighbor coordinates, denoted as a, b, c, d respectively; and obtain the descriptor feature vector corresponding to the 4-neighbor coordinates from the descriptor feature map.

[0038] Step 403: According to A = a × A1 + b × A2 + c × A3 + d × A4, obtain the descriptor feature vector A of the final key point; where A1 represents the descriptor feature vector corresponding to the coordinates of the left pixel; A2 represents the descriptor feature vector corresponding to the coordinates of the right pixel; A3 represents the descriptor feature vector corresponding to the coordinates of the upper pixel; and A4 represents the descriptor feature vector corresponding to the coordinates of the lower pixel.

[0039] Step 404: Repeat steps 401 to 403 multiple times to obtain the descriptor feature vector for each final key point;

[0040] Step 405: Following the methods of steps 401 to 404, obtain the descriptor feature vector of each final key point in the candidate region image block;

[0041] Step 406: Using the mutual nearest neighbor matching method, based on the cosine similarity between the descriptor feature vector of each final key point in the real-time image and the descriptor feature vector of each final key point in the candidate region image block, if the cosine similarity is greater than 0.85, the corresponding final key points in the real-time image and the candidate region image block are recorded as mutual nearest neighbor matching pairs, thus obtaining the set of mutual nearest neighbor matching pairs.

[0042] The above-mentioned image matching and localization method for aircraft based on pose-guided candidate regions is further elaborated in step five, the specific process of which is as follows:

[0043] The PROSAC algorithm is used to process the set of mutual nearest neighbor matching pairs to obtain the homography transformation matrix from the real-time image to the candidate region image patch. It also outputs the interior point set, which consists of optimized matching point pairs;

[0044] Step Six, the specific process is as follows:

[0045] Step 601: Set the homogeneous coordinates of the center pixel of the real-time image as follows: ,and ; and according to This yields the homogeneous coordinates of the center of the real-time image mapped onto the candidate region image patch. The center of the real-time image is mapped to the local pixel coordinates on the candidate region image patch. ;in, , and They are respectively The three components;

[0046] Step 602, according to This allows us to obtain the global pixel coordinates of the center of the real-time image mapped onto the reference image. It also maps the center of the real-time graph to the global pixel coordinates on the baseline graph. Converting to geodetic latitude and longitude coordinates allows for the positioning of the aircraft.

[0047] The above-mentioned image matching and localization method for aircraft based on pose-guided candidate regions further requires training the XFeat lightweight network before step three. The specific process is as follows:

[0048] Step A: Establish a lightweight network architecture for XFeat;

[0049] Step B: Perform perspective transformation and photometric transformation on any visible light image in the VEDAI dataset to obtain the first transformed image; at the same time, perform perspective transformation, photometric transformation and thin plate spline transformation on the corresponding infrared image to obtain the second transformed image.

[0050] Step C: Perform grayscale processing on the first transformed image and the second transformed image respectively to obtain the first transformed grayscale image and the second transformed grayscale image;

[0051] Step D: Perform instance normalization on the first transformed grayscale image and the second transformed grayscale image respectively to obtain the first instance normalized image and the second instance normalized image.

[0052] Step E: Input the normalized image of the first instance and the normalized image of the second instance into the XFeat lightweight network architecture for training to obtain the XFeat lightweight network.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] 1. This invention obtains candidate region image patches from a reference image based on the aircraft's pose, thereby transforming the global matching problem in a large-scale image into an image matching problem within a small area. This transformation reduces the search range for scene matching.

[0055] 2. Under low-security conditions, this invention solves the possible observation area of ​​flight through visual geometry, effectively narrowing the range of the matching zone; and performs perspective transformation based on the observation angle of the aircraft, reducing the perspective difference between the reference image and the real-time image.

[0056] 3. This invention expands the calculated corner positions by using an expansion factor to increase the field of view, ensuring that the candidate region image patch completely encompasses the corresponding region of the real-time image, while avoiding over-cropping issues that might occur with adding fixed safety boundaries. Candidates typically occupy only a small portion of the reference image area, thus laying an efficient foundation for subsequent processing.

[0057] 4. This invention uses the XFeat lightweight network to extract features from the real-time image of the aircraft and candidate region image blocks. By using the XFeat lightweight network and based on the intelligent matching method, robust matching is achieved when facing weak textures and occlusions, which improves the performance of the matching method and significantly enhances robustness and accuracy.

[0058] 5. This invention employs the PROSAC algorithm, prioritizing sampling from high-confidence matching pairs to complete the homography transformation matrix. Solve to effectively resist the interference of mismatches and obtain an optimized set of interior points with matching point pairs.

[0059] In summary, the method of this invention is simple in steps. Based on the pose of the aircraft, candidate region image patches containing the real-time image of the aircraft are obtained from the reference image. The real-time image and candidate region image patches are then used to extract key point features and perform feature matching through the XFeat lightweight network and the nearest neighbor matching method. This reduces the search range of scene matching, improves computational efficiency and matching robustness, and thus improves the positioning accuracy.

[0060] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0061] Figure 1 This is a flowchart of the present invention.

[0062] Figure 2 This is a schematic diagram of the baseline image and candidate region image blocks of the present invention.

[0063] Figure 3 This is a schematic diagram of the matching results of the present invention. Detailed Implementation

[0064] like Figure 1 The method shown is an image matching and localization method for aircraft based on pose-guided candidate regions. The method includes the following steps:

[0065] Step 1: Obtain the pixel coordinates of the corner points of the aircraft's field of view mapped onto the reference map based on the aircraft's pose;

[0066] Step 2: Based on the pixel coordinates of the corner points mapped to the reference image, obtain candidate region image patches from the reference image;

[0067] Step 3: Use the XFeat lightweight network to extract features from the real-time image of the aircraft and candidate region image blocks to obtain descriptor feature maps and pixel coordinates of the final key points; wherein, the XFeat lightweight network includes a descriptor generation network branch and a key point detection network branch;

[0068] Step 4: Obtain the set of mutual nearest neighbor pairs:

[0069] Based on the pixel coordinates and descriptor feature maps of the final key points, the mutual nearest neighbor matching method is used to match the descriptor feature vectors of the final key points in the real-time image and the candidate region image blocks to obtain a set of mutual nearest neighbor matching pairs.

[0070] Step 5: Based on the set of mutual nearest neighbor matching pairs, the PROSAC algorithm is used to obtain the homography transformation matrix from the real-time image to the candidate region image patch and the optimized matching point pairs;

[0071] Step 6: Based on the homography transformation matrix, project the center pixel coordinates of the real-time image onto the candidate region image block, and then convert them into geodetic latitude and longitude coordinates to achieve the positioning of the aircraft.

[0072] In this embodiment, step one, the specific process is as follows:

[0073] Step 101, according to The horizontal field of view of the camera on the aircraft was obtained. and vertical field of view ;in, This indicates the camera's focal length, expressed in meters. Indicates pixel size, in meters per pixel; The image resolution is represented by W (width) and H (height), with units of pixels.

[0074] Step 102, according to The expanded horizontal field of view is obtained. and the expanded vertical field of view ;in, Indicates the expansion factor. The value range is 1.02 to 1.05;

[0075] Step 103, according to , , and This yields the direction vectors of the four corner points (top right, top left, bottom left, and bottom right) in the camera coordinate system. Where i is a positive integer, and i can take the value 1, 2, 3 or 4;

[0076] Step 104, according to This yields the direction vector of the i-th corner point transformed into the navigation coordinate system. ;in, This represents the rotation matrix from the camera coordinate system to the body coordinate system. This represents the rotation matrix from the navigation coordinate system to the body coordinate system;

[0077] Step 105: Transform the i-th corner point into the direction vector in the navigation coordinate system. After unfolding, you get ;in, Indicates the position of the i-th corner point in the navigation coordinate system Axial components, Indicates the position of the i-th corner point in the navigation coordinate system Axial components, Indicates the position of the i-th corner point in the navigation coordinate system Axial components;

[0078] Step 106, according to The scaling factor for the i-th corner point is obtained; where the ground plane is represented in the navigation coordinate system as... =-h, where h is the aircraft altitude;

[0079] Step 107: Transform the direction vector of the i-th corner point into the navigation coordinate system, and obtain the i-th ground point intersecting the ground plane through perspective projection. Then, based on... To obtain the coordinates of the i-th ground point in the navigation coordinate system. ;in, Indicates the position of the i-th ground point in the navigation coordinate system Axial components, Indicates the position of the i-th ground point in the navigation coordinate system Axial components; T indicates transpose;

[0080] Step 108, according to Obtain the latitude and longitude of the i-th corner point. ;in, The average radius of the Earth Pi The latitude and longitude of the spacecraft's center of mass;

[0081] Step 109, according to The geographic coordinates of the i-th corner point are mapped to the pixel coordinates of the base map. ;in, This represents the latitude and longitude coordinates of the upper right corner of the reference map. This represents the latitude and longitude coordinates of the lower left corner of the reference map. This indicates the width and height pixel dimensions of the baseline image.

[0082] In this embodiment, step two is as follows:

[0083] Step 201: Based on the pixel coordinates of the four corner points mapped to the reference image, according to... ,get The enclosed candidate area; among which, This represents the floor function. This represents the floor function. Represents the minimum x-coordinate. Indicates the maximum x-coordinate. Represents the minimum y-coordinate. Indicates the maximum ordinate;

[0084] Step 202, based on The enclosed candidate region is cropped from the reference image to obtain candidate region image blocks.

[0085] In this embodiment, step three is as follows:

[0086] Step 301: Input the real-time graph into the descriptor generation network branch of the XFeat lightweight network to obtain the descriptor feature map and reliability heatmap;

[0087] Step 302: Divide the real-time image into 8×8 grid regions, input each grid region into the keypoint detection network branch of the XFeat lightweight network for processing, and output a 65-dimensional vector; where the first 64 dimensions represent probability values ​​and the 65th dimension represents the probability of no keypoints.

[0088] Step 303: Reshape the first 64-dimensional vector into a local keypoint heatmap; and stitch together each local keypoint heatmap to form a global keypoint heatmap;

[0089] Step 304: For the global key point heatmap, set the probability value of the grid area with no key point probability greater than 0.5 to zero to obtain the filtered global key point heatmap;

[0090] Step 305: Use the Non-Maximum Suppression (NMS) method to evaluate the filtered global key point heatmap and obtain the key point grid coordinates; wherein, the size of the NMS window is 5×5;

[0091] Step 306: Obtain the reliability value corresponding to the grid coordinates of the key point from the reliability heatmap; and obtain the score corresponding to the key point according to score=w1×a′+w2×b′; where w1 represents the first weight, a′ is the probability value corresponding to the grid coordinates of the key point, w2 represents the second weight, and b′ is the reliability value corresponding to the grid coordinates of the key point; the values ​​of w1 and w2 are 0 to 1, and w1+w2=1;

[0092] Step 307: Sort the scores corresponding to the key points in descending order, and take the first K key points as the final key points. This will give you the descriptor feature map corresponding to the real-time image and the pixel coordinates of the final key points; where K is a positive integer.

[0093] Step 308: Input the candidate region image block and process it according to the methods in steps 301 to 307 to obtain the descriptor feature map and the pixel coordinates of the final key point corresponding to the candidate region image block.

[0094] In this embodiment, step four is as follows:

[0095] Step 401: Map the pixel coordinates of the final key points corresponding to the real-time image onto the descriptor feature map to obtain the mapped coordinates of the final key points;

[0096] Step 402: On the descriptor feature map, obtain the 4-neighbor coordinates that surround the mapped coordinates of the final key point, and obtain the Euclidean distances between the left pixel coordinate, right pixel coordinate, upper pixel coordinate, lower pixel coordinate and the mapped coordinates of the final key point in the 4-neighbor coordinates, denoted as a, b, c, d respectively; and obtain the descriptor feature vector corresponding to the 4-neighbor coordinates from the descriptor feature map.

[0097] Step 403: According to A = a × A1 + b × A2 + c × A3 + d × A4, obtain the descriptor feature vector A of the final key point; where A1 represents the descriptor feature vector corresponding to the coordinates of the left pixel; A2 represents the descriptor feature vector corresponding to the coordinates of the right pixel; A3 represents the descriptor feature vector corresponding to the coordinates of the upper pixel; and A4 represents the descriptor feature vector corresponding to the coordinates of the lower pixel.

[0098] Step 404: Repeat steps 401 to 403 multiple times to obtain the descriptor feature vector for each final key point;

[0099] Step 405: Following the methods of steps 401 to 404, obtain the descriptor feature vector of each final key point in the candidate region image block;

[0100] Step 406: Using the mutual nearest neighbor matching method, based on the cosine similarity between the descriptor feature vector of each final key point in the real-time image and the descriptor feature vector of each final key point in the candidate region image block, if the cosine similarity is greater than 0.85, the corresponding final key points in the real-time image and the candidate region image block are recorded as mutual nearest neighbor matching pairs, thus obtaining the set of mutual nearest neighbor matching pairs.

[0101] In this embodiment, step five, the specific process is as follows:

[0102] The PROSAC algorithm is used to process the set of mutual nearest neighbor matching pairs to obtain the homography transformation matrix from the real-time image to the candidate region image patch. It also outputs the interior point set, which consists of optimized matching point pairs;

[0103] Step Six, the specific process is as follows:

[0104] Step 601: Set the homogeneous coordinates of the center pixel of the real-time image as follows: ,and ; and according to This yields the homogeneous coordinates of the center of the real-time image mapped onto the candidate region image patch. The center of the real-time image is mapped to the local pixel coordinates on the candidate region image patch. ;in, , and They are respectively The three components;

[0105] Step 602, according to This allows us to obtain the global pixel coordinates of the center of the real-time image mapped onto the reference image. It also maps the center of the real-time graph to the global pixel coordinates on the baseline graph. Converting to geodetic latitude and longitude coordinates allows for the positioning of the aircraft.

[0106] In this embodiment, before step three, the XFeat lightweight network needs to be trained. The specific process is as follows:

[0107] Step A: Establish a lightweight network architecture for XFeat;

[0108] Step B: Perform perspective transformation and photometric transformation on any visible light image in the VEDAI dataset to obtain the first transformed image; at the same time, perform perspective transformation, photometric transformation and thin plate spline transformation on the corresponding infrared image to obtain the second transformed image.

[0109] Step C: Perform grayscale processing on the first transformed image and the second transformed image respectively to obtain the first transformed grayscale image and the second transformed grayscale image;

[0110] Step D: Perform instance normalization on the first transformed grayscale image and the second transformed grayscale image respectively to obtain the first instance normalized image and the second instance normalized image.

[0111] Step E: Input the normalized image of the first instance and the normalized image of the second instance into the XFeat lightweight network architecture for training to obtain the XFeat lightweight network.

[0112] In this embodiment, both the camera coordinate system and the pixel coordinate system are based on coordinate systems conventional in the art.

[0113] In this embodiment, the body coordinate system The origin is the center of mass of the spacecraft, and points passing through the origin and pointing forward of the spacecraft are... The axis passes through the origin and intersects with the axis. The axis is perpendicular to and points to the right side of the aircraft. The axis passes through the origin and intersects with the axis. shaft and All axes are vertical and point downwards from the aircraft. Axis; where the optical center of the camera coincides with the center of mass of the spacecraft;

[0114] Navigation coordinate system The origin is the center of mass of the spacecraft, and the direction pointing due east from the origin is... The axis, passing through the origin and pointing due north, is The axis, passing through the origin and pointing upwards, is parallel to... shaft and All axes are perpendicular to .

[0115] In this embodiment, the following settings are provided: The value ranges from 1.02 to 1.05, which is equivalent to expanding the field of view by 2% to 5%. This strategy of expanding the field of view ensures that the cropping area completely contains the real-time image content.

[0116] In this embodiment, step one solves the possible observation area of ​​the flight through visual geometry, which effectively narrows the range of the matching area; and the perspective transformation is performed according to the observation perspective of the aircraft, which reduces the perspective difference between the reference image and the real-time image.

[0117] In this embodiment, the rotation matrix from the camera coordinate system to the body coordinate system and the rotation matrix from the navigation coordinate system to the body coordinate system Obtained using conventional methods in this field.

[0118] In this embodiment, preprocessing steps B to D generate two heterogeneous images. The coordinate correspondence between these two heterogeneous images can be accurately obtained through the transformation matrix used in the preprocessing process, providing a basis for supervised training of the network matching performance, thereby improving the generalization ability of the network model and providing supervision for subsequent matching tasks.

[0119] In this embodiment, it should be noted that in step three, the real-time image and candidate region image patches need to be normalized in grayscale and instance before being input into the XFeat lightweight network for feature extraction. The real-time image is an infrared image, and the candidate region image patches are visible light images.

[0120] In this embodiment, the average radius of the Earth It is 6,371,000 meters.

[0121] In this embodiment, the size of the descriptor feature map is W / 8×H / 8×64, the size of the reliability heatmap is W / 8×H / 8×1, the size of the local key point heatmap is 8×8×1, and the size of the global key point heatmap is H×W×1.

[0122] In this embodiment, it should be noted that the first weight w1 is 0.3 and the second weight w2 is 0.7. By weighting the probability value and the reliability value, the final key point selection is achieved, allowing XFeat to output truly useful, reliable and efficient features while maintaining a lightweight design, thereby improving the subsequent matching effect. In addition, it improves the natural distribution of key points, avoids excessive density in local areas, and ensures global coverage.

[0123] In this embodiment, it should be noted that the descriptor feature vector of each pixel on the descriptor feature map is 64×1, so the descriptor feature vector of the final key point is 64×1.

[0124] In this embodiment, it should be noted that K is set to 1024, and can be adjusted according to actual needs.

[0125] In this embodiment, it should be noted that when training the XFeat lightweight network, the total number of training samples is 5000 and the total number of training rounds is 150.

[0126] In this embodiment, it should be noted that the real-time image and reference image datasets from the large tilt angle real-time image pixel-level positioning competition area of ​​the 2025 National Aircraft Visual Navigation Challenge were used for testing. The real-time image is an infrared image with a resolution of 1280×1024, i.e., W is 1280 and H is 1024. The reference image is a visible light image with a resolution of 8000×8000.

[0127] like Figure 2 As shown, (a) is the original image of the baseline map, and (b) is a candidate region image patch obtained from the baseline map. Figure 3 As shown, the matching results of the candidate region image patch (b) and the real-time image (c) after passing through the PROSAC algorithm.

[0128] In summary, the method of this invention is simple in steps. Based on the pose of the aircraft, candidate region image patches containing the real-time image of the aircraft are obtained from the reference image. The real-time image and candidate region image patches are then used to extract key point features and perform feature matching through the XFeat lightweight network and the nearest neighbor matching method. This reduces the search range of scene matching, improves computational efficiency and matching robustness, and thus improves the positioning accuracy.

[0129] The above description is merely a preferred embodiment of the present invention and does not constitute any limitation on the present invention. Any simple modifications, alterations, or equivalent structural changes made to the above embodiments based on the technical essence of the present invention shall still fall within the protection scope of the present invention.

Claims

1. An image matching and localization method for aircraft based on pose-guided candidate regions, characterized in that, The method includes the following steps: Step 1: Obtain the pixel coordinates of the corner points of the aircraft's field of view mapped onto the reference map based on the aircraft's pose; Step 2: Based on the pixel coordinates of the corner points mapped to the reference image, obtain candidate region image patches from the reference image; Step 3: Use the XFeat lightweight network to extract features from the real-time image of the aircraft and candidate region image blocks to obtain descriptor feature maps and pixel coordinates of the final key points; wherein, the XFeat lightweight network includes a descriptor generation network branch and a key point detection network branch; Step 4: Obtain the set of mutual nearest neighbor pairs: Based on the pixel coordinates and descriptor feature maps of the final key points, the mutual nearest neighbor matching method is used to match the descriptor feature vectors of the final key points in the real-time image and the candidate region image blocks to obtain a set of mutual nearest neighbor matching pairs. Step 5: Based on the set of mutual nearest neighbor matching pairs, the PROSAC algorithm is used to obtain the homography transformation matrix from the real-time image to the candidate region image patch and the optimized matching point pairs; Step 6: Based on the homography transformation matrix, project the center pixel coordinates of the real-time image onto the candidate region image patch, and then convert them into geodetic latitude and longitude coordinates to achieve the positioning of the aircraft. Step one, the specific process is as follows: Step 101, according to The horizontal field of view of the camera on the aircraft was obtained. and vertical field of view ;in, This indicates the camera's focal length, expressed in meters. Indicates pixel size, in meters per pixel; The image resolution is represented by W (width) and H (height), with units of pixels. Step 102, according to The expanded horizontal field of view is obtained. and the expanded vertical field of view ;in, Indicates the expansion factor. The value range is 1.02 to 1.05; Step 103, according to , , and This yields the direction vectors of the four corner points (top right, top left, bottom left, and bottom right) in the camera coordinate system. Where i is a positive integer, and i can take the value 1, 2, 3 or 4; Step 104, according to This yields the direction vector of the i-th corner point transformed into the navigation coordinate system. ;in, This represents the rotation matrix from the camera coordinate system to the body coordinate system. This represents the rotation matrix from the navigation coordinate system to the body coordinate system; Step 105: Transform the i-th corner point into the direction vector in the navigation coordinate system. After unfolding, you get ;in, Indicates the position of the i-th corner point in the navigation coordinate system Axial components, Indicates the position of the i-th corner point in the navigation coordinate system Axial components, Indicates the position of the i-th corner point in the navigation coordinate system Axial components; Step 106, according to The scaling factor for the i-th corner point is obtained; where the ground plane is represented in the navigation coordinate system as... =-h, where h is the aircraft altitude; Step 107: Transform the direction vector of the i-th corner point into the navigation coordinate system, and obtain the i-th ground point intersecting the ground plane through perspective projection. Then, based on... To obtain the coordinates of the i-th ground point in the navigation coordinate system. ;in, Indicates the position of the i-th ground point in the navigation coordinate system Axial components, Indicates the position of the i-th ground point in the navigation coordinate system Axial components; T indicates transpose; Step 108, according to Obtain the latitude and longitude of the i-th corner point. ;in, The average radius of the Earth Pi The latitude and longitude of the spacecraft's center of mass; Step 109, according to The geographic coordinates of the i-th corner point are mapped to the pixel coordinates of the base map. ;in, This represents the latitude and longitude coordinates of the upper right corner of the reference map. This represents the latitude and longitude coordinates of the lower left corner of the reference map. This indicates the width and height pixel dimensions of the baseline image.

2. The image matching and localization method for aircraft based on pose-guided candidate regions according to claim 1, characterized in that: Step two, the specific process is as follows: Step 201: Based on the pixel coordinates of the four corner points mapped to the reference image, according to... ,get The enclosed candidate area; among which, This represents the floor function. This represents the floor function. Represents the minimum x-coordinate. Indicates the maximum x-coordinate. Represents the minimum y-coordinate. Indicates the maximum ordinate; Step 202, based on The enclosed candidate region is cropped from the reference image to obtain candidate region image blocks.

3. The image matching and localization method for aircraft based on pose-guided candidate regions according to claim 2, characterized in that: Step three, the specific process is as follows: Step 301: Input the real-time graph into the descriptor generation network branch of the XFeat lightweight network to obtain the descriptor feature map and reliability heatmap; Step 302: Divide the real-time image into 8×8 grid regions, input each grid region into the keypoint detection network branch of the XFeat lightweight network for processing, and output a 65-dimensional vector; where the first 64 dimensions represent probability values ​​and the 65th dimension represents the probability of no keypoints. Step 303: Reshape the first 64-dimensional vector into a local keypoint heatmap; and stitch together each local keypoint heatmap to form a global keypoint heatmap; Step 304: For the global key point heatmap, set the probability value of the grid area with no key point probability greater than 0.5 to zero to obtain the filtered global key point heatmap; Step 305: Use the Non-Maximum Suppression (NMS) method to evaluate the filtered global key point heatmap and obtain the key point grid coordinates; wherein, the size of the NMS window is 5×5; Step 306: Obtain the reliability value corresponding to the grid coordinates of the key point from the reliability heatmap; and obtain the score corresponding to the key point according to score=w1×a′+w2×b′; where w1 represents the first weight, a′ is the probability value corresponding to the grid coordinates of the key point, w2 represents the second weight, and b′ is the reliability value corresponding to the grid coordinates of the key point; the values ​​of w1 and w2 are 0 to 1, and w1+w2=1; Step 307: Sort the scores corresponding to the key points in descending order, and take the first K key points as the final key points. This will give you the descriptor feature map corresponding to the real-time image and the pixel coordinates of the final key points; where K is a positive integer. Step 308: Input the candidate region image block and process it according to the methods in steps 301 to 307 to obtain the descriptor feature map and the pixel coordinates of the final key point corresponding to the candidate region image block.

4. The image matching and localization method for aircraft based on pose-guided candidate regions according to claim 3, characterized in that: Step four, the specific process is as follows: Step 401: Map the pixel coordinates of the final key points corresponding to the real-time image onto the descriptor feature map to obtain the mapped coordinates of the final key points; Step 402: On the descriptor feature map, obtain the 4-neighbor coordinates that surround the mapped coordinates of the final key point, and obtain the Euclidean distances between the left pixel coordinate, right pixel coordinate, upper pixel coordinate, lower pixel coordinate and the mapped coordinates of the final key point in the 4-neighbor coordinates, denoted as a, b, c, d respectively; and obtain the descriptor feature vector corresponding to the 4-neighbor coordinates from the descriptor feature map. Step 403: According to A = a × A1 + b × A2 + c × A3 + d × A4, obtain the descriptor feature vector A of the final key point; where A1 represents the descriptor feature vector corresponding to the coordinates of the left pixel; A2 represents the descriptor feature vector corresponding to the coordinates of the right pixel; A3 represents the descriptor feature vector corresponding to the coordinates of the upper pixel; and A4 represents the descriptor feature vector corresponding to the coordinates of the lower pixel. Step 404: Repeat steps 401 to 403 multiple times to obtain the descriptor feature vector for each final key point; Step 405: Following the methods of steps 401 to 404, obtain the descriptor feature vector of each final key point in the candidate region image block; Step 406: Using the mutual nearest neighbor matching method, based on the cosine similarity between the descriptor feature vector of each final key point in the real-time image and the descriptor feature vector of each final key point in the candidate region image block, if the cosine similarity is greater than 0.85, the corresponding final key points in the real-time image and the candidate region image block are recorded as mutual nearest neighbor matching pairs, thus obtaining the set of mutual nearest neighbor matching pairs.

5. The image matching and localization method for aircraft based on pose-guided candidate regions according to claim 4, characterized in that: Step 5, the specific process is as follows: The PROSAC algorithm is used to process the set of mutual nearest neighbor matching pairs to obtain the homography transformation matrix from the real-time image to the candidate region image patch. It also outputs the interior point set, which consists of optimized matching point pairs; Step Six, the specific process is as follows: Step 601: Set the homogeneous coordinates of the center pixel of the real-time image as follows: ,and ; and according to This yields the homogeneous coordinates of the center of the real-time image mapped onto the candidate region image patch. The center of the real-time image is mapped to the local pixel coordinates on the candidate region image patch. ;in, , and They are respectively The three components; Step 602, according to This allows us to obtain the global pixel coordinates of the center of the real-time image mapped onto the reference image. It also maps the center of the real-time graph to the global pixel coordinates on the baseline graph. Converting to geodetic latitude and longitude coordinates allows for the positioning of the aircraft.

6. The image matching and localization method for aircraft based on pose-guided candidate regions according to claim 1, characterized in that: Before step three, the XFeat lightweight network needs to be trained. The specific process is as follows: Step A: Establish a lightweight network architecture for XFeat; Step B: Perform perspective transformation and photometric transformation on any visible light image in the VEDAI dataset to obtain the first transformed image; at the same time, perform perspective transformation, photometric transformation and thin plate spline transformation on the corresponding infrared image to obtain the second transformed image. Step C: Perform grayscale processing on the first transformed image and the second transformed image respectively to obtain the first transformed grayscale image and the second transformed grayscale image; Step D: Perform instance normalization on the first transformed grayscale image and the second transformed grayscale image respectively to obtain the first instance normalized image and the second instance normalized image. Step E: Input the normalized image of the first instance and the normalized image of the second instance into the XFeat lightweight network architecture for training to obtain the XFeat lightweight network.

Images