A visual navigation positioning method and system based on heterogeneous image matching and a readable storage medium

Abstract

The application provides a visual navigation positioning method and system based on heterogeneous image matching, and a readable storage medium, belongs to the technical field of image registration and visual navigation, and is suitable for unmanned aerial vehicle autonomous navigation positioning in a GPS denial environment. The application first completes high-precision satellite reference map block production; scale normalization and distortion correction are carried out on real-time aerial images to eliminate perspective distortion caused by large-angle maneuver; CFOG dense features are used to complete global coarse matching of heterogeneous images, and then RIFT sparse features are combined to perform local fine matching; the positions and postures of the unmanned aerial vehicle are solved through Gauss projection and Newton iteration inversion of multiple groups of control points. The application effectively overcomes the problems of nonlinear gray difference and geometric distortion of heterogeneous images, and improves the navigation positioning precision, robustness and real-time performance.

Description

Technical Field

[0001] This invention relates to the fields of image registration and visual navigation technology, and in particular to a visual navigation and positioning method, system and readable storage medium based on heterogeneous image matching. Background Technology

[0002] Currently, the navigation systems commonly used by UAVs are multi-source fusion navigation systems that integrate satellite, inertial, and other information sources. However, in real-world combat conditions, the electromagnetic environment is complex, and GPS signals are highly susceptible to interference and attacks. Navigation systems that heavily rely on external positioning information are inherently unsafe. Visual-inertial integrated navigation systems, relying on image registration technology, can serve as an alternative to GPS integrated navigation and positioning systems, significantly improving the autonomy and safety of UAV navigation systems.

[0003] Extensive research has been conducted both domestically and internationally on visual navigation and positioning technologies. The main approach involves seeking correspondences between images based on their similarity by matching local features. Extracted features can be points, contours, edges, regions, and shape context. Currently, Accelerated Robust Features (SURF) and Scale Invariant Feature Transform are widely used for remote sensing image matching. Although these features are robust to scaling, rotation, and linear intensity changes, they are not effective in matching heterogeneous images because they are weakly sensitive to nonlinear differences.

[0004] Despite decades of research in visual navigation and localization, matching heterogeneous images (such as optical, synthetic aperture radar (SAR), and maps) remains a challenge due to significant differences in nonlinear intensity among these data.

[0005] Therefore, it is necessary to study a visual navigation and localization method, system, and readable storage medium based on heterogeneous image matching to address the shortcomings of existing technologies and solve or mitigate one or more of the aforementioned problems. Summary of the Invention

[0006] In view of this, the present invention provides a visual navigation and localization method based on heterogeneous image matching, applicable to aircraft equipped with airborne cameras, comprising the following steps: S1. Real-time image acquisition: Acquire real-time images through the airborne camera; S2. Reference map creation: Combining aircraft flight path and continuous pose data, a preset range is captured on the satellite image with the current pose as the center to serve as the satellite reference map. S3. Perform distortion correction on the real-time image: derive the mapping relationship between the real image and the distorted image through the distortion model, and resample the corrected image using bilinear interpolation to obtain the corrected image; S4. Image matching: The matching step adopts a hierarchical strategy combining global coarse matching and local fine matching. The global coarse matching performs feature extraction and matching between the reference image and the real-time image processed in step S3 based on the CFOG algorithm. Based on the global coarse matching result, the transformation matrix from the real-time image coordinates to the reference image coordinates is calculated, and the neighborhood image of the center point is extracted according to the transformation matrix, and then the neighborhood image is subjected to local fine matching. S5: Multi-control point inversion solution: Select multiple matching points with latitude and longitude information, perform Gaussian projection, construct a nonlinear equation system, and use the inertial navigation output as the initial value to perform Newton iteration to obtain the spatial coordinates and attitude of the UAV in Gaussian frame.

[0007] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein S3 specifically includes: image distortion correction: obtaining the camera intrinsic parameter matrix based on the physical length of the camera unit pixel and the focal length, obtaining the rotation matrix and translation vector based on the flight attitude, calculating the physical equivalent coordinates of the corrected image points, calculating the image plane coordinates before correction using the physical equivalent coordinates and flight attitude parameters, completing the mapping from the pixel coordinates of the corrected image to the pixel coordinates of the image before correction, and resampling the corrected image using bilinear interpolation to obtain the corrected image; wherein the flight attitude includes pitch angle, roll angle and yaw angle, and the physical equivalent coordinates are the coordinates obtained by dividing the coordinates in the physical coordinate system by the aircraft flight altitude.

[0008] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the global coarse matching in S4 specifically involves: scaling and grayscale processing of the real-time image and the reference image; calculating the horizontal and vertical gradients based on the Sobel operator to obtain phase gradient features; performing Gaussian filtering on the phase gradient features; performing dense feature extraction and matching based on the CFOG algorithm; using SSD feature matching to calculate the sum of squares of the pixel differences between the real-time image and the reference image at corresponding coordinates; determining the region with the smallest sum of squares as the matching region; and using discrete Fourier transform to convert the time-domain circular convolution into frequency-domain dot product to accelerate computation. Here, the real-time image and the reference image are shrunk to 1 / 4 of their original size, and the Gaussian filtering kernel size is 3×3 to 7×7.

[0009] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the local fine matching in S4 specifically involves: based on the coarse matching result, solving the transformation matrix from real-time image coordinates to reference image coordinates; extracting the neighborhood image of the center point according to the transformation matrix; performing local fine matching based on the RIFT algorithm; generating a feature description matrix by statistically analyzing the information of the neighborhood points of the feature points on the tangent plane where the feature points are located; transforming the feature description matrix to generate feature description vectors for the feature points; using phase consistency to detect the feature points; and describing the features on the maximum exponential map, wherein the neighborhood image of the center point ranges from 50 to 200 pixels around the center point.

[0010] In addition to the aspects and any possible implementations described above, a further implementation is provided in which, in the S2 reference map production step, the preset range is centered on the current pose and extends to 2-3 times the field of view of the camera of the UAV in that pose. This range is cropped from the satellite image as the satellite reference map, and the resolution of the satellite reference map is 0.3-1 meters / pixel.

[0011] In addition to the aspects and any possible implementations described above, a further implementation is provided, wherein the multi-control point inversion solution in S5 specifically includes: selecting multiple successfully matched pixels in the image matching results of S4 as multi-point positioning inputs; using a reference map and a digital elevation map to obtain the latitude, longitude, and altitude information of the landmarks corresponding to the pixels; using Gaussian projection to transform them into the Gaussian system for positioning solution; constructing a nonlinear equation system using the coordinates of the matching points in the Gaussian system and the slant distance information from the matching points to the platform as observations; using the output of the inertial measurement element as the initial value for iterative positioning solution; using the Newton iteration method to iteratively solve using the Jacobi matrix; and inversely calculating the spatial coordinates and attitude of the UAV in the Gaussian system.

[0012] In addition to the aspects and any possible implementations described above, a further implementation is provided in which at least three matching points are selected in the S5 multi-control point inversion solution step, each matching point containing latitude and longitude information; the Newton iteration convergence condition is that the iteration error is less than a preset threshold or the number of iterations reaches a preset upper limit.

[0013] In addition to the aspects described above and any possible implementations, a further implementation is provided, which is applied to an aircraft visual navigation and positioning scenario under conditions where satellite navigation signals are limited.

[0014] In accordance with the aspects described above and any possible implementation, a visual navigation and positioning system based on heterogeneous image matching is further provided, comprising: an image acquisition module, a reference map production module, an image preprocessing module, a global coarse matching module, a local fine matching module, and a multi-control point inversion and solution module. The output of the image acquisition module is connected to the reference map production module and the image preprocessing module respectively, the output of the reference map production module is connected to the image preprocessing module, and the image preprocessing module, the global coarse matching module, the local fine matching module, and the multi-control point inversion and solution module are connected to each other in sequence. The image acquisition module is used to acquire real-time images via an airborne camera; The baseline image creation module is used to combine aircraft flight path and continuous pose data to capture a satellite baseline image with a resolution of 0.3-1 meter / pixel, centered on the current pose and within a range of 2-3 times the field of view of the UAV camera. The image preprocessing module is used to combine the pitch angle, roll angle, and yaw angle of the flight attitude to complete the solution of camera intrinsic parameters, coordinate mapping, and bilinear interpolation resampling for image distortion correction. The global coarse matching module is used for image processing including 1 / 4 scale scaling, grayscale conversion, Sobel gradient calculation, 3×3 to 7×7 kernel Gaussian filtering, CFOG dense feature matching, SSD matching, and discrete Fourier transform acceleration. The local fine matching module is used to extract a 50-200 pixel neighborhood image around the center point based on the coarse matching result, and complete the local fine matching based on the RIFT algorithm. The multi-control point inversion solution module is used to select at least 3 matching points with latitude and longitude information, and then perform Newton iteration solution by Gaussian projection, constructing a nonlinear equation system, and using the inertial navigation output as the initial value to obtain the spatial coordinates and attitude of the UAV in Gaussian frame. The system is suitable for aircraft operating under conditions where satellite navigation signals are limited.

[0015] In addition to the aspects described above and any possible implementations, a computer-readable storage medium is further provided, which is a non-volatile storage medium storing a computer program that, when executed by a processor, implements the visual navigation and positioning method based on heterogeneous image matching.

[0016] Compared with the prior art, the present invention can achieve the following technical effects: 1. Adapts to strong nonlinear differences in heterogeneous images, significantly improving matching robustness. This invention addresses the problems of gray-scale nonlinear differences, viewpoint distortion, scale changes, and rotational shifts that exist between airborne real-time optical images and satellite reference images. It abandons the shortcomings of traditional SIFT and SURF features in adapting to heterogeneous images and adopts a two-level matching architecture of distortion correction + CFOG dense coarse matching + RIFT sparse fine matching. Combined with phase gradient feature and phase consistency feature detection, it effectively suppresses the interference caused by gray-scale inconsistency, texture loss, and illumination changes in heterogeneous images, and significantly improves the registration success rate and matching accuracy of optical and satellite heterogeneous images. It can also maintain stable matching performance in complex terrain and light and shadow change scenarios. 2. The multi-level matching architecture balances matching accuracy and real-time computation. This invention employs a hierarchical strategy of global coarse matching + local fine matching: First, the image is reduced to 1 / 4 scale, Gaussian filtering is used for noise reduction, and SSD matching is combined with discrete Fourier transform for frequency domain acceleration. The time-domain convolution is converted into frequency-domain dot product, which significantly reduces the computational load of global search and improves the matching speed by more than 10 times compared with traditional traversal matching. Then, a small-range neighborhood fine matching is performed within the coarse matching locked area. This avoids the computational redundancy of global full-image fine matching, constrains the matching search range, and eliminates false matching. It achieves both fast coarse positioning and high-precision fine positioning, meeting the real-time navigation computing requirements of UAV airborne embedded platforms. 3. High-precision image distortion correction to eliminate flight attitude imaging deviations. This invention constructs a rotation matrix and translation vector based on the camera intrinsic parameter matrix and flight pitch / roll / yaw angles. It introduces physically equivalent coordinates to achieve accurate mapping of pixel coordinates before and after correction. Combined with bilinear interpolation resampling, it effectively corrects image geometric distortion and projection distortion caused by UAV flight attitude tilt, altitude change, and lens perspective. This makes the imaging perspective of the airborne real-time image and the satellite reference image more consistent, reducing the difficulty of heterogeneous image registration from the source and laying a high-quality image foundation for subsequent feature matching. 4. High accuracy in navigation, positioning, and attitude calculation through multi-control point joint inversion. This invention selects at least three high-precision matching control points, integrates satellite reference images and digital elevation maps to obtain real geographic information of landmark latitude, longitude, and altitude, and models them using Gaussian projection to unify them into a Gaussian plane coordinate system. A nonlinear equation system is constructed using the geographic coordinates of the matching points and the slant distance of the airborne platform as observations. The output of the inertial measurement element is used as the initial value for iteration, and iterative solutions are obtained by combining Newton iteration and Jacobi matrix. Compared with single-point positioning methods, this invention effectively suppresses positioning drift caused by single-point matching errors and noise interference. It can simultaneously calculate the high-precision spatial position and flight attitude of the UAV, and its positioning stability and anti-interference capability are significantly better than traditional single-feature-point visual positioning methods. 5. It does not rely on satellite navigation signals and has strong independent anti-interference capabilities. In scenarios where satellite navigation signals are limited or fail due to complex electromagnetic interference, obstruction, or denial, this invention does not rely on external satellite positioning signals such as GPS. It can achieve autonomous navigation and positioning throughout the entire process by relying solely on airborne visual imaging, satellite reference image matching, and inertial navigation information fusion. This completely solves the shortcomings of traditional integrated navigation, such as excessive reliance on satellite signals, susceptibility to interference, and susceptibility to failure. It greatly improves the navigation autonomy, survivability, and mission reliability of UAVs in harsh scenarios such as complex electromagnetic environments on the battlefield, deep mountain valleys, and urban high-rise buildings. 6. The baseline map cropping strategy is reasonable and suitable for long-endurance continuous navigation. This invention uses the current pose of the UAV as the center and intelligently captures a satellite reference image within a range of 2-3 times the camera's field of view. At the same time, it limits the resolution of the reference image to 0.3-1 meter / pixel. This avoids the problem of excessive storage and increased matching time caused by an excessively large reference image area, while also preventing matching failure due to an excessively small reference image area causing the real-time field of view to exceed the matching area. It is suitable for uninterrupted visual positioning under continuous flight and continuous pose changes of UAVs, and is suitable for continuous navigation operations of long-endurance, long-distance aircraft. 7. The parameters are highly adaptable to engineering applications, making them easy to deploy in airborne engineering. This invention provides optimal engineering constraints for key parameters such as image scaling ratio, Gaussian filter kernel size, fine matching neighborhood pixel range, Newton iteration convergence condition, and number of matching control points. The entire method is modularized and the parameters are standardized, making it adaptable to airborne cameras with different focal lengths and pixel specifications, as well as various types of fixed-wing and rotary-wing UAV platforms. The corresponding system adopts a modular architecture, with each unit having independent logic and clear data links, facilitating hardware integration, software porting, and embedded deployment, and meeting the requirements for lightweight, real-time airborne operation. 8. It integrates multiple technologies, has good fault tolerance, and is widely adaptable to various environments. This invention integrates multiple technologies such as camera calibration, image distortion correction, dense feature matching, sparse feature description, geographic projection transformation, nonlinear iterative calculation, and inertial navigation-assisted initial values ​​to form a closed-loop navigation calculation system. It has good fault tolerance for complex working conditions such as sparse terrain texture, similar surface features, drastic changes in flight attitude, and altitude fluctuations. It can be widely used in visual autonomous navigation scenarios in many fields such as agricultural and forestry inspection, surveying and mapping remote sensing, low-altitude security, emergency rescue, and military unmanned aerial vehicles.

[0017] Of course, any product implementing this invention does not necessarily need to achieve all of the technical effects described above at the same time. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart of a visual navigation and positioning system based on heterogeneous image matching provided in one embodiment of the present invention; Figure 2 This is a satellite reference image creation diagram provided in one embodiment of the present invention; Figure 3 This is a distortion correction effect diagram provided by an embodiment of the present invention; Figure 4 This is a comparison image of distortion correction effect provided by an embodiment of the present invention; Figure 5 This is a real-time graph feature extraction graph provided in one embodiment of the present invention; Figure 6 This is a baseline image feature extraction map provided in one embodiment of the present invention; Figure 7 This is a local fine-matching control point map provided in one embodiment of the present invention; Figure 8 This is a schematic diagram of multi-control point inversion positioning provided in one embodiment of the present invention. Detailed Implementation

[0020] To better understand the technical solution of the present invention, the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0021] It should be understood that the described embodiments are merely some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0022] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0023] This invention discloses a visual navigation and positioning method based on heterogeneous image matching, applicable to aircraft equipped with airborne cameras, comprising the following steps: Real-time image acquisition steps: Acquire real-time images using an airborne camera; The steps for creating the baseline image are as follows: Combining the aircraft flight path and continuous pose data, with the current pose as the center, and the range being 2-3 times the field of view of the UAV camera in that pose, this range is cropped from the satellite image as the satellite baseline image. Image distortion correction steps: Obtain the camera intrinsic parameter matrix based on the physical length of the camera unit pixel and the focal length; obtain the rotation matrix and translation vector based on the flight attitude; calculate the physical equivalent coordinates of the corrected image points; calculate the image plane coordinates before correction using the physical equivalent coordinates and flight attitude parameters; complete the mapping from the pixel coordinates of the corrected image to the pixel coordinates of the image before correction; resample the corrected image using bilinear interpolation to obtain the corrected image. Global coarse matching steps: The real-time image and the reference image are scaled and grayscaled. The horizontal and vertical gradients are calculated based on the Sobel operator to obtain the phase gradient features. The phase gradient features are Gaussian filtered. Dense feature extraction and matching are performed based on the CFOG algorithm. SSD feature matching is used. The sum of squares of the pixel differences between the real-time image and the reference image at the corresponding coordinates is calculated. The region with the smallest sum of squares is determined as the matching region. The time-domain circular convolution is converted into frequency-domain dot product using discrete Fourier transform to accelerate the calculation. Local fine matching steps: Based on the coarse matching results, calculate the transformation matrix from real-time image coordinates to reference image coordinates, extract the neighborhood image of the center point according to the transformation matrix, perform local fine matching based on the RIFT algorithm, generate a feature description matrix by statistically analyzing the information of the neighborhood points of the feature points in the tangent plane where the feature points are located, transform the feature description matrix to generate feature description vectors of the feature points, use phase consistency to detect feature points, and describe features on the maximum exponential map. Multi-control point inversion solution steps: Select multiple successfully matched pixels from the image matching results as multi-point positioning inputs. Use the benchmark map and digital elevation map to obtain the latitude, longitude, and altitude information of the landmarks corresponding to the pixels. Use Gaussian projection to transform them into the Gaussian system for positioning solution. Use the coordinates of the matching point in the Gaussian system and the slant distance information from the matching point to the platform as observations to construct a nonlinear equation system. Use the output of the inertial measurement element as the initial value for iterative positioning solution. Use the Newton iteration method to iteratively solve the problem using the Jacobi matrix to obtain the spatial coordinates and attitude of the UAV in the Gaussian system.

[0024] In one specific embodiment, in the image distortion correction step, the flight attitude includes pitch angle, roll angle and yaw angle, and the physical equivalent coordinates refer to the coordinates of the physical coordinate system divided by the aircraft's flight altitude.

[0025] In one specific embodiment, in the reference map production step, the resolution of the satellite reference map is 0.3-1 meter / pixel.

[0026] In one specific embodiment, in the global coarse matching step, the scaling is to reduce the real-time graph and the baseline graph to 1 / 4 of their original size.

[0027] In one specific embodiment, in the global coarse matching step, the filter kernel size of the Gaussian filter is 3×3 to 7×7.

[0028] In one specific embodiment, in the local fine matching step, the range of the neighborhood image of the center point is 50-200 pixels around the center point.

[0029] The number of successfully matched pixels is at least three, and each matching point contains latitude and longitude information.

[0030] In the multi-control point inversion solution step, the convergence condition of the Newton iteration is that the iteration error is less than a preset threshold or the number of iterations reaches a preset upper limit.

[0031] The method is applied to visual navigation and positioning of aircraft under conditions where satellite navigation signals are limited.

[0032] This invention also provides a visual navigation and positioning system based on heterogeneous image matching, applied to an aircraft equipped with an airborne camera, comprising: The image acquisition module is configured to acquire real-time images via an airborne camera. The baseline image creation module is configured to combine aircraft flight path and continuous pose data, take the current pose as the center, and take a range of 2-3 times the field of view of the camera of the UAV in that pose as the range, and crop this range on the satellite image as the satellite baseline image. The image preprocessing module is configured to obtain the camera intrinsic parameter matrix based on the physical length of the camera unit pixel and the focal length, obtain the rotation matrix and translation vector based on the flight attitude, calculate the physical equivalent coordinates of the corrected image points, calculate the image plane coordinates before correction using the physical equivalent coordinates and flight attitude parameters, complete the mapping from the pixel coordinates of the corrected image to the pixel coordinates of the image before correction, and resample the correction effect using bilinear interpolation to obtain the corrected image. The global coarse matching module is configured to scale and grayscale the real-time image and the reference image, calculate the horizontal and vertical gradients based on the Sobel operator to obtain phase gradient features, apply Gaussian filtering to the phase gradient features, perform dense feature extraction and matching based on the CFOG algorithm, use SSD feature matching, calculate the sum of squares of the pixel differences between the real-time image and the reference image at the corresponding coordinates, determine the region with the smallest sum of squares as the matching region, and use discrete Fourier transform to convert the time-domain circular convolution into frequency-domain dot product to accelerate the calculation. The local fine matching module is configured to calculate the transformation matrix from real-time image coordinates to reference image coordinates based on the coarse matching result, extract the neighborhood image of the center point according to the transformation matrix, perform local fine matching based on the RIFT algorithm, generate a feature description matrix by statistically analyzing the information of the neighborhood points of the feature point in the tangent plane where the feature point is located, transform the feature description matrix to generate the feature description vector of the feature point, use phase consistency to detect the feature point, and describe the feature on the maximum exponential map. The multi-control point inversion solution module is configured to select multiple successfully matched pixels in the image matching results as multi-point positioning inputs, use the reference map and digital elevation map to obtain the latitude, longitude and altitude information of the landmarks corresponding to the pixels, use Gaussian projection to transform them into the Gaussian system for positioning solution, use the coordinates of the matching point in the Gaussian system and the slant distance information from the matching point to the platform as observations to construct a nonlinear equation system, use the output of the inertial measurement element as the initial value for iterative positioning solution, use the Newton iteration method to iteratively solve using the Jacobi matrix, and inversely calculate the spatial coordinates and attitude of the UAV in the Gaussian system; The output of the image acquisition module is connected to the reference map creation module, the image preprocessing module, the reference map creation module, and the image preprocessing module. The output of the image preprocessing module is connected to the input of the global coarse matching module, the output of the global coarse matching module is connected to the input of the local fine matching module, and the output of the local fine matching module is connected to the input of the multi-control point inversion solution module.

[0033] In one specific embodiment, the reference map creation module is configured to take the current pose as the center, and take a range of 2-3 times the field of view of the camera of the UAV in that pose as the range, and crop this range on the satellite image as the satellite reference map. The resolution of the satellite reference map is 0.3-1 meters / pixel.

[0034] In one specific embodiment, in the image preprocessing module, the flight attitude includes pitch angle, roll angle and yaw angle, and the physical equivalent coordinates refer to the coordinates in the physical coordinate system divided by the aircraft's flight altitude.

[0035] In one specific embodiment, in the global coarse matching module, the scaling is to reduce the real-time graph and the reference graph to 1 / 4 of their original size, and the Gaussian filter kernel size is 3×3 to 7×7.

[0036] In one specific embodiment, in the local fine matching module, the range of the neighborhood image of the center point is 50-200 pixels around the center point.

[0037] In one specific embodiment, in the multi-control point inversion solution module, the multiple successfully matched pixels are at least 3 matching points, each matching point contains latitude and longitude information, and the convergence condition of the Newton iteration is that the iteration error is less than a preset threshold or the number of iterations reaches a preset upper limit.

[0038] In one specific embodiment, the system is applied to aircraft operating under conditions where satellite navigation signals are limited.

[0039] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the visual navigation and positioning method based on heterogeneous image matching.

[0040] Example 1:

[0041] To improve the accuracy and real-time performance of visual navigation and positioning systems based on heterogeneous image matching, this invention designs algorithms from aspects such as preprocessing, dense and sparse feature matching, multi-control point inversion positioning, and outlier removal. It proposes a visual navigation and positioning system based on heterogeneous image matching that exhibits high stability and real-time performance. Continuous testing results show stable and reliable performance. The system diagram is shown below. Figure 1 As shown.

[0042] A visual navigation and positioning method and system based on heterogeneous image matching, the method being used for visual navigation in GPS operation, the method comprising: S1. Reference Map Creation: This invention combines aircraft flight paths and continuous pose data to divide the aircraft's cruising area into a high-precision satellite reference map, such as... Figure 2 As shown.

[0043] S2. Image Preprocessing: Due to the large-angle maneuvering of the aircraft, the images taken of the ground will suffer from "nearer images appearing larger and farther images appearing smaller" distortion, changing the original pixel unit length and causing inconsistencies in pixel unit length within the same image, affecting geographic matching and positioning results. Therefore, the system first performs scale normalization and distortion correction on all real-time images (uniformly correcting them to orthophotos with vertical projection). The mapping relationship between the real image and the distorted image is derived through a distortion model, and bilinear interpolation is used to resample the correction effect to obtain the corrected image. The correction effect is shown in the image below. Figure 3 , Figure 4 As shown.

[0044] Figure 3 In the diagram, (1), (2), (3), and (4) represent the real-time image, the real-time image after distortion correction, the matching effect of the original image, and the matching effect after correction, respectively. It can be seen that after distortion correction, the "nearer is larger, farther is smaller" problem is solved in the real-time image, and the entire image scale is consistent. A comparison of the correction effects is shown below. Figure 4 As shown, Figure 4In the diagram, (1), (2), (3), and (4) represent the matching results of the real-time image, the baseline image, the original image, and the corrected matching result, respectively. It can be seen that when the input image itself is distorted and deformed, it can only align the features of the central area (the blue line), and cannot align all areas (the orange line). However, after the image distortion is corrected, it can align all areas and obtain a more accurate center point.

[0045] S3. Global Coarse Matching (Dense Features): In the global coarse matching stage, the satellite baseline image and the preprocessed real-time image are used for feature extraction and matching based on the CFOG algorithm. The feature extraction results of the real-time image and the baseline image are as follows: Figure 5 , Figure 6 As shown, the CFOG feature extraction method has good feature description ability and good characterization ability for nonlinear brightness distortion in multi-source images.

[0046] S4. Local Fine-Grained Matching (Sparse Features): Based on the coarse matching results, the transformation matrix from real-time image coordinates to reference image coordinates is calculated. Then, the neighborhood image of the center point is extracted based on the transformation matrix, and local fine-grained matching is performed on the neighborhood image. The results of the local fine-grained matching are as follows: Figure 7 As shown.

[0047] S5. Multi-control point inversion solution: Since the previous four steps yielded control point pairs with high accuracy, multiple successfully matched pixels (containing latitude and longitude information, at least 3 matching points are required) can be selected from the image matching results as input for multi-point localization. A schematic diagram of the multi-control point inversion solution is shown below. Figure 8 As shown.

[0048] The specific implementation process of this invention is as follows: 1. Baseline map creation: Visual navigation and positioning rely on high-precision satellite reference maps. If the reference map is too large, the matching difficulty and time increase; if the reference map is too small, the current field of view may not appear in the reference map, making matching impossible. This invention combines aircraft flight paths and continuous pose data to divide the aircraft's cruising area into high-precision satellite reference maps. Specifically, the reference map is a cropped area from the satellite image, centered on the current pose state and extending to 2-3 times the camera's field of view at that pose. Figure 2 As shown.

[0049] 2. Image preprocessing: Based on the physical length of a camera pixel and camera focal length The camera intrinsic parameter matrix is ​​obtained. As shown in equation (1).

[0050] in This represents the coordinates of the light source in the image pixel coordinate system. Indicates sensor shaft and The size of a unit pixel on the axis, This refers to the camera's focal length.

[0051] Corrected image points The corresponding physical equivalent coordinates are The physical equivalent coordinates refer to the coordinates of the physical coordinate system divided by the coordinates of the aircraft's flight altitude, and are expressed by equation (2).

[0052] Rotation matrix obtained from flight attitude Translation vector .

[0053] in These are the pitch angle, roll angle, and yaw angle of the UAV, respectively.

[0054] Based on the physical equivalent coordinates and flight attitude parameters obtained above, the coordinates of the image plane before correction are calculated by formula (7).

[0055] For each corrected coordinate By following steps 1 and 2, the coordinate transformation relationship is obtained, and the pixel coordinates of the image before correction are obtained. The process involves mapping the pixel coordinates of the corrected image back to those of the original image. After the coordinate transformation, bilinear interpolation is used to resample the image to account for the correction effects, thus obtaining the coordinates in the corrected image. The result is as follows Figure 3 As shown.

[0056] 3. Global coarse matching (dense features): The main steps in the global coarse matching stage are as follows: Scale scaling: The real-time and baseline plots are reduced to 1 / 4 of their original size.

[0057] Grayscale conversion: If it is a three-channel image, convert the image to grayscale.

[0058] Gradient calculation: The Sobel operator is used to calculate the horizontal and vertical gradients of the grayscale image. The phase gradient features of each pixel are calculated. Gaussian filtering is applied to the obtained phase gradient features to reduce noise and distortion interference.

[0059] SSD feature matching: This involves calculating the region where the sum of squared pixel differences between the real-time image and the reference image is minimized at corresponding coordinates. From the Discrete Fourier Transform of a 2D image, we know that circular convolution in the time domain can be considered as dot product in the frequency domain. Therefore, using the Discrete Fourier Transform accelerates this part of the calculation, increasing the matching speed by more than 10 times.

[0060] 4. Local fine-grained matching (sparse features): The system utilizes RIFT-based local fine-grained matching to generate a feature description matrix by statistically analyzing the information of neighboring points of a feature point within the tangent plane of that feature point. This matrix is ​​then transformed to generate feature description vectors for the feature points. By employing phase consistency (PC) detection and describing features on a maximum exponential map (MIM), robustness to rotation is achieved. The control point extraction results are shown below. Figure 4 As shown in the diagram, the red dots represent the positions corresponding to the center point. It can be seen that fine-grained matching yields high-precision control point pairs, providing strong support for subsequent positioning.

[0061] 5. Multi-control point inversion solution: like Figure 5 As shown, the latitude, longitude, and altitude information of the corresponding landmarks can be obtained using the benchmark map and digital elevation map. To facilitate the calculation, Gaussian projection is used to transform the data into the Gaussian system for positioning calculation. Among them, the matching points... In the Gaussian coordinate system, the coordinates are... The true coordinates of the aircraft at the moment it is located at the center of the image in the Gaussian frame are: The slant distance from the matching point to the platform at this time is the observation, and thus the nonlinear equation system (8) is constructed.

[0062] The output of the inertial measurement element is used as the initial value for the iterative positioning solution. The Jacob matrix in the Newton iteration is shown in equation (9).

[0063] in This allows for the calculation of the UAV's spatial coordinates and attitude using the real physical coordinates of the matched control points and reference map.

[0064] As can be seen from the above embodiment 1, the present invention can provide better support for the visual navigation and positioning needs of UAVs under GPS denial conditions.

[0065] The foregoing has provided a detailed description of a visual navigation and positioning method, system, and readable storage medium based on heterogeneous image matching provided in the embodiments of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas; furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

[0066] Certain terms are used in the specification and claims to refer to specific components. Those skilled in the art will understand that hardware manufacturers may use different names to refer to the same component. This specification and claims do not distinguish components based on differences in name, but rather on differences in function. The terms "comprising" and "including" used throughout the specification and claims are open-ended and should be interpreted as "comprising / including but not limited to". "Approximately" means that within an acceptable margin of error, those skilled in the art can solve the technical problem and substantially achieve the technical effect within a certain margin of error. The following descriptions in the specification are preferred embodiments for carrying out this application; however, these descriptions are for the purpose of illustrating the general principles of this application and are not intended to limit the scope of this application. The scope of protection of this application shall be determined by the appended claims.

[0067] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0068] It should be understood that the term "and / or" used in this invention is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Furthermore, the character " / " in this invention generally indicates that the preceding and following related objects have an "or" relationship.

[0069] The foregoing description illustrates and describes several preferred embodiments of this application. However, as mentioned above, it should be understood that this application is not limited to the forms disclosed herein and should not be construed as excluding other embodiments. It can be used in various other combinations, modifications, and environments, and can be altered within the scope of the inventive concept described herein through the foregoing teachings or related technical or knowledge. Any modifications and variations made by those skilled in the art that do not depart from the spirit and scope of this application should be within the protection scope of the appended claims.

Claims

1. A visual navigation and localization method based on heterogeneous image matching, applied to an aircraft equipped with an airborne camera, characterized in that, Includes the following steps: S1. Real-time image acquisition: Acquire real-time images through the airborne camera; S2. Reference map creation: Combining aircraft flight path and continuous pose data, a preset range is captured on the satellite image with the current pose as the center to serve as the satellite reference map. S3. Perform distortion correction on the real-time image: derive the mapping relationship between the real image and the distorted image through the distortion model, and resample the corrected image using bilinear interpolation to obtain the corrected image; S4. Image matching: The matching step adopts a hierarchical strategy that combines global coarse matching and local fine matching. The global coarse matching performs feature extraction and matching between the reference image and the real-time image processed by step S3 based on the CFOG algorithm. Based on the global coarse matching results, the transformation matrix from the real-time map coordinates to the reference map coordinates is calculated, and the neighborhood image of the center point is extracted according to the transformation matrix, and then the neighborhood image is subjected to local fine matching. S5: Multi-control point inversion solution: Select multiple matching points with latitude and longitude information, perform Gaussian projection, construct a nonlinear equation system, and use the inertial navigation output as the initial value to perform Newton iteration to obtain the spatial coordinates and attitude of the UAV in Gaussian frame.

2. The visual navigation and localization method based on heterogeneous image matching according to claim 1, characterized in that: S3 specifically includes: image distortion correction: obtaining the camera intrinsic parameter matrix based on the physical length of the camera unit pixel and the focal length, obtaining the rotation matrix and translation vector based on the flight attitude, calculating the physical equivalent coordinates of the corrected image points, calculating the image plane coordinates before correction using the physical equivalent coordinates and flight attitude parameters, completing the mapping from the pixel coordinates of the corrected image to the pixel coordinates of the image before correction, and resampling the corrected image using bilinear interpolation to obtain the corrected image; wherein the flight attitude includes pitch angle, roll angle and yaw angle, and the physical equivalent coordinates are the coordinates obtained by dividing the coordinates in the physical coordinate system by the aircraft flight altitude.

3. The visual navigation and localization method based on heterogeneous image matching according to claim 1, characterized in that: The global coarse matching in S4 specifically involves: scaling and grayscale processing of the real-time image and the reference image; calculating the horizontal and vertical gradients based on the Sobel operator to obtain phase gradient features; applying Gaussian filtering to the phase gradient features; performing dense feature extraction and matching based on the CFOG algorithm; using SSD feature matching to calculate the sum of squares of the pixel differences between the real-time image and the reference image at corresponding coordinates; determining the region with the smallest sum of squares as the matching region; and using discrete Fourier transform to convert the time-domain circular convolution into frequency-domain dot product to accelerate computation. The real-time image and the reference image are shrunk to 1 / 4 of their original size, and the Gaussian filtering kernel size is 3×3 to 7×7.

4. The visual navigation and localization method based on heterogeneous image matching according to claim 3, characterized in that: The local fine matching in S4 specifically involves: based on the coarse matching result, solving the transformation matrix from real-time image coordinates to reference image coordinates; extracting the neighborhood image of the center point based on the transformation matrix; performing local fine matching based on the RIFT algorithm; generating a feature description matrix by statistically analyzing the information of the neighborhood points of the feature point on the tangent plane of the feature point; transforming the feature description matrix to generate feature description vectors for the feature points; using phase consistency to detect the feature points; and describing the features on the maximum exponent image. The neighborhood image of the center point is within a range of 50-200 pixels around the center point.

5. The visual navigation and localization method based on heterogeneous image matching according to claim 1, characterized in that: In the S2 reference map production step, the preset range is centered on the current pose and extends to 2-3 times the field of view of the camera of the UAV in that pose. This range is cropped from the satellite image as the satellite reference map, and the resolution of the satellite reference map is 0.3-1 meters / pixel.

6. The visual navigation and localization method based on heterogeneous image matching according to claim 1, characterized in that: The multi-control point inversion solution in S5 specifically includes: selecting multiple successfully matched pixels from the image matching results in S4 as multi-point positioning inputs; using the benchmark map and digital elevation map to obtain the latitude, longitude, and altitude information of the landmarks corresponding to the pixels; using Gaussian projection to transform them into the Gaussian system for positioning solution; constructing a nonlinear equation system using the coordinates of the matching point in the Gaussian system and the slant distance information from the matching point to the platform as observations; using the output of the inertial measurement element as the initial value for iterative positioning solution; using the Newton iteration method to iteratively solve using the Jacobi matrix; and inversely calculating the spatial coordinates and attitude of the UAV in the Gaussian system.

7. The visual navigation and localization method based on heterogeneous image matching according to claim 1, characterized in that: In the S5 multi-control point inversion solution step, at least 3 matching points are selected, and each matching point contains latitude and longitude information; the Newton iteration convergence condition is that the iteration error is less than a preset threshold or the number of iterations reaches a preset upper limit.

8. The visual navigation and localization method based on heterogeneous image matching according to claim 1, characterized in that: The visual navigation and positioning method based on heterogeneous image matching is applied to aircraft visual navigation and positioning scenarios under conditions where satellite navigation signals are limited.

9. A visual navigation and positioning system based on heterogeneous image matching, applied to an aircraft equipped with an airborne camera, characterized in that, include: The system comprises an image acquisition module, a reference map creation module, an image preprocessing module, a global coarse matching module, a local fine matching module, and a multi-control point inversion solution module. The output of the image acquisition module is connected to the reference map creation module and the image preprocessing module, respectively. The output of the reference map creation module is connected to the image preprocessing module. The image preprocessing module, the global coarse matching module, the local fine matching module, and the multi-control point inversion solution module are connected to each other in sequence. The image acquisition module is used to acquire real-time images via an airborne camera; The baseline image creation module is used to combine aircraft flight path and continuous pose data to capture a satellite baseline image with a resolution of 0.3-1 meter / pixel, centered on the current pose and within a range of 2-3 times the field of view of the UAV camera. The image preprocessing module is used to combine the pitch angle, roll angle, and yaw angle of the flight attitude to complete the solution of camera intrinsic parameters, coordinate mapping, and bilinear interpolation resampling for image distortion correction. The global coarse matching module is used for image processing including 1 / 4 scale scaling, grayscale conversion, Sobel gradient calculation, 3×3 to 7×7 kernel Gaussian filtering, CFOG dense feature matching, SSD matching, and discrete Fourier transform acceleration. The local fine matching module is used to extract a 50-200 pixel neighborhood image around the center point based on the coarse matching result, and complete the local fine matching based on the RIFT algorithm. The multi-control point inversion solution module is used to select at least 3 matching points with latitude and longitude information, and then perform Newton iteration solution by Gaussian projection, constructing a nonlinear equation system, and using the inertial navigation output as the initial value to obtain the spatial coordinates and attitude of the UAV in Gaussian frame. The system is suitable for aircraft operating under conditions where satellite navigation signals are limited.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is a non-volatile storage medium, and the computer-readable storage medium stores a computer program. When the computer program is executed by a processor, it implements the visual navigation and positioning method based on heterogeneous image matching as described in any one of claims 1 to 8.

Images