Deep Learning-Based Absolute Visual Navigation Method and Device for Aircraft

Abstract

This application relates to the field of aircraft navigation technology, specifically to a deep learning-based absolute visual navigation method and apparatus for aircraft. The method adaptively selects the visual navigation process based on the prior pose uncertainty of the aircraft. When the prior uncertainty is large, a two-stage strategy of "retrieval before matching" is adopted; when the prior information is relatively accurate, a single-stage strategy of "direct matching" is adopted. The method includes steps such as: reference image preparation, real-time image acquisition, prior information estimation, navigation strategy decision-making, pose calculation, and result output. This application improves the accuracy and robustness of cross-view image retrieval and matching through a deep learning model, enabling high-precision visual navigation of aircraft under a wide range of search conditions. This application adopts a modular design approach; the core modules such as image retrieval and matching in the technical framework can be flexibly equipped with different deep learning models according to the hardware platform performance and usage requirements, possessing good scalability.

Description

Technical Field

[0001] This application relates to the field of aircraft navigation technology, and in particular to an absolute visual navigation method and apparatus for aircraft based on deep learning. Background Technology

[0002] Visual navigation technology for aircraft primarily uses visual sensors mounted on the flight platform to image the ground and combine this image with a reference map containing geographic location information to estimate navigation parameters such as the aircraft's attitude. Compared to satellite navigation and inertial navigation, visual navigation offers advantages such as high precision, drift-free operation, and resistance to electromagnetic interference, and has been applied in numerous projects. Existing visual navigation technologies can be divided into two categories: relative visual navigation and absolute visual navigation. Relative visual navigation methods obtain the relative pose of the camera at different times through frame-to-frame matching, mainly including visual odometry and instant localization and mapping (SMR). These methods suffer from error accumulation issues and are difficult to apply independently to visual navigation for long-endurance, cross-scene flight missions outdoors. Absolute visual navigation methods match real-time images captured by the aircraft with a reference map containing geographic coordinate information, and then calculate the aircraft's absolute pose in geographic space based on the geographic information of the reference map and the imaging model. Since single-frame pose calculation can be performed independently, this method does not suffer from error accumulation issues. In recent years, many studies have integrated inertial navigation, relative visual navigation, and absolute visual navigation into a unified integrated navigation framework. This integrated navigation method effectively combines the advantages of different navigation methods, but the improvement of its core performance still depends on the accuracy and robustness of individual modules, such as the absolute visual navigation module.

[0003] In absolute vision navigation, image-level retrieval and pixel-level matching play distinct but complementary roles. When GPS signals are disrupted or inertial navigation experiences significant drift due to prolonged operation, the prior uncertainty of the aircraft's pose is substantial, leading to a dramatic expansion of the search area on the reference map. In such cases, image retrieval technology can quickly filter candidate regions most similar to the real-time image from a large-scale reference map, achieving coarse-to-fine localization. This process not only rapidly pinpoints the possible location on the real-time image but also significantly reduces the search space for subsequent pixel-level matching. Building upon this, pixel-level matching further refines the feature alignment between candidate regions and the real-time image, obtaining multi-point correspondences with pixel-level precision, providing reliable data support for subsequent high-precision pose calculation. Combining retrieval and matching to construct a two-stage strategy of "retrieval first, then matching" balances the efficiency of large-scale search with the accuracy of final pose estimation, making it the preferred solution for scenarios with high prior pose uncertainty. The significant viewpoint differences between real-time and baseline images of aircraft pose a major challenge to high-precision image retrieval and matching. Hand-designed image retrieval and matching methods, relying on hand-designed feature descriptors, suffer from limited feature discrimination when dealing with drastic changes in viewpoint and lighting, making robust retrieval or matching difficult. In recent years, deep learning-based image retrieval and matching models have learned more discriminative feature representations through data-driven approaches, demonstrating superior performance over traditional methods on cross-viewpoint image tasks. However, existing technologies largely focus on improving individual modules, lacking a unified framework that can adaptively select navigation strategies (single-stage matching or two-stage retrieval-matching) based on prior information uncertainty and deeply integrate the advantages of deep learning.

[0004] In summary, although deep learning has brought new opportunities to improve the robustness of cross-view image retrieval and matching, some typical problems still exist, such as: most visual navigation methods optimize the retrieval or matching modules in isolation, failing to place them in the whole navigation process to systematically consider strategy selection and module connection; and there is a lack of visual navigation frameworks that can dynamically adjust the processing flow based on prior pose uncertainty. Summary of the Invention

[0005] Therefore, it is necessary to provide a deep learning-based absolute visual navigation method and device for aircraft that addresses the aforementioned technical problems, such as the difficulty in balancing efficiency and accuracy in fixed-process navigation methods and the insufficient robustness of traditional image features under cross-view conditions.

[0006] A deep learning-based absolute visual navigation method for aircraft, the method comprising:

[0007] Step 1: Acquire surface image data of the mission area of ​​the spacecraft, geocode the surface image data, and generate a two-dimensional reference map and three-dimensional terrain information associated with the two-dimensional reference map;

[0008] Step 2: Use the visual perception equipment onboard the aircraft to observe the ground target area and collect real-time observation images of the aircraft;

[0009] Step 3: Based on the pose information provided by the non-visual navigation source of the aircraft, obtain the initial pose prior of the aircraft and the uncertainty corresponding to the initial pose prior; determine whether there is a visual pose calculation result of the previous time step; if so, use the visual pose calculation result of the previous time step to update the initial pose prior and uncertainty, and obtain the updated initial pose prior and the updated uncertainty.

[0010] Step 4: Based on the updated initial pose prior and the updated uncertainty, extract a reference map sub-map from the two-dimensional reference map that includes the field of view of the real-time observation image of the aircraft; calculate the ratio between the range of the reference map sub-map and the field of view of the real-time observation image of the aircraft; compare the ratio with a preset threshold, and adaptively select a single-stage visual navigation strategy or a two-stage visual navigation strategy based on the comparison result.

[0011] Step 5: Execute the selected visual navigation strategy, obtain the pixel matching point correspondence between the real-time observation image of the aircraft and the reference map sub-image through the deep learning model; solve the spatial pose calculation problem based on the pixel matching point correspondence according to the intrinsic parameter matrix of the aircraft's onboard camera and the three-dimensional terrain information, and obtain the absolute pose estimate of the aircraft.

[0012] Step 6: Calculate the confidence level of the absolute pose estimate, compare the confidence level with the preset confidence threshold, and if the confidence level is higher than the preset confidence threshold, output the absolute pose estimate to the aircraft navigation system.

[0013] A deep learning-based absolute visual navigation device for aircraft, the device comprising:

[0014] The data acquisition and encoding module is used to acquire surface image data of the mission area of ​​the spacecraft, perform geocoding on the surface image data, and generate a two-dimensional reference map and three-dimensional terrain information associated with the two-dimensional reference map.

[0015] The real-time observation image acquisition module is used to acquire real-time observation images of the aircraft by using the visual perception equipment on board the aircraft to observe the ground target area.

[0016] The pose and uncertainty update module is used to obtain the initial pose prior and the uncertainty corresponding to the initial pose prior based on the pose information provided by the non-visual navigation source of the aircraft; determine whether there is a visual pose calculation result of the previous time step; if so, update the initial pose prior and uncertainty using the visual pose calculation result of the previous time step to obtain the updated initial pose prior and the updated uncertainty.

[0017] The navigation strategy selection module is used to extract a reference map sub-map containing the field of view of the real-time observed image of the aircraft from the two-dimensional reference map based on the updated initial pose prior and the updated uncertainty; calculate the ratio between the range of the reference map sub-map and the field of view of the real-time observed image of the aircraft; compare the ratio with a preset threshold, and adaptively select a single-stage visual navigation strategy or a two-stage visual navigation strategy based on the comparison result.

[0018] The absolute pose estimation module is used to execute the selected visual navigation strategy. It obtains the pixel matching point correspondence between the real-time observation image of the aircraft and the reference map sub-image through a deep learning model. Based on the intrinsic parameter matrix of the aircraft's onboard camera and the three-dimensional terrain information, it solves the spatial pose calculation problem based on the pixel matching point correspondence to obtain the absolute pose estimation value of the aircraft.

[0019] The absolute pose output module is used to calculate the confidence level of the absolute pose estimate, compare the confidence level with a preset confidence threshold, and output the absolute pose estimate to the aircraft navigation system if the confidence level is higher than the preset confidence threshold.

[0020] The aforementioned deep learning-based absolute visual navigation method and device for aircraft provides a reliable basis for strategy decision-making by accurately estimating the initial pose priors and uncertainties, combined with non-visual navigation sources and preceding visual calculation results. When the ratio of the field of view of the reference map sub-map to the real-time map exceeds a preset threshold, a two-stage strategy of first searching and then matching is selected, first narrowing the search space and then finely matching to avoid the inefficiency and lack of robustness of direct matching over a large area. When the ratio is lower than the threshold, a single-stage strategy of direct matching is selected to reduce redundant steps and improve efficiency. This design of dynamically adjusting the process based on prior information breaks the limitation of a fixed, one-size-fits-all approach and achieves the optimal balance between efficiency and accuracy in different scenarios. To address the insufficient robustness of traditional image features across different viewpoints, this application replaces traditional manual feature methods with a deep learning model to construct pixel matching relationships between real-time and baseline sub-images. The deep learning model learns discriminative features across viewpoints and under varying illumination through data-driven learning. Compared to traditional features that rely on manual design, it can more stably capture the essential relationships between images, significantly improving the accuracy of image retrieval and matching in cross-viewpoint scenarios. Simultaneously, it combines camera intrinsic matrix and 3D terrain information to solve for pose, further ensuring navigation accuracy through the rigor of traditional geometric calculations. Furthermore, this application can independently constitute a complete absolute visual navigation system or be embedded as a core positioning module in existing GNSS / INS integrated navigation systems, significantly improving the adaptability and reliability of visual navigation in complex environments from an algorithmic framework perspective. By achieving high-precision and robust absolute pose estimation in a wide range of uncertain scenarios, this application effectively enhances the autonomous navigation capabilities of aircraft in satellite signal rejection environments. It is applicable to various platforms such as UAVs and deep space exploration vehicles, possessing significant engineering application value and economic benefits. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating a deep learning-based absolute vision navigation method for aircraft in one embodiment.

[0022] Figure 2 This is a structural block diagram of a deep learning-based absolute vision navigation device for aircraft in one embodiment;

[0023] Figure 3 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0025] In one embodiment, such as Figure 1As shown, a deep learning-based absolute vision navigation method for aircraft is provided, including the following steps:

[0026] Step 1: Acquire surface image data of the mission area of ​​the spacecraft, geocode the surface image data, and generate a two-dimensional reference map and three-dimensional terrain information associated with the two-dimensional reference map;

[0027] Acquire surface imagery data of the spacecraft's mission area and perform geocoding to generate a two-dimensional reference map with geographic coordinates. And associated 3D terrain information. Surface imagery data includes, but is not limited to, satellite imagery and aerial photographs.

[0028] By acquiring surface imagery data and generating a 2D reference map with geographic coordinates and associated 3D terrain information through geocoding, accurate geographic reference data is provided for subsequent absolute pose calculation. This technique solves the problem of missing 3D information in reference data in traditional navigation, providing key terrain dimension data for subsequent pose calculation based on 2D-3D matching point pairs. It avoids pose estimation errors caused by missing terrain information, ensuring navigation accuracy from the data source.

[0029] Step 2: Use the visual perception equipment onboard the aircraft to observe the ground target area and collect real-time observation images of the aircraft;

[0030] By using the aircraft's onboard camera to observe the ground, real-time images of the aircraft can be obtained. The aforementioned airborne cameras include, but are not limited to, optical sensors capable of acquiring ground images, such as visible light cameras, infrared cameras, and event cameras.

[0031] Step 3: Based on the pose information provided by the non-visual navigation source of the aircraft, obtain the initial pose prior of the aircraft and the uncertainty corresponding to the initial pose prior; determine whether there is a visual pose calculation result of the previous time step; if so, use the visual pose calculation result of the previous time step to update the initial pose prior and uncertainty, and obtain the updated initial pose prior and the updated uncertainty.

[0032] The initial attitude prior of the aircraft is obtained based on the attitude information provided by other navigation sources. and its uncertainty Determine if a previous visual pose calculation result exists. If it does, update the initial pose prior based on the previous visual pose calculation result. and its uncertainty ;

[0033] Based on the initial pose prior And camera imaging models, to display real-time images of the aircraft Corrected to an observation angle approximately similar to the baseline image. This step is generally placed after the real-time map acquisition step and before the navigation strategy decision step. Whether to add this step depends on the use case. If this step is added, subsequent steps will follow the same order. Replace the original real-time graph As input.

[0034] By acquiring real-time observation images through the visual perception equipment on board the aircraft, it is ensured that real environmental data synchronized with the current flight scenario is obtained, providing input that fits the actual working conditions for subsequent matching and calculation. Combining non-visual navigation sources with the results of previous visual pose calculation, the initial pose prior and uncertainty are dynamically updated. This dynamic update mechanism provides a more accurate basis for subsequent strategy decisions, reduces the strategy misselection caused by prior information errors, and provides an initial pose reference for the alignment of the real-time image and the reference image, indirectly improving the success rate of cross-view matching.

[0035] Step 4: Based on the updated initial pose prior and the updated uncertainty, extract a reference map sub-map from the two-dimensional reference map that includes the field of view of the real-time observation image of the aircraft; calculate the ratio between the range of the reference map sub-map and the field of view of the real-time observation image of the aircraft; compare the ratio with a preset threshold, and adaptively select a single-stage visual navigation strategy or a two-stage visual navigation strategy based on the comparison result.

[0036] Based on the initial pose prior and its uncertainty Extract a sub-map from the baseline map that includes the field of view of the real-time map. And calculate the ratio between the range of the reference map sub-map and the field of view range of the real-time map. According to the proportional relationship With preset threshold Based on the comparison results, an adaptive selection of a single-stage or two-stage visual navigation strategy is made.

[0037] In the above navigation strategy decision-making steps, the proportional relationship ,in For the real-time image field of view, The range of the base map sub-map.

[0038] like Greater than the preset threshold Then, a two-stage strategy of retrieval followed by matching is enabled. This strategy includes: Image retrieval stage: using a deep learning model in the reference subgraph Retrieved from the real-time image Most similar candidate regions; Image matching stage: Utilizing a deep learning-based image matching model for real-time image matching. A fine-grained matching is performed with the candidate regions to obtain the correspondence of matching points. .

[0039] If R is less than or equal to the preset threshold If the first step is to use a direct matching single-stage strategy, this strategy utilizes a deep learning model to compare the real-time graph with the base graph subgraph. Perform a direct match to obtain the corresponding relationship G of the matching points.

[0040] The purpose of navigation strategy selection is to adapt the visual navigation framework to different prior pose uncertainties (search ranges). If the satellite navigation system failure time is long and the inertial navigation drift is not large, the range of the reference image sub-image may be much larger than the field of view of the real-time image. Direct matching of these two images is computationally intensive and may lack robustness due to local texture pixels. Therefore, an image retrieval model can be used first to roughly determine the position of the real-time image by quickly comparing the features of the sub-regions of the real-time image and the reference image, thus narrowing the search space of subsequent image matching algorithms and improving computational efficiency and accuracy. If the satellite navigation system failure time is short and the inertial navigation accuracy is high, the range of the reference image sub-image may be comparable to the field of view of the real-time image, and direct matching can be performed to determine the geometric correspondence between the two images.

[0041] The aforementioned deep learning-based image retrieval model and deep learning-based image matching model are built on general network architectures such as convolutional neural networks, graph neural networks, or Transformers. The technical framework described in this patent can adopt different models according to the performance and usage requirements of different hardware platforms.

[0042] The image matching model described above takes a real-time image of the UAV and a reference sub-image as input, and outputs, but is not limited to, the matching point pairs between the real-time UAV image and the reference sub-image. Other information The mapping relationship between network input and output can be expressed as follows:

[0043]

[0044] in: Indicates the input image pair, the other information Including but not limited to camera intrinsics corresponding to drone images Camera pose Depth map .

[0045] Step 5: Execute the selected visual navigation strategy, obtain the pixel matching point correspondence between the real-time observation image of the aircraft and the reference map sub-image through the deep learning model; solve the spatial pose calculation problem based on the pixel matching point correspondence according to the intrinsic parameter matrix of the aircraft's onboard camera and the three-dimensional terrain information, and obtain the absolute pose estimate of the aircraft.

[0046] The selected visual navigation strategy is executed, and the correspondence between matching points between the real-time graph and the base graph subgraph is obtained through a deep learning model. ,in For real-time image pixel coordinates, Using the pixel coordinates of the base image sub-image, and combining the camera intrinsic parameter matrix K with the 3D ground information, the PnP problem based on 2D-3D matching point pairs is solved to obtain the absolute pose estimate of the aircraft. ;

[0047] The above-described solution to the Perspective-n-Point (PnP) problem based on 2D-3D matching point pairs is a common procedure in the field of visual navigation. The 3D information (height) of the matching points on the reference map comes from the 3D terrain information associated with the reference map in step 102. Specific implementation methods include, but are not limited to, the Direct Linear Transform (DLT) method, the P3P method, and the EPnP method.

[0048] This step specifically addresses the lack of robustness of traditional handcrafted features across different viewpoints: On one hand, deep learning models learn discriminative features across viewpoints and under varying lighting conditions through data-driven learning. Compared to traditional features that rely on manual design, these models can more stably capture the essential relationships between images, generating reliable pixel-matching pairs even when there are significant viewpoint differences between the real-time image and the baseline image. On the other hand, by combining the camera intrinsic matrix with 3D terrain information to solve the pose, the pixel-level relationships matched by deep learning are transformed into accurate geospatial poses through the rigor of traditional geometric calculations. This avoids the problem that end-to-end pose calculation models based on deep learning may have correct feature matching but biased pose calculations, thus overcoming the shortcomings of cross-viewpoint matching and ensuring the accuracy of the final pose estimation.

[0049] Step 6: Calculate the confidence level of the absolute pose estimate, compare the confidence level with the preset confidence threshold, and if the confidence level is higher than the preset confidence threshold, output the absolute pose estimate to the aircraft navigation system.

[0050] By calculating the confidence level of the pose estimate and comparing it with a threshold, the system selects the best results: only pose results with high confidence are output, avoiding erroneous navigation commands caused by low-quality matching, thus ensuring navigation reliability at the result level. Simultaneously, if the confidence level is insufficient, it can drive the re-execution of prior update, strategy selection, or matching solution steps, forming a closed-loop optimization. This further reduces the risk of navigation failure due to random errors and improves the stability of the entire navigation process.

[0051] The aforementioned deep learning-based absolute visual navigation method and device for aircraft provides a reliable basis for strategy decision-making by accurately estimating the initial pose priors and uncertainties, combined with non-visual navigation sources and preceding visual calculation results. When the ratio of the field of view of the reference map sub-map to the real-time map exceeds a preset threshold, a two-stage strategy of first searching and then matching is selected, first narrowing the search space and then finely matching to avoid the inefficiency and lack of robustness of direct matching over a large area. When the ratio is lower than the threshold, a single-stage strategy of direct matching is selected to reduce redundant steps and improve efficiency. This design of dynamically adjusting the process based on prior information breaks the limitation of a fixed, one-size-fits-all approach and achieves the optimal balance between efficiency and accuracy in different scenarios. To address the insufficient robustness of traditional image features across different viewpoints, this application replaces traditional manual feature methods with a deep learning model to construct pixel matching relationships between real-time and baseline sub-images. The deep learning model learns discriminative features across viewpoints and under varying illumination through data-driven learning. Compared to traditional features that rely on manual design, it can more stably capture the essential relationships between images, significantly improving the accuracy of image retrieval and matching in cross-viewpoint scenarios. Simultaneously, it combines camera intrinsic matrix and 3D terrain information to solve for pose, further ensuring navigation accuracy through the rigor of traditional geometric calculations. Furthermore, this application can independently constitute a complete absolute visual navigation system or be embedded as a core positioning module in existing GNSS / INS integrated navigation systems, significantly improving the adaptability and reliability of visual navigation in complex environments from an algorithmic framework perspective. By achieving high-precision and robust absolute pose estimation in a wide range of uncertain scenarios, this application effectively enhances the autonomous navigation capabilities of aircraft in satellite signal rejection environments. It is applicable to various platforms such as UAVs and deep space exploration vehicles, possessing significant engineering application value and economic benefits.

[0052] In one embodiment, calculating the ratio between the extent of the reference map sub-map and the field of view of the aircraft's real-time observation image includes:

[0053] The ratio between the range of the base map sub-map and the field of view of the real-time observation image of the aircraft is calculated as follows:

[0054]

[0055] in, To improve the field of view of real-time observation images for aircraft, The range of the base map subplot.

[0056] In one embodiment, the two-stage visual navigation strategy includes a coarse image retrieval stage and a fine image matching stage. The coarse image retrieval stage employs a deep learning-based image retrieval model to filter out candidate matching regions with the highest similarity to the real-time observation image of the aircraft from the reference image sub-image. The image retrieval model is built based on a general network architecture, including convolutional neural networks, graph neural networks, or Transformer networks. The fine image matching stage employs a deep learning-based image matching model to perform pixel-level fine matching between the real-time observation image of the aircraft and the candidate matching regions to obtain the pixel matching correspondence between the two. The image matching model is built based on a convolutional neural network, graph neural network, or Transformer network architecture.

[0057] In one embodiment, the single-stage visual navigation strategy is a direct matching strategy, which includes: using a deep learning-based image matching model to directly match pixels between the real-time observation image of the aircraft and a reference image sub-image to obtain the pixel matching correspondence between the two; the input and output of the image matching model satisfy the following mapping relationship:

[0058]

[0059] in, Indicates the input image pair, Represents an image matching model. This indicates real-time observation images from the aircraft. Represents the base map subplot. This indicates the pixel matching correspondence. This indicates auxiliary navigation information.

[0060] In one embodiment, the surface image data includes satellite remote sensing images, aerial photographs, and ground mapping images; the visual sensing devices include visible light cameras, infrared imaging cameras, event cameras, and multispectral cameras.

[0061] In one embodiment, the non-visual navigation source includes a satellite navigation module, an inertial measurement unit, a barometric altimeter, and a geomagnetic navigation module. If the currently acquired real-time observation image of the aircraft is the first frame image after the visual navigation system is started, the initial pose prior is provided by the non-visual navigation source. If the currently acquired real-time observation image of the aircraft is not the first frame image, the initial pose prior is determined by weighted fusion of the visual pose calculation results from the previous time step and the pose data provided by the non-visual navigation source.

[0062] In a specific embodiment, if the aircraft is equipped with a satellite-inertial integrated navigation system in addition to a visual navigation system... And satellite navigation systems in It is prone to failure due to electromagnetic interference and other factors, while this visual navigation system... Start working immediately, then Initial pose prior of the spacecraft For the output of satellite-inertial integrated navigation system pose uncertainty The error during normal operation of the satellite-inertial integrated navigation system, and the satellite navigation failure time. It is a function of factors such as inertial navigation error drift rate. time( (For the working time of the visual navigation system), if there are pose calculation results from previous time steps, then For the output of satellite-inertial integrated navigation system Result of pose calculation with previous time steps ( The function, The error during normal operation of the satellite-inertial integrated navigation system, and the satellite navigation failure time. Inertial navigation error drift rate, confidence level of pose estimation at preceding time steps ( A function of factors such as ).

[0063] In one embodiment, after step 2 and before step 4, a real-time image correction step is included; the real-time image correction step includes geometric distortion correction and viewpoint unification processing of the real-time observation image of the aircraft based on the updated pose prior information and the imaging model of the visual perception device, to obtain a corrected real-time image with the same observation viewpoint as the two-dimensional reference image; in subsequent steps, the corrected real-time image is used instead of the real-time observation image of the aircraft as the data input source.

[0064] In one embodiment, the pixel matching correspondence is denoted as... ,in, This represents the coordinates of the i-th pixel in the real-time observation image from the aircraft. Let represent the coordinates of the i-th pixel on the base map submap, and N represent the total number of pixel matching pairs, where N≥3.

[0065] In one embodiment, if the confidence level is lower than or equal to a preset confidence threshold, steps 3 to 6 are re-executed, and the extraction range of the baseline sub-map is adjusted according to the previous calculation error during the re-execution.

[0066] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0067] In one embodiment, such as Figure 2 As shown, a deep learning-based absolute vision navigation device for aircraft is provided, comprising: a data acquisition and encoding module 202, a real-time observation image acquisition module 204, a pose and uncertainty update module 206, a navigation strategy selection module 208, an absolute pose estimation module 210, and an absolute pose output module 212, wherein:

[0068] The data acquisition and encoding module 202 is used to acquire surface image data of the mission area of ​​the spacecraft, perform geocoding on the surface image data, and generate a two-dimensional reference map and three-dimensional terrain information associated with the two-dimensional reference map.

[0069] The real-time observation image acquisition module 204 is used to acquire real-time observation images of the aircraft by using the visual perception equipment on board the aircraft to observe the ground target area.

[0070] The pose and uncertainty update module 206 is used to obtain the initial pose prior and the uncertainty corresponding to the initial pose prior based on the pose information provided by the non-visual navigation source of the aircraft; determine whether there is a visual pose calculation result of the previous moment; if there is, update the initial pose prior and uncertainty using the visual pose calculation result of the previous moment to obtain the updated initial pose prior and the updated uncertainty.

[0071] The navigation strategy selection module 208 is used to extract a reference map sub-map containing the field of view of the real-time observation image of the aircraft from the two-dimensional reference map based on the updated initial pose prior and the updated uncertainty; calculate the ratio between the range of the reference map sub-map and the field of view of the real-time observation image of the aircraft; compare the ratio with a preset threshold; and adaptively select a single-stage visual navigation strategy or a two-stage visual navigation strategy based on the comparison result.

[0072] The absolute pose estimation module 210 is used to execute the selected visual navigation strategy, obtain the pixel matching point correspondence between the real-time observation image of the aircraft and the reference map sub-image through a deep learning model, and solve the spatial pose calculation problem based on the pixel matching point correspondence according to the intrinsic parameter matrix of the aircraft's onboard camera and the three-dimensional terrain information to obtain the absolute pose estimation value of the aircraft.

[0073] The absolute pose output module 212 is used to calculate the confidence level of the absolute pose estimate, compare the confidence level with a preset confidence threshold, and output the absolute pose estimate to the aircraft navigation system if the confidence level is higher than the preset confidence threshold.

[0074] Specific limitations regarding deep learning-based absolute vision navigation devices for aircraft can be found in the limitations of deep learning-based absolute vision navigation methods for aircraft described above, and will not be repeated here. Each module in the aforementioned deep learning-based absolute vision navigation device for aircraft can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0075] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities and includes, but is not limited to, any one or more combinations of a Central Processing Unit (CPU), Graphics Processing Unit (GPU), Neural Processing Unit (NPU), or Tensor Processing Unit (TPU). The processor supports deep learning frameworks (such as PyTorch, TensorFlow, ONNX, tensorRT, etc.) and can perform inference computations for deep learning models. The memory of the computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface of the computer device is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a deep learning-based absolute vision navigation method for aircraft. The display screen of the computer device can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs or touchpads set on the casing of the computer device, or external keyboards, touchpads or mice, etc.

[0076] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0077] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0078] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A deep learning based aircraft absolute visual navigation method, characterized in that, The method comprises: Step 1: obtaining the ground surface image data of the aircraft task area, geocoding the ground surface image data, generating a two-dimensional reference map and three-dimensional terrain information associated with the two-dimensional reference map; Step 2: observing the ground target area by using the visual perception device carried by the aircraft to collect the real-time observation image of the aircraft; Step 3: obtaining the initial pose prior of the aircraft based on the pose information provided by the non-visual navigation source of the aircraft and the uncertainty corresponding to the initial pose prior; judging whether there is a visual pose solution result at a previous time, if there is, updating the initial pose prior and the uncertainty by using the visual pose solution result at the previous time, to obtain the updated initial pose prior and the updated uncertainty; Step 4: according to the updated initial pose prior and the updated uncertainty, cutting the reference map subgraph containing the field of view range of the real-time observation image of the aircraft from the two-dimensional reference map; calculating the proportional relationship between the range of the reference map subgraph and the field of view range of the real-time observation image of the aircraft; comparing the proportional relationship with a preset threshold, and adaptively selecting a single-stage visual navigation strategy or a two-stage visual navigation strategy according to the comparison result; the two-stage visual navigation strategy includes an image coarse retrieval stage and an image fine matching stage; the image coarse retrieval stage adopts an image retrieval model based on deep learning to filter out the candidate matching area with the highest similarity to the real-time observation image of the aircraft in the reference map subgraph, and the image retrieval model is constructed based on a general network architecture, the general network architecture includes a convolutional neural network, a graph neural network or a Transformer network; the image fine matching stage adopts an image matching model based on deep learning to perform pixel-level fine matching between the real-time observation image of the aircraft and the candidate matching area, to obtain the pixel matching corresponding relationship therebetween, and the image matching model is constructed based on a convolutional neural network, a graph neural network or a Transformer network architecture; The single-stage visual navigation strategy is a direct matching strategy, which includes: adopting an image matching model based on deep learning to directly perform pixel matching between the real-time observation image of the aircraft and the reference map subgraph, to obtain the pixel matching corresponding relationship therebetween; the input and output of the image matching model satisfy the following mapping relationship: wherein, represents an input image pair, represents the image matching model, represents the real-time observation image of the aircraft, represents the reference graph subgraph, represents the pixel matching correspondence relationship, represents auxiliary navigation information; Step 5: executing the selected visual navigation strategy, obtaining the pixel matching point corresponding relationship between the real-time observation image of the aircraft and the reference map subgraph by the deep learning model; solving the spatial pose solving problem based on the pixel matching point corresponding relationship according to the intrinsic matrix of the aircraft on-board camera and the three-dimensional terrain information, to obtain the absolute pose estimation value of the aircraft; Step 6: calculating the confidence of the absolute pose estimation value, comparing the confidence with a preset confidence threshold, if the confidence is higher than the preset confidence threshold, outputting the absolute pose estimation value to the aircraft navigation system.

2. The method of claim 1, wherein, The proportional relationship between the range of the reference map subgraph and the field of view range of the real-time observation image of the aircraft comprises: A proportion of a range of the reference map subgraph to a range of a field of view of the real-time observation image of the aerial vehicle is calculated wherein, is the range of the field of view of the real-time observation image for the aircraft, is the range of the reference map submap.

3. The method of claim 1, wherein, The ground surface image data includes satellite remote sensing images, aerial photography images, and ground survey images; and the visual perception device includes a visible light camera, an infrared imaging camera, an event camera, and a multispectral camera.

4. The method of claim 1, wherein, The non-visual navigation source includes a satellite navigation module, an inertial measurement unit, a barometric altimeter, and a geomagnetic navigation module; if the currently collected real-time observation image of the aerial vehicle is the first image after the visual navigation system is started, the initial pose prior is provided by the non-visual navigation source; if the currently collected real-time observation image of the aerial vehicle is not the first image, the initial pose prior is determined by weighted fusion of a previous visual pose solution and pose data provided by the non-visual navigation source.

5. The method of claim 1, wherein, The pixel matching correspondence relationship is recorded as wherein, represents the coordinates of the i-th pixel on the real-time observation image of the aircraft, represents the coordinates of the i-th pixel on the reference graph subgraph, and N represents the total number of pixel matching point pairs, N≥3.

6. The method of claim 1, wherein, The method further includes: If the confidence is lower than or equal to the preset confidence threshold, the steps 3 to 6 are re-executed, and the extraction range of the reference map subgraph is adjusted according to a previous solution error when re-execution is performed.

7. A deep learning based aircraft absolute visual navigation apparatus, characterized by, The device includes: a data acquisition and encoding module configured to acquire ground surface image data of a task area of an aerial vehicle, to perform geographic encoding on the ground surface image data, to generate a two-dimensional reference map, and to generate three-dimensional terrain information associated with the two-dimensional reference map; a real-time observation image acquisition module configured to acquire a real-time observation image of the aerial vehicle by observing a target area on the ground using a visual perception device carried by the aerial vehicle; a pose and uncertainty updating module configured to obtain an initial pose prior of the aerial vehicle and an uncertainty corresponding to the initial pose prior based on pose information provided by a non-visual navigation source of the aerial vehicle, to determine whether there is a previous visual pose solution, and to update the initial pose prior and the uncertainty using the previous visual pose solution if the previous visual pose solution exists, to obtain an updated initial pose prior and an updated uncertainty. The navigation strategy selection module is configured to: cut a reference map subgraph containing a field of view range of the real-time observation image of the aircraft from the two-dimensional reference map according to the updated initial pose prior and the updated uncertainty; calculate a proportional relationship between a range of the reference map subgraph and the field of view range of the real-time observation image of the aircraft; compare the proportional relationship with a preset threshold value, and adaptively select a single-stage visual navigation strategy or a two-stage visual navigation strategy according to a comparison result; the two-stage visual navigation strategy includes an image coarse retrieval stage and an image fine matching stage; the image coarse retrieval stage adopts an image retrieval model based on deep learning to filter out a candidate matching region with the highest similarity to the real-time observation image of the aircraft from the reference map subgraph, and the image retrieval model is constructed based on a general network architecture, the general network architecture including a convolutional neural network, a graph neural network or a Transformer network; the image fine matching stage adopts an image matching model based on deep learning to perform pixel-level fine matching on the real-time observation image of the aircraft and the candidate matching region, and obtain a pixel matching corresponding relationship therebetween, and the image matching model is constructed based on a convolutional neural network, a graph neural network or a Transformer network architecture; The single-stage visual navigation strategy is a direct matching strategy, including: adopting an image matching model based on deep learning to directly perform pixel matching on the real-time observation image of the aircraft and the reference map subgraph, and obtain a pixel matching corresponding relationship therebetween; and the input and the output of the image matching model satisfy the following mapping relationship: wherein, represents an input image pair, represents the image matching model, represents the real-time observation image of the aircraft, represents the reference graph subgraph, represents the pixel matching correspondence relationship, represents auxiliary navigation information; The absolute pose estimation module is configured to execute the selected visual navigation strategy, and obtain a pixel matching point corresponding relationship between the real-time observation image of the aircraft and the reference map subgraph through a deep learning model; and solve a spatial pose solving problem based on the pixel matching point corresponding relationship according to an intrinsic matrix of an on-board camera of the aircraft and the three-dimensional terrain information, and obtain an absolute pose estimation value of the aircraft. The absolute pose output module is configured to calculate a confidence degree of the absolute pose estimation value, compare the confidence degree with a preset confidence threshold value, and output the absolute pose estimation value to a navigation system of the aircraft if the confidence degree is higher than the preset confidence threshold value.

Images