Method and apparatus for perception-based autonomous landing of an aircraft

Abstract

This application relates to perception-based autonomous landing of aircraft and provides a method for supporting aircraft approach to a runway. An example implementation involves receiving a sequence of images captured by a camera mounted on the aircraft. An example implementation involves applying the received images to a machine learning model trained to detect runways or runway markings in the images and generating a mask comprising pixel segments in the images assigned to object categories of the runway or markings. An example implementation may also involve applying the mask to a corner detector to detect points of interest on the mask and matching the points of interest to corresponding points on the runway or markings with known local coordinates bounded by the runway. An example implementation may also involve performing perspective n-point estimation to determine a current attitude estimate of the aircraft relative to the runway or markings and outputting the current attitude estimate for use during the final approach.

Description

Technical Field

[0001] This topic generally relates to aircraft operation, and in particular to the autonomous detection of runways and runway markings. Background Technology

[0002] Safe, predictable, and reliable landing is an essential aspect of an effective aircraft-based transportation system. In many situations, piloted aircraft can only land visually under high visibility and clear weather conditions. While external navigation systems, such as Instrument Landing Systems (ILS) and Global Positioning Systems (GPS), can be used in some scenarios, certain situations require visibility and cloud conditions below the designated decision altitude to allow pilots to land visually. Even where external systems are available, they are susceptible to interruptions, inaccuracies, and interference, posing technical challenges to efforts to automate the landing process.

[0003] The technical challenges associated with automatic landing or automatically generating guidance during landing to assist pilots are complicated by a number of factors, such as the diversity and variability of weather and visibility conditions, the availability and feasibility of specialized equipment at different airports, and the robustness required for algorithmic approaches to landing operations. Summary of the Invention

[0004] The exemplary implementations disclosed in this subject matter address autonomous detection of runways and runway markings. These exemplary implementations solve and overcome the technical challenges associated with autonomous approach and landing operations in aircraft by using machine learning models to identify features in images of the landing environment, determine the aircraft's attitude relative to the landing environment, and generate the horizontal, vertical, and other positional guidance required for landing. Some exemplary implementations are flexible, at least in the sense that they can be adapted to a range of passive and active sensors, including but not limited to visual imaging devices, hyperspectral imaging devices, LiDAR, radar, etc., to sense the landing environment and determine the relevant aircraft attitude, and can be used with a variety of manned and unmanned aircraft of different sizes for landing on visual, precise, and non-precise runways. Some exemplary implementations are also cost-effective, at least in the sense that they do not require the installation of expensive ground infrastructure at a given airport.

[0005] Therefore, this subject matter discloses, including but not limited to, the following example implementations.

[0006] Some example implementations provide a method for supporting an aircraft approaching a runway at an airport, the method comprising receiving a sequence of images of the airport captured by a camera mounted on the approaching aircraft; and for at least one image in the image sequence, applying the image to a machine learning model trained to perform object detection and segmentation, wherein a runway or runway markings on the runway are detected in the image, and wherein a mask is generated comprising pixel segments in the image assigned to an object category of the runway or runway markings; applying the mask to a corner detector to detect points of interest on the mask, and thereby detecting points of interest on the runway or runway markings in the image; matching the points of interest on the runway or runway markings in the image with corresponding points on the runway or runway markings having local coordinates known to be bounded by the runway; performing perspective n-point estimation using the points of interest and the local coordinates known to be bounded by the runway to determine a current attitude estimate of the camera, and thereby determining a current attitude estimate of the aircraft relative to the runway or runway markings; and outputting the current attitude estimate for at least one of guidance or control of the aircraft during final approach.

[0007] In some example implementations of the method of any of the foregoing example implementations, or in any combination of any of the foregoing example implementations, the image is a color image, and the method further includes converting the color image into a monochrome image.

[0008] In some example implementations of the method of any of the foregoing example implementations, or in any combination of any of the foregoing example implementations, the method further includes cropping the image to reduce the camera's field of view and magnifying only a portion of the airport where the runway or runway markings are located before the image is applied to the machine learning model.

[0009] In some example implementations of the method of any of the foregoing example implementations, or in any combination of any of the foregoing example implementations, the method further includes accessing an earlier image in the image sequence, and the earlier masking includes a corresponding pixel segment in the earlier image assigned to an object category of a runway or runway sign; and identifying a portion of the earlier masking in the earlier image, thereby identifying a portion of the airport where the runway is located, and wherein cropping the image includes cropping the image to a portion of the image corresponding to a portion of the earlier image.

[0010] In some example implementations of the methods of any of the foregoing example implementations, or in any combination of any of the foregoing example implementations, object detection and segmentation is one-class object detection and segmentation, wherein only runways or runway signs are detected in the image, and the masking includes only pixel segments in the image assigned to the object class of the runway or runway sign.

[0011] In some example implementations of the methods of any of the foregoing example implementations, or in any combination of any of the foregoing example implementations, object detection and segmentation is multi-class object detection and segmentation, wherein runways and runway signs are detected in an image, and the masking includes pixel segments in the image assigned to the object categories of the runways and runway signs, and points of interest include points of interest on the runways and runway signs.

[0012] In some example implementations of the method of any of the foregoing example implementations, or in any combination of any of the foregoing example implementations, the method further includes determining the operational state of the aircraft; and, based on the operational state, selecting a first machine learning model or a second machine learning model as the machine learning model to which the image is applied, the first machine learning model being trained to perform single-class object detection and segmentation, and the second machine learning model being trained to perform multi-class object detection and segmentation.

[0013] In some example implementations of the method of any of the foregoing example implementations, or in any combination of the foregoing example implementations, the machine learning model further determines a confidence interval associated with the detection of a runway or runway marking, and the method further includes applying a mask to a confidence filter that passes the mask to the corner detector only if the confidence interval is higher than a threshold confidence interval dynamically set based on the operational state of the aircraft.

[0014] In some example implementations of the method of any of the foregoing example implementations, or in any combination of any of the foregoing example implementations, performing perspective n-point estimation includes determining the current attitude estimate of the camera in camera coordinates and transforming the camera coordinates into local coordinates corresponding to the runway frame, which are output for at least one of the guidance or control of the aircraft.

[0015] In some example implementations of the method of any of the foregoing example implementations, or in any combination of any of the foregoing example implementations, performing perspective n-point estimation further includes determining a confidence interval associated with the current attitude estimate, and the method further includes: accessing one or more additional attitude estimates and associated confidence intervals; using the confidence intervals and associated confidence intervals, performing sensor fusion on the current attitude estimate and one or more additional attitude estimates to generate an updated current attitude estimate; and outputting the updated current attitude estimate for at least one of guidance or control of the aircraft.

[0016] Some example implementations provide an apparatus for supporting an aircraft approaching a runway at an airport, the apparatus comprising: a memory configured to store computer-readable program code; and a processing circuitry configured to access the memory and execute the computer-readable program code to cause the apparatus to perform at least any of the foregoing example implementations or any combination of the foregoing example implementations.

[0017] Some example implementations provide a computer-readable storage medium for supporting aircraft approach to a runway at an airport. The computer-readable storage medium is non-transitory and has computer-readable program code stored therein, which, in response to execution by a processing circuitry system, causes the apparatus to perform at least any of the foregoing example implementations or any combination of the foregoing example implementations.

[0018] By following detailed description and appendix Figure 1 Upon reading this subject matter, these and other features, aspects, and advantages disclosed herein will be apparent, as briefly described below with reference to the accompanying drawings. This subject matter disclosure includes any combination of two, three, four, or more features or elements set forth in this disclosure, whether or not such features or elements are explicitly combined or otherwise mentioned in the specific exemplary embodiments described herein. This disclosure is intended to be read in its entirety such that any separable feature or element of this disclosure, in any aspect and exemplary embodiment thereof, should be considered composable unless the context of this disclosure expressly specifies otherwise.

[0019] Therefore, it should be understood that the purpose of providing this invention is merely to summarize some exemplary embodiments in order to provide a basic understanding of some aspects of this disclosure. Consequently, it should be understood that the above exemplary embodiments are merely examples and should not be construed as limiting the scope or spirit of this disclosure in any way. Other exemplary embodiments, aspects, and advantages will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate the principles of some of the described exemplary embodiments by way of example. Attached Figure Description

[0020] Having thus given a general description of exemplary embodiments of the present disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in which:

[0021] Figure 1 An example aircraft is shown that can benefit from the exemplary implementations disclosed in this subject matter;

[0022] Figure 2 This is a block diagram of a system for supporting an aircraft approaching a runway at an airport, according to an example embodiment disclosed in this subject matter.

[0023] Figure 3This is a block diagram of a system pipeline that can be used in conjunction with a system for supporting aircraft approaching runways at an airport, according to an example implementation disclosed in this subject matter.

[0024] Figure 4A It is a block diagram of an airport image that can be used in conjunction with the exemplary implementations disclosed in this subject matter;

[0025] Figure 4B This is a block diagram illustrating an airport with images of an example embodiment of the present subject matter.

[0026] Figure 5 This is a flowchart illustrating various steps in a method for supporting an aircraft's approach to a runway at an airport, according to an exemplary embodiment disclosed in this subject matter; and

[0027] Figure 6 An apparatus according to some example embodiments disclosed in this subject matter is shown. Detailed Implementation

[0028] Some embodiments of the subject matter disclosure will now be described more fully below with reference to the accompanying drawings, which illustrate some, but not all, of the embodiments of the disclosure. In fact, various embodiments of the disclosure may be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these exemplary embodiments are provided so that the disclosure will be thorough and complete and will fully convey the scope of the disclosure to those skilled in the art. For example, unless otherwise indicated, references to things as first, second, etc., should not be construed as implying a particular order. Moreover, something that can be described as above something else (unless otherwise indicated) may actually be below, and vice versa; and similarly, something described as to the left of something else may actually be to the right, and vice versa. Similar reference numerals are used throughout to denote similar elements. With regard to the terms “comprising,” “including,” “having,” “containing,” and variations thereof as used herein, such terms are intended to be included in a manner similar to the term “comprising” as an open transitional term without excluding any additional or other elements.

[0029] The example implementations disclosed in this subject matter pertain to the autonomous detection of runways (and / or heliports) and runway markings. To address and overcome the technical challenges associated with autonomous approach and landing operations for aircraft, the example implementations disclosed in this subject matter involve using machine learning models to identify features in landing environment images and then determining the aircraft's attitude relative to the landing environment in order to generate the horizontal, vertical, and other positional guidance required for landing.

[0030] Conventional methods using computer-generated landing maneuvers employ resource-intensive, rule-based algorithms to provide limited runway edge detection. These rule-based, resource-intensive runway detection methods lack the robustness required for real-world implementations of many aircraft and airports. In particular, some conventional methods require prior knowledge of runway characteristics, such as the location, orientation, length, width, marking type, and surface material of a given runway, and often necessitate the installation of expensive ground equipment at the airport. Such methods are not easily generalized to new runways and may be less effective when aircraft are redirected to new or different airports. Furthermore, even when a portion of the runway is detected, conventional methods lack the ability to effectively guide and control aircraft approaching the airport using this detection.

[0031] The exemplary implementations disclosed in this subject combine computer vision techniques with machine learning perception techniques in a way that allows for the identification of different sets of runways, runway markings, runway corners, threshold marks, aiming pads, etc., under a wide range of visibility conditions. Some exemplary implementations are adapted to a range of passive and active sensors, including but not limited to visual imaging devices, hyperspectral imaging devices, LiDAR, radar, etc., for example, to sense the landing environment and determine the relevant attitude of the aircraft, and can be used with manned and unmanned aircraft of various sizes for landing on visual, precise, and non-precise runways.

[0032] Some example implementations involve convolutional neural networks as part of a multimodal approach for detecting runway and / or runway features. In some cases, the example implementations disclosed in this subject matter supplement visual data with distance measurements of the ground and associated obstacles by LiDAR and / or radar. Regardless of the underlying source of the visual and / or supplementary data used, the example implementations disclosed in this subject matter are configured to generate and provide accurate and precise attitude estimates and guidance without requiring extensive prior knowledge of the runway or expensive ground infrastructure installed at a given airport.

[0033] Figure 1 An aircraft 100 is shown that can benefit from an example implementation disclosed in this subject matter. As shown, the aircraft includes a fuselage 102 having a fuselage 104, wings 106, and a tail 108. The aircraft also includes multiple advanced systems 110, such as a propulsion system 112, an electrical system 114, a hydraulic system 116, and / or an environmental system 118. Any number of other systems may be included on the aircraft.

[0034] Figure 2A system 200 for supporting an aircraft approaching a runway, according to an example implementation disclosed in this subject matter, is illustrated. The system 200 may include any one of several different subsystems (each a separate system) for performing one or more functions or operations. As shown, in some examples, the system 200 includes one or more cameras 202 (located on the aircraft and configured to generate image sequences 204), an attitude estimation system 205, an image processing system 206, a first machine learning model 208 and a second machine learning model 210 (configured to generate one or more masks 212 that can be applied to the images), a corner detector 214, a perspective n-point system 216 (configured to generate one or more attitude estimates 218), a guidance system 220, and a control system 222. The subsystems including the cameras, image processing system, first machine learning model, second machine learning model, corner detector, perspective n-point system, guidance system, and control system may be co-located with each other or directly coupled to each other, or in some examples, the various subsystems within the subsystems may communicate with each other across one or more computer networks 224. In some example implementations, any one of the subsystems may access or interact with one or more additional information sources 226, any of which may be co-located with or directly coupled to each other, and / or co-located with and / or directly coupled to system elements, or may be accessed via computer networks and / or cloud-based architectures.

[0035] Furthermore, although shown as part of system 200, it should be understood that any one or more of the above can operate or function as a separate system, without regard to any of the other subsystems. It should also be understood that the system may include... Figure 2 One or more additional or alternative subsystems in addition to those subsystems shown.

[0036] In the example embodiments disclosed in this subject matter, one or more components from system 200 may be placed on the aircraft or otherwise added to the aircraft, such as Figure 1 The aircraft 100 is shown. In such example embodiments, any of the systems 200 and their components can communicate with and / or otherwise interact with any of the systems 110 of the aircraft 100. For example, a camera 202 can be mounted to the aircraft 100, configured to capture a sequence 204 of images of the environment in the direction of travel of the aircraft 100. In some such example embodiments, an attitude estimation system 205 is configured to be combined with an image processing system 206 and / or a perspective n-point system 216 (both of which can also be combined with the aircraft 100), which provides information to a guidance system 220 and / or a control system 222 of the aircraft 100 to assist in the landing of the aircraft 100.

[0037] In some example implementations, system 200 is configured to receive a sequence of images 204 of the airport as the aircraft 100 approaches the runway. In some such example implementations, the image sequence is captured by a camera 202 mounted on the aircraft 100 approaching the runway.

[0038] Camera 202 can be any camera capable of capturing image sequences, including but not limited to visual imaging devices, hyperspectral imaging devices, lidar imaging devices, radar imaging devices, etc. In some cases, camera 202 is mounted to and / or otherwise integrated into aircraft 100, and its configuration allows the camera to capture a view of the environment in front of aircraft 100 in the direction of travel of aircraft 100.

[0039] In some example implementations, for at least one image 204A in the image sequence 204, system 200 is configured to apply that image to a machine learning model, such as a first machine learning model 208 or a second machine learning model 210. For example, the learning model is trained to perform object detection and segmentation, wherein a runway or runway markings on an airport are detected in the image. In some example implementations, the first machine learning model 208 and / or the second machine learning model 210 incorporates a convolutional neural network. It should be understood that any neural network or other machine learning model type can be used in the example implementations of system 200.

[0040] In some example implementations, image 204A may be processed by image processing system 206 before, simultaneously with, and / or after applying image 204A to one or more machine learning models 208, 210. For example, in some example implementations, image 204A is a color image, and another aspect of image processing system 206 and / or system 200 is configured to convert the color image to a monochrome image.

[0041] In some example implementations, another aspect of image processing system 206 and / or system 200 is configured to further crop image 204A to reduce the field of view of camera 202 and magnify only a portion of the airport where the relevant runway or relevant runway markings are located. In some such example implementations, the cropping of image 204A is performed before it is applied to machine learning models 208 and / or 210.

[0042] In some example implementations, the first machine learning model 208 and / or the second machine learning model 210 are configured to generate at least one mask 212. In some example implementations, the mask 212 is generated such that it is configured to include pixel segments in image 204A that are assigned to object categories of the relevant runway or relevant runway marker. In some example implementations, the mask 212 is stored and can be accessed by system 200 and / or another system for use with subsequent image sequences.

[0043] In some example implementations of system 200, the object detection and segmentation performed by the first machine learning model 208 or the second machine learning model 210 is single-class object detection and segmentation, wherein only runways or runway signs are detected in image 204A. In some such example implementations, the resulting mask 212 includes only pixel segments in image 204A that are assigned to the object category of runway or runway sign.

[0044] Some example implementations involve instance-wise multi-class object detection and segmentation. In some example implementations of system 200, the object detection and segmentation performed by a first machine learning model 208 or a second machine learning model 210 is multi-class object detection and segmentation, wherein a runway and runway markers are detected in image 204A, and mask 212 includes pixel segments in image 204A assigned to the object categories of the runway and runway markers. In some such example implementations, the points of interest identified in the image include points of interest on the runway and runway markers.

[0045] In some example implementations, system 200 is configured to apply mask 212 to corner detector 214. In some such example implementations, the corner detector is configured to detect points of interest on mask 212, and thereby detect points of interest on the runway or runway markings in image 204A. In some example implementations, the points of interest may correspond to portions of the runway itself, such as corners, edges, or other features of the runway. In some example implementations, the points of interest may correspond to portions of runway markings, such as alignment pads, letters, numbers, geometry, etc., present on a given runway.

[0046] Thus, in some example embodiments, system 200 is further configured to match points of interest on the runway or runway markers in image 204A with corresponding points on the runway or runway markers that have known local coordinates that define the runway. It should be understood that, in order to provide accurate and precise information that can be used to guide or control aircraft 100 during landing operations, the system may need to determine how the appearance and relative positioning of runway-related elements and / or runway marker-related elements in a given image correspond to the actual appearance and relative positioning of those elements on the runway itself.

[0047] In example implementations where the local coordinates of the runway are known, those known, runway-bound local coordinates can be matched with corresponding points in the image by image processing system 206 or another component of system 200. For example, the local coordinates of the runway bounding can be pre-captured or otherwise determined for a given runway, such as by identifying the positions of various symbols, lights, equipment, or other runway markings maintained relative to a given orientation of the runway. In some example implementations, the local coordinates of the runway bounding are stored in a local data store and / or accessed by system 200 from a remote data store prior to an attempt to land on a given runway. In some example implementations, information from one or more additional information sources 226 can be used in conjunction with machine learning models 208, 210 and / or corner detector 214 to identify points of interest in the image and / or match points of interest in the image with corresponding points on the runway. Such additional information may include, for example, Global Positioning System (GPS) information, Missed Flight Point (MAP) information, Instrument Landing System (ILS) information, etc.

[0048] like Figure 2 As shown, the image processing system 206 is illustrated as combining at least a first machine learning model 208, a second machine learning model 210, and a corner detector 214, such that the image processing system is configured to receive an image sequence 204 and process at least one of the received images by applying the image(s)204A to the machine learning model, generating a mask 212, applying the mask 212 to the corner detector 214, and otherwise processing the received image 204A. It should be understood that in some example embodiments, the image processing system 206 and / or its components may be implemented individually and / or in different groups, and may incorporate additional components.

[0049] Regardless of the precise configuration of the image processing system 206 and its associated components, some example implementations of system 200 are also configured to perform perspective n-point estimation. In some such example implementations, system 200, such as through the operation of perspective n-point system 216, is configured to use points of interest and known local coordinates of the runway frame to determine the current attitude estimate 218 of camera 202, and thereby determine the attitude estimate of aircraft 100 relative to the runway or runway markings.

[0050] In some example embodiments, the perspective n-point system 216 is configured to determine the current attitude estimate 218 of the camera 202 in camera coordinates and transform the camera coordinates into corresponding runway-framed local coordinates 236, which are output for at least one of the guidance or control of the aircraft 100. In some such example embodiments, the runway-framed local coordinates are passed from the perspective n-point system 216 to the guidance system 220, control system 222, and / or another advanced system 110 of the aircraft 100.

[0051] In some example implementations, the perspective n-point system 216 is further configured to determine a confidence interval 228 associated with the current pose estimate 218. In some example implementations, the confidence interval may be a confidence value and / or a multidimensional confidence matrix.

[0052] In some such example implementations, system 200 is further configured to access one or more additional attitude estimates 218A and associated confidence intervals 228A. System 200 can then use the confidence intervals 228 and associated confidence intervals to perform sensor fusion on the current attitude estimate 218 and the one or more additional attitude estimates 218A to generate an updated current attitude estimate 218B. In some such example implementations, the system is further configured to output the updated current attitude estimate 218B for at least one of the guidance or control of aircraft 100, such as by providing the updated attitude estimate to aircraft 100's guidance system 220, control system 222, or another advanced system 110.

[0053] In some such example implementations, system 200 is further configured to output a current attitude estimate 218 for at least one of guidance or control of aircraft 100 during final approach. For example, the current attitude estimate is passed by perspective n-point system 216 to aircraft guidance system 220, aircraft control system 222 and / or another advanced system 110 of aircraft 100, where the current attitude estimate can be used for autonomous landing operations and / or to support pilots in performing landing operations.

[0054] In some example implementations, system 200 is configured to iteratively process image 204A captured by camera 202. In some such example implementations, another aspect of image processing system 206 and / or system 200 is configured to access earlier images, such as image 204B in an image sequence. The system may also access earlier masks, such as mask 212A, which includes corresponding pixel segments in the earlier image assigned to object categories related to the runway or runway markings. By accessing the earlier image and the earlier mask, system 200 can be configured to identify a portion of the earlier mask within the earlier image, and thereby identify a portion of the relevant airport where the runway is located. In example implementations involving cropping images (such as image 204A), system 200 can be configured to crop the image to a portion corresponding to a portion of the earlier image.

[0055] In some example implementations, system 200 uses and generates additional information that can be combined with the received image 204A, mask 212, and attitude estimation 218 to provide guidance and / or control information to support the landing operation of aircraft 100. In some such example implementations, the system is configured to determine the operational state 230 of the aircraft, such as the state of the propulsion system 112, electrical system 114, hydraulic system 116, environmental system 118, and / or any of the higher-level systems 110. In some such example implementations, image processing system 206 or other aspects of system 200 are configured to select a first machine learning model 208 or a second machine learning model 210 as the machine learning model to which image 204A is applied, based on the operational state. For example, the first machine learning model 208 may be configured and trained to perform single-class object detection and segmentation, and the second machine learning model 210 may be configured and trained to perform multi-class object detection and segmentation. In some example implementations, the operational state of aircraft 100 is determined based on information obtained from one or more of the higher-level systems 110.

[0056] In another example implementation of system 200 that uses and generates additional information, a given machine learning model, such as a first machine learning model 208 or a second machine learning model 210, is further configured, for example, to determine a confidence interval 232 associated with the detection of a relevant runway or relevant runway markings. In some such example implementations, another aspect of image processing system 206 or system 200 is further configured to apply a mask 212 to a confidence filter 234, which is configured to pass the mask 212 to corner detector 214 only if the confidence interval 232 is higher than a threshold confidence interval dynamically set based on the operating state 230 of aircraft 100.

[0057] Figure 3This is a block diagram of a system pipeline 300 that can be used in conjunction with a system (such as system 200) for supporting an aircraft approaching a runway at an airport, according to an example implementation disclosed herein. As discussed herein and otherwise disclosed, image sequences (such as image sequence 204 captured by (a plurality of) cameras 202) are applied to one or more trained machine learning models (such as, for example, a first machine learning model 208 and a second machine learning model 210) and processed to generate information that can be used to support the guidance and / or control of the aircraft 100 during landing operations. Figure 3 The system pipeline 300 shown presents an example method for acquiring and processing information for training machine learning models and providing information for the guidance and control of aircraft.

[0058] like Figure 3 As shown, system pipeline 300 includes block 302, which relates to constructing a dataset that can be used to train one or more machine learning models. In some example implementations of block 302, images 304 (which may include one or more instances of image sequence 204) are processed to generate a set of extracted frames 306. In some example implementations, the extracted frames 306 include frames from various image sequences and may include depictions of one or more features that one or more machine learning models will be trained to detect. For example, extracted frames 306 may include various views of runways, runway signs, and / or other image features related to the detection of the presence of runways or runway signs in the image. The extracted frames 306 are then annotated with one or more pose labels 307 such that the pose labels 307 and information associated with a given pose are linked to the corresponding images in the extracted frames 306.

[0059] like Figure 3 As shown, system pipeline 300 also includes block 308, which involves using image 304 and / or extracted frame 306 as part of a dataset for deep learning and training of a relevant machine learning model (e.g., 208, 210). Any deep learning method suitable for use with the relevant machine learning model can be used in connection with the example implementation of block 308. In some example implementations, at least one model is trained at block 310 using a deep learning method to detect runways and / or portions of runways. In some example implementations, at least one model is trained at block 312 using a deep learning method to detect one or more runway markers.

[0060] like Figure 3As shown, system pipeline 300 also includes block 314 involving attitude estimation. In some example implementations of block 314, trained machine learning models, such as first machine learning model 208 and / or second machine learning model 210, are configured, for example, to generate images in a defined format that can be used to perform attitude estimation, including masks and / or images processed in conjunction with the generated masks. For example, perspective n-point system 216 may receive a version of mask 212 and / or image 204A processed with the mask, and generate attitude estimation 218 that correlates relevant points of interest on the image with their associated points on the runway itself. At block 316, the attitude estimation is output in the form of information that can be used by aircraft 100 or its pilot in conjunction with landing operations.

[0061] Figure 4A An airport 400 is depicted, in which example implementations disclosed in this subject matter can be deployed, and this airport illustrates various aspects of the disclosure in this subject matter. As shown in Figure 4, airport 400 includes a runway 402 configured to allow aircraft (such as aircraft 100) to land. Figure 4A As shown, runway 402 also includes one or more runway markings 404. It should be understood that... Figure 4A The runway sign 404 shown is a non-exclusive example of runway signage that can be used on a given runway, and other signs (such as letters, numbers, lines, curves, geometric shapes, etc.) can be used in the implementation of runway signage. Figure 4A Point 406, representing a known location on runway 402 and / or runway marker 404, is also shown. In some example embodiments, the location of point 406 is known and represented by coordinates within the runway frame.

[0062] Figure 4B An image 408 of an airport 400 according to some example embodiments disclosed in this subject matter is shown. In some example embodiments, image 408 may be one of an image 204A in a sequence of images captured by a camera 202 on an aircraft 100 as it approaches the airport 400. Figure 4B As shown, image 408 depicts a view of runway 402 and one or more of runway signs 404. Although Figure 4B The example images depict an overall view of runway 402, but it should be understood that in some implementations, the images may provide a partial view of runway 402 and / or runway markings 404, depending on the orientation of the aircraft 100 and / or camera 202, as well as the conditions of the surrounding environment of the airport.

[0063] like Figure 4BAs shown, a mask 410 (e.g., mask 212) is applied to image 408 to identify regions in image 408 that include runway 402 and / or one or more runway markers 404. A set of points of interest 412 also exists on the image, corresponding to portions of the runway and / or runway markers in image 408. In some example embodiments disclosed in this subject matter, points of interest 412 are identified in image 408 and then matched with corresponding points 406 having known, runway-framed coordinates.

[0064] As discussed herein and otherwise disclosed, the points of interest 412 identified in a given image 408 and the corresponding known, runway-bound points 406 at the associated airport 400 are used in conjunction with perspective n-point estimation to determine the current attitude estimate 218 of the camera 202, and thereby determine the current attitude estimate 218 of the aircraft 100 relative to the runway 402 or the runway markings 404.

[0065] Perspective n-point estimation can be used to find a deterministic solution that maps real-world points to pixel values ​​in a given image of the environment in which those points are located. The camera matrix associated with a given camera (e.g., such as camera 202) can be represented as a scaling and principal point matrix. A And it can be expressed as shown in Formula 1 below:

[0066] (1)

[0067] In Formula 1, the first n Each point is considered to have real-world coordinates. X n , Y n , Z n ] t and camera coordinates[ U n , V n , W n ] t The existence of this point also makes... W Rays projected onto the plane =1 u n , v n The points of intersection. As shown in Formula 1, c x , c y It is a zoom of focal length and f x , f y This is the main point.

[0068] No. n The transformation of a point from the camera coordinate system to the global coordinate system can be expressed as shown in Equation 2 below:

[0069] (2)

[0070] In Formula 2, R It is the relevant fundamental rotation matrix and T It is a translation vector. Therefore, the point in the camera coordinate system is to the plane. W The projection of = 1 can be expressed as shown in Formula 3 below:

[0071] (3)

[0072] In Formula 3, s The scaling factor is unknown. To generate a transformation from real-world coordinates to a projection onto the image plane, equations 1, 2, and 3 can be combined and expressed as shown in equation 4 below:

[0073] (4)

[0074] As shown in Formula 4, the scaling factor s Rotation matrix R The value and translation vector T It is unknown. However, the example implementations disclosed in this subject matter involve solving for unknowns in the formulas. For example, the perspective n-point system 216 can use an interactive algorithm or a procedural method. One such method involves receiving four or more mappings from pixels to real-world coordinates and solving for the unknowns. In some such methods, the values ​​returned from a given algorithm or program are rotation and translation vectors to move from global coordinates to camera coordinates. In an example implementation for estimating the orientation of camera 202 (or aircraft 100) relative to the desired real-world landing orientation (which is represented by the origin of the global coordinate system), the origin of the camera's coordinate system must be transformed to a global coordinate point. This transformation can be expressed as listed in Formula 5 below:

[0075] (5)

[0076] Formula 5 can be adjusted to directly determine the linear positioning of the aircraft 100 (equipped with camera 202) relative to the origin of the landing point, as shown in Formula 6 below:

[0077] (6)

[0078] In some example implementations, a difference matrix is ​​constructed, taking into account the variation in the order of Euler angles for a conventional aircraft, to determine the attitude of a given aircraft 100 from the rotation matrix output. The rotation matrix from the camera coordinate system to the aircraft coordinate system representing stable horizontal flight can be expressed as listed in Equation 7 below:

[0079] (7)

[0080] In such an example implementation, the difference matrix between the output of SolvePnP and the stable level flight can be expressed as shown in Equation 8 below:

[0081] (8)

[0082] It should be understood that the difference matrix reveals the rotation matrix used to change the aircraft 100's current attitude from stable level flight. In some example implementations, this rotation matrix is ​​still based on the camera coordinate system, while the aircraft attitude is typically applied in a 3-2-1 Euler angle order (yaw, then pitch, then roll). To maintain the correct order in the formulas presented herein, the Euler angles are taken in a 2-1-3 order (such that […] X , Y , Z Mapping to [ W , U , V ]). Scroll ϕ , up and down θ and yaw ψ The combined rotation matrix can be expressed in camera coordinates as shown in Equation 9 below:

[0083] (9)

[0084] Formula 9 can be expanded and expressed as shown in Formula 10 below:

[0085] (10)

[0086] The matrices in Formula 10 can be multiplied together to establish the terms listed below in Formulas 11, 12, and 13:

[0087] Formula 11:

[0088] (11)

[0089] Formula 12:

[0090] (12)

[0091] Formula 13:

[0092] (13)

[0093] In some example implementations, terms from Equations 11, 12, and 13 are used to determine the attitude of the aircraft 100 in conventional aircraft coordinates from the difference matrix, which can then be used to allow the system 200 to generate information for guiding and / or controlling the aircraft as it approaches the airport 400 and the runway 402 thereon.

[0094] Figure 5 This is a flowchart illustrating various steps in a method 500 for supporting an aircraft 100 approaching a runway 402 on an airport 400, according to an example embodiment disclosed in this subject matter. As shown at block 502, the method includes receiving a sequence of images captured by an onboard camera(s). Some example embodiments of block 502 involve receiving a sequence of airport images captured by a camera mounted on an aircraft approaching the runway. In some such example embodiments, an image processing system 206 receives a sequence 204 of images captured by a camera 202 mounted on the aircraft 100 approaching the runway 402.

[0095] As shown at block 504, method 500 includes applying at least one image in an image sequence to a machine learning model. Some example implementations of block 504 involve applying an image to a machine learning model trained to perform object detection and segmentation, wherein a runway or runway markings on an airport are detected in the image, and wherein a mask is generated comprising pixel segments in the image assigned to an object category of the runway or runway markings. In some such example implementations, image 204A is applied by image processing system 206 or another aspect of system 200 to a machine learning model, such as a first machine learning model 208 or a second machine learning model 210, to perform object detection and segmentation and generate mask 212.

[0096] As discussed herein with respect to block 504, some example implementations of method 500 include object detection and image segmentation, as shown at block 504A. Some example implementations of block 504A involve performing object detection. For example, a first machine learning model 208 or a second machine learning model 210 may be trained and configured to detect runway 402 or runway marker 404, and may further identify interest points 412 in image 204A and / or image 408.

[0097] In some example implementations, as shown at block 504B, method 500 includes generating a mask. Some example implementations of block 504B involve performing segmentation to generate a mask. For example, a first machine learning model 208 or a second machine learning model 210 may be trained and configured to generate masks, such as mask 212 or mask 410. In some such example implementations, the mask includes pixel segments in the image assigned to object categories of the relevant runway or relevant runway sign. In some example implementations, object detection and segmentation is single-class object detection and segmentation, wherein only runways or runway signs are detected in the image, and the mask includes only pixel segments in the image assigned to object categories of the runway or runway sign. In some example implementations, object detection and segmentation is multi-class object detection and segmentation, wherein runways and runway signs are detected in the image, and the mask includes pixel segments in the image assigned to object categories of the runway and runway signs, with points of interest including points of interest on the runway and runway signs.

[0098] As shown at block 506, method 500 includes applying a mask to a corner detector. Some example implementations of block 506 involve applying a mask to a corner detector to detect points of interest on the mask, and thereby detecting points of interest on a runway or runway marking in the image. In some such example implementations, another aspect of image processing system 206 or system 200 applies a mask 212 to corner detector 214 to detect points of interest on the mask 212. (See reference...) Figure 4B The point of interest can be, for example, point of interest 412 on mask 410.

[0099] As shown at block 508, method 500 includes matching points of interest in an image with corresponding real-world points. Some example implementations of block 508 involve matching points of interest on a runway or runway marker in an image with corresponding points on the runway or runway marker having known local coordinates bounded by the runway. For example, another aspect of image processing system 206 or system 200 may be configured to match point 412 of interest in image 408 with corresponding point 406 on runway 402 or runway marker 404.

[0100] As shown at block 510, method 500 includes performing perspective n-point estimation. Some example implementations of block 510 involve performing perspective n-point estimation using points of interest and known runway-bound local coordinates to determine the current attitude estimate of the camera, and thereby determine the current attitude estimate of the aircraft relative to the runway or runway marker. For example, perspective n-point system 216 can determine the current attitude estimate 218 of camera 202 by using point of interest 412 and known runway-bound local coordinates 236 or point 406 to ascertain the attitude of aircraft 100 relative to runway 402 or runway marker 404.

[0101] As shown at block 512, method 500 includes outputting a current attitude estimate. Some example implementations of block 512 involve outputting the current attitude estimate for at least one of guidance or control of the aircraft during final approach. For example, system 200 may be configured to output attitude estimate 218 to guidance system 220 (e.g., including a GUI or other display device) or control system 222, wherein the attitude estimate may be used to guide or control aircraft 100 during final approach to airport 400.

[0102] In some example implementations, method 500 may include one or more additional blocks. For example, in some implementations, the method includes, as shown in block 514, converting a multi-channel image into a single-channel image. Some example implementations of block 514 relate to cases where the image is a multi-channel image, and the method further includes converting the multi-channel image into a single-channel image. For example, another aspect of image processing system 206 or system 200 may convert image 204A from a color image to a monochrome image. In another example, image 204A may be an image captured by a thermal camera, and a multi-channel thermal image may be converted into a single-channel image.

[0103] In some example implementations, method 500 includes, as shown in block 516, cropping an image. Some example implementations of block 516 involve cropping an image to reduce the camera's field of view and magnify only a portion of the airport where the runway or runway markings are located before the image is applied to a machine learning model. For example, before the image is applied to a first machine learning model 208 and / or a second machine learning model 210, image processing system 206 or another aspect of system 200 may crop image 204A to reduce the field of view of camera 202 and magnify only a portion of airport 400 where runway 402 or runway markings 404 are located.

[0104] In some example implementations involving cropping, method 500 includes, as shown in block 518, accessing an earlier image and a mask. Some example implementations of block 518 involve accessing an earlier image in an image sequence, where the earlier mask includes a corresponding pixel segment from the earlier image assigned to an object category of the runway or runway sign. For example, another aspect of image processing system 206 or system 200 may access an earlier image 204B from image sequence 204 and access an earlier mask 212A, which includes a corresponding pixel segment from the earlier image assigned to an object category of runway 402 or runway sign 404.

[0105] In some example implementations involving cropping, method 500 includes, as shown in block 520, identifying a portion of an earlier mask in an earlier image. Some example implementations of block 520 involve identifying a portion of an earlier mask in an earlier image, and thereby identifying a portion of the airport where the runway is located. For example, image processing system 206 or some other part of system 200 may be configured to identify a portion of an earlier mask 212A in an earlier image 204B, and thereby identify a portion of airport 400 where runway 402 is located.

[0106] In some example implementations involving cropping, method 500 includes, as shown in block 522, cropping an image to a portion corresponding to an earlier image. In some example implementations of block 522, cropping the image includes cropping the image to a portion corresponding to a portion of the earlier image. For example, another aspect of image processing system 206 or system 200 may be configured to crop image 204A to a portion corresponding to a portion of an earlier image 204B (such as the portion framing an earlier mask 212A).

[0107] In some example implementations, method 500 includes, as shown in block 524, determining the operational state of the aircraft. For example, system 200 may be configured to determine the operational state 230 of aircraft 100. In some example implementations, the system may receive information associated with the operational state of the aircraft from one or more advanced systems of aircraft 100 and / or from one or more additional information sources 226.

[0108] In some example implementations, method 500 includes, as shown in block 526, selecting a first machine learning model or a second machine learning model. Some example implementations of block 526 involve selecting a first machine learning model or a second machine learning model as the machine learning model applied to the image, wherein the first machine learning model is trained to perform single-class object detection and segmentation, and the second machine learning model is trained to perform multi-class object detection and segmentation. For example, another aspect of image processing system 206 or system 200 may be configured to select a first machine learning model 208 or a second machine learning model 210 as the machine learning model applied to image 204A, wherein the first machine learning model is trained to perform single-class object detection and segmentation, and the second machine learning model is trained to perform multi-class object detection and segmentation.

[0109] In some example implementations, method 500 includes, as shown in block 528, determining a confidence interval for object detection. Some example implementations of block 528 involve a machine learning model further configured to determine a confidence interval associated with the detection of a runway or runway marker. For example, a first machine learning model 208 and / or a second machine learning model 210 may be configured to determine a confidence interval 232 associated with the detection of runway 402 or runway marker 404.

[0110] In some example implementations, method 500 includes, as shown in block 530, applying a mask to a confidence filter. Some example implementations of block 530 involve applying a mask to a confidence filter that passes the mask to the corner detector only if the confidence interval is higher than a threshold confidence interval dynamically set based on the operational state of the aircraft. For example, another aspect of image processing system 206 or system 200 may be configured to apply a mask 212 to confidence filter 234, which is configured to pass the mask to corner detector 214 only if the confidence interval 232 is higher than a threshold confidence interval dynamically set by the image processing system or another aspect of system 200 based on the operational state 230 of the aircraft 100.

[0111] In some example implementations, method 500 includes, as shown in block 532, determining a pose estimate in camera coordinates. Some example implementations of block 532 involve performing perspective n-point estimation by determining the current pose estimate of the camera in at least camera coordinates. For example, another aspect of perspective n-point system 216 or system 200 may be configured to determine a current pose estimate 218 of camera 202.

[0112] In some example implementations, method 500 includes, as shown in block 534, transforming camera coordinates to runway-framed coordinates. Some example implementations of block 534 involve transforming the camera coordinates to corresponding runway-framed local coordinates, which are output for use in at least one of the guidance or control of the aircraft. For example, another aspect of the perspective n-point system 216 or system 200 may be configured to transform camera coordinates to runway-framed local coordinates 236, which are output by the perspective n-point system or another aspect of system 200 for use by the guidance system 220, control system 222, or another advanced system 110 of the aircraft 100.

[0113] In some example implementations, method 500 includes, as shown in block 536, determining a confidence interval for the attitude estimate. In some example implementations, the confidence interval may take the form of a multidimensional confidence matrix. Some example implementations of block 536 involve performing a perspective n-point estimation by at least further determining a confidence interval associated with the current attitude estimate. For example, another aspect of perspective n-point system 216 or system 200 may be configured to determine a confidence interval 228 associated with the current attitude estimate 218.

[0114] In some example implementations, method 500 includes, as shown in block 538, accessing additional pose estimates and confidence intervals. Some example implementations of block 538 involve accessing one or more additional pose estimates and associated confidence intervals. For example, another aspect of perspective n-point system 216 or system 200 may be configured to access one or more additional pose estimates 218A and one or more associated confidence intervals 228A.

[0115] In some example implementations, method 500 includes, as shown in block 540, performing sensor fusion. Some example implementations of block 540 involve performing sensor fusion of a current attitude estimate and one or more additional attitude estimates using confidence intervals and associated confidence intervals to generate an updated current attitude estimate. For example, another aspect of perspective n-point system 216 or system 200 may be configured to perform sensor fusion of a current attitude estimate 218 and at least an additional attitude estimate 218A, as well as confidence interval 228 and at least an associated confidence interval 228A, to generate an updated attitude estimate 218B.

[0116] In some example implementations, method 500 includes, as shown in block 542, outputting an updated current attitude estimate. Some example implementations of block 542 involve outputting an updated current attitude estimate for use in at least one of the guidance or control of the aircraft. For example, another aspect of the perspective n-point system 216 or system 200 may be configured to output an updated attitude estimate 218B for use by the guidance system 220, control system 222, or another advanced system 110 of the aircraft 100.

[0117] According to the example implementations disclosed herein, system 200 and its subsystems (including camera 202, pose estimation system 205, image processing system 206, first machine learning model 208, second machine learning model 210, corner detector 214, perspective n-point system 216, guidance system 220, control system 222, and confidence filter 234) can be implemented by various components. Components used to implement the system and its subsystems can include hardware, either alone or under the guidance of one or more computer programs from a computer-readable storage medium. In some examples, one or more devices can be configured to act as or otherwise implement the system and its subsystems shown and described herein. In examples involving more than one device, the respective devices can be connected to or otherwise communicate with each other in many different ways, such as directly or indirectly via wired or wireless networks.

[0118] Figure 6 An apparatus 600 according to some exemplary embodiments disclosed in this subject matter is illustrated. Generally, the apparatus of the exemplary embodiments disclosed in this subject matter may be included, incorporated into, or embodied in one or more fixed or portable electronic devices. Examples of suitable electronic devices include smartphones, tablet computers, laptop computers, desktop computers, workstation computers, server computers, etc. The apparatus may include one or more of each of a plurality of components, such as, for example, a processing circuitry system 602 (e.g., a processor unit) connected to memory 604 (e.g., a storage device).

[0119] The processing circuit system 602 may consist of one or more processors individually or in combination with one or more memories. A processing circuit system is generally any computer hardware capable of processing information, such as, for example, data, computer programs, and / or other suitable electronic information. A processing circuit system consists of a collection of electronic circuits, some of which may be packaged as integrated circuits or multiple interconnected integrated circuits (integrated circuits are sometimes more commonly referred to as “chips”). The processing circuit system may be configured to execute computer programs, which may be stored on the processing circuit system or otherwise stored in memory 604 (on the same device or another device).

[0120] The processing circuitry system 602 may be multiple processors, a multi-core processor, or some other type of processor, depending on the specific implementation. Furthermore, the processing circuitry system may be implemented using multiple heterogeneous processor systems, where a main processor and one or more secondary processors reside on a single chip. As another illustrative example, the processing circuitry system may be a symmetric multiprocessor system comprising multiple processors of the same type. In yet another example, the processing circuitry system may be embodied as or otherwise include one or more ASICs, FPGAs, etc. Therefore, while the processing circuitry system is capable of executing computer programs to perform one or more functions, the various examples of processing circuitry systems are also capable of performing one or more functions without the assistance of a computer program. In any instance, the processing circuitry system may be appropriately programmed to perform functions or operations according to the exemplary embodiments disclosed in this subject matter.

[0121] Memory 604 is typically any computer hardware capable of temporarily and / or permanently storing information such as, for example, data, computer programs (e.g., computer-readable program code 606), and / or other suitable information. Memory may include volatile and / or non-volatile memory and may be fixed or removable. Examples of suitable memory include random access memory (RAM), read-only memory (ROM), hard disk drive, flash memory, thumb drive, removable computer disk, optical disk, magnetic tape, or some combination thereof. Optical disk may include compact disc-read-only memory (CD-ROM), compact disc-read / write (CD-R / W), DVD, etc. In various contexts, memory may be referred to as a computer-readable storage medium. A computer-readable storage medium is a non-transitory device capable of storing information and is distinguished from a computer-readable transmission medium (such as an electronic transient signal capable of carrying information from one location to another). As used herein, computer-readable medium may generally refer to either a computer-readable storage medium or a computer-readable transmission medium.

[0122] In addition to memory 604, processing circuitry system 602 may also be connected to one or more interfaces for displaying, transmitting, and / or receiving information. Interfaces may include communication interface 608 (e.g., a communication unit) and / or one or more user interfaces. Communication interfaces may be configured to transmit and / or receive information, such as to and / or from (multiple) other devices, (multiple) networks, etc. Communication interfaces may be configured to transmit and / or receive information via physical (wired) and / or wireless communication links. Examples of suitable communication interfaces include network interface controllers (NICs), wireless NICs (WNICs), etc.

[0123] The user interface may include a display 610 and / or one or more user input interfaces 612 (e.g., input / output units). The display may be configured to present or otherwise display information to a user; suitable examples of displays include liquid crystal displays (LCDs), light-emitting diode displays (LEDs), plasma display panels (PDPs), etc. The user input interface may be wired or wireless and may be configured to receive information from a user into the device, such as for processing, storage, and / or display. Suitable examples of user input interfaces include microphones, image or video capture devices, keyboards or keypads, joysticks, touch-sensitive surfaces (separate from or integrated into the touchscreen), biometric sensors, etc. The user interface may further include one or more interfaces for communicating with peripheral devices, such as printers, scanners, etc.

[0124] As indicated above, program code instructions can be stored in memory and executed by a processing circuitry system programmed thereon to implement the functions of the systems, subsystems, tools, and their respective elements described herein. It should be understood that any suitable program code instructions can be loaded from a computer-readable storage medium onto a computer or other programmable device to produce a particular machine, such that the particular machine becomes a component for implementing the functions specified herein. These program code instructions can also be stored in a computer-readable storage medium that can instruct a computer, processing circuitry system, or other programmable device to operate in a particular manner, thereby producing a particular machine or article of manufacture. Instructions stored in a computer-readable storage medium can produce an article of manufacture, wherein the article of manufacture becomes a component for implementing the functions described herein. Program code instructions can be retrieved from a computer-readable storage medium and loaded into a computer, processing circuitry system, or other programmable device to configure the computer, processing circuitry system, or other programmable device to perform operations that will be performed on or by the computer, processing circuitry system, or other programmable device.

[0125] The retrieval, loading, and execution of program code instructions can be performed sequentially, such that one instruction is retrieved, loaded, and executed at a time. In some implementations, retrieval, loading, and / or execution can be performed in parallel, such that multiple instructions are retrieved, loaded, and / or executed together. The execution of program code instructions can produce computer-implemented processes, such that the instructions, which are executed by a computer, processing circuitry system, or other programmable device, provide operations for carrying out the functions described herein.

[0126] The processing circuitry executes instructions, or stores instructions in a computer-readable storage medium, supporting combinations of operations for performing specified functions. In this way, device 600 may include a processing circuitry 602 and a computer-readable storage medium or memory 604 coupled to the processing circuitry, wherein the processing circuitry is configured to execute computer-readable program code 606 stored in the memory. It should also be understood that one or more functions, and combinations thereof, may be implemented by a dedicated hardware-based computer system and / or processing circuitry that performs the specified function, or a combination of dedicated hardware and program code instructions.

[0127] Furthermore, this disclosure includes embodiments pursuant to the following provisions:

[0128] Clause 1: An apparatus for supporting an aircraft approaching a runway at an airport, the apparatus comprising: a memory configured to store computer-readable program code; and a processing circuitry configured to access the memory and execute the computer-readable program code such that the apparatus at least: receives a sequence of images of the airport captured by a camera mounted on the aircraft approaching the runway; and for at least one image of the image sequence, applies the image to a machine learning model trained to perform object detection and segmentation, wherein the runway or runway markings on the runway are detected in the image, and wherein a mask is generated, the mask including markings on the runway or runway markings on the runway in the image. The image is defined as follows: a pixel segment representing the object category of the runway marker; the mask is applied to a corner detector to detect points of interest on the mask, and thereby detects points of interest on the runway or runway marker in the image; the points of interest on the runway or runway marker in the image are matched with corresponding points on the runway or runway marker having known local coordinates bounded by the runway; perspective n-point estimation is performed using the points of interest and the known local coordinates bounded by the runway to determine the current attitude estimate of the camera, and thereby determine the current attitude estimate of the aircraft relative to the runway or runway marker; and the current attitude estimate is output for at least one of guidance or control of the aircraft during final approach.

[0129] Clause 2: The apparatus according to Clause 1, wherein the image is a multi-channel image, and the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further convert the multi-channel image into a single-channel image.

[0130] Clause 3: The apparatus according to any one of Clauses 1 to 2, wherein the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further crop the image to reduce the field of view of the camera and magnify only a portion of the image in which the runway or the runway markings are located before the image is applied to the machine learning model.

[0131] Clause 4: An apparatus according to any one of Clauses 1 to 3, wherein the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further at least: access an earlier image in the image sequence, and the earlier mask includes a corresponding pixel segment in the earlier image assigned to the object category of the runway or the runway sign; and identify a portion of the earlier image that frames the earlier mask, and thereby identify the portion of the airport where the runway is located, and wherein causing the apparatus to crop the image includes causing the apparatus to crop the image to a portion of the image corresponding to the portion of the earlier image.

[0132] Clause 5: The apparatus according to any one of Clauses 1 to 4, wherein the object detection and segmentation is at least one-class object detection and segmentation, wherein only the runway or the runway sign is detected in the image, and the mask includes only the pixel segments in the image assigned to the object class of the runway or the runway sign.

[0133] Clause 6: The apparatus according to any one of Clauses 1 to 5, wherein the object detection and segmentation is multi-class object detection and segmentation, wherein the runway and the runway sign are detected in the image, and the mask includes pixel segments in the image assigned to object categories of the runway and the runway sign, and the points of interest include points of interest on the runway and the runway sign.

[0134] Clause 7: The apparatus according to any one of Clauses 1 to 6, wherein the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further at least: determine the operational state of the aircraft; and, based on the operational state, select a first machine learning model or a second machine learning model as the machine learning model to which the image is applied, wherein the first machine learning model is trained to perform at least single-class object detection and segmentation, and the second machine learning model is trained to perform multi-class object detection and segmentation.

[0135] Clause 8: The apparatus according to any one of Clauses 1 to 7, wherein the machine learning model is further configured to determine a confidence interval or multidimensional confidence matrix associated with the detection of the runway or the runway markings, and the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further apply the mask to a confidence filter, the confidence filter being configured to pass the mask to the corner detector only if the confidence interval is higher than a threshold confidence interval dynamically set based on the operational state of the aircraft.

[0136] Clause 9: The apparatus according to any one of Clauses 1 to 8, wherein causing the apparatus to perform the perspective n-point estimation includes causing the apparatus to determine the current attitude estimate of the camera in camera coordinates and transforming the camera coordinates into local coordinates corresponding to the runway frame, the local coordinates being output for at least one of the guidance or control of the aircraft.

[0137] Clause 10: An apparatus according to any one of Clauses 1 to 9, wherein causing the apparatus to perform the perspective n-point estimation further comprises causing the apparatus to determine a confidence interval associated with the current attitude estimation, and the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further at least: access one or more additional attitude estimates and associated confidence intervals; perform sensor fusion of the current attitude estimation and the one or more additional attitude estimates using the confidence intervals and the associated confidence intervals to generate an updated current attitude estimation; and output the updated current attitude estimation for at least one of the guidance or control of the aircraft.

[0138] Clause 11: A method for supporting an aircraft approaching a runway at an airport, the method comprising: receiving a sequence of images of the airport captured by a camera mounted on the aircraft approaching the runway; and for at least one image in the image sequence, applying the image to a machine learning model trained to perform object detection and segmentation, wherein the runway or runway markings on the runway are detected in the image, and wherein a mask is generated comprising pixel segments in the image assigned to an object category of the runway or runway markings; and applying the mask to a corner detector to detect the mask. The camera identifies points of interest on the runway or runway markers in the image; matches the points of interest on the runway or runway markers with corresponding points on the runway or runway markers that have known local coordinates bounded by the runway; performs perspective n-point estimation using the points of interest and the known local coordinates bounded by the runway to determine the current attitude estimate of the camera, and thereby determines the current attitude estimate of the aircraft relative to the runway or runway markers; and outputs the current attitude estimate for at least one of guidance or control of the aircraft during final approach.

[0139] Clause 12: The method according to Clause 11, wherein the image is a multi-channel image, and the method further includes converting the multi-channel image into a single-channel image.

[0140] Clause 13: The method according to any one of Clauses 11 to 12 further includes cropping the image to reduce the field of view of the camera and magnifying only a portion of the image where the runway or the runway markings are located before applying the image to the machine learning model.

[0141] Clause 14: The method according to any one of Clauses 11 to 13 further comprises: accessing an earlier image in the image sequence, and the earlier masking includes a corresponding pixel segment in the earlier image assigned to an object category of the runway or the runway sign; and identifying a portion of the earlier image that frames the earlier masking, and thereby identifying the portion of the airport where the runway is located, and wherein cropping the image comprises cropping the image to a portion of the image corresponding to the portion of the earlier image.

[0142] Clause 15: The method according to any one of Clauses 11 to 14, wherein the object detection and segmentation is at least single-class object detection and segmentation, wherein only the runway or the runway sign is detected in the image, and the mask includes only pixel segments in the image assigned to the object category of the runway or the runway sign.

[0143] Clause 16: The method according to any one of Clauses 11 to 15, wherein the object detection and segmentation is multi-class object detection and segmentation, wherein the runway and the runway sign are detected in the image, and the mask includes pixel segments in the image assigned to object categories of the runway and the runway sign, and the points of interest include points of interest on the runway and the runway sign.

[0144] Clause 17: The method according to any one of Clauses 11 to 16 further includes: determining the operational state of the aircraft; and, based on the operational state, selecting a first machine learning model or a second machine learning model as the machine learning model to which the image is applied, wherein the first machine learning model is trained to perform at least single-class object detection and segmentation, and the second machine learning model is trained to perform multi-class object detection and segmentation.

[0145] Clause 18: The method of any one of Clauses 11 to 17, wherein the machine learning model further determines a confidence interval or multidimensional confidence matrix associated with the detection of the runway or the runway markings, and the method further includes applying the mask to a confidence filter that passes the mask to the corner detector only if the confidence interval is higher than a threshold confidence interval dynamically set based on the operational state of the aircraft.

[0146] Clause 19: The method according to any one of Clauses 11 to 18, wherein performing perspective n-point estimation includes determining the current attitude estimate of the camera in camera coordinates and transforming the camera coordinates into local coordinates of the corresponding runway frame, the local coordinates being output for at least one of guidance or control of the aircraft.

[0147] Clause 20: The method according to any one of Clauses 11 to 19, wherein performing the perspective n-point estimation further includes determining a confidence interval associated with the current attitude estimation, and the method further includes: accessing one or more additional attitude estimates and the associated confidence interval; using the confidence interval and the associated confidence interval, performing sensor fusion of the current attitude estimation and the one or more additional attitude estimates to generate an updated current attitude estimation; and outputting the updated current attitude estimation for at least one of the guidance or control of the aircraft.

[0148] Those skilled in the art to which this disclosure pertains, having benefited from the teachings set forth in the foregoing description and associated drawings, will conceive of numerous modifications and other embodiments of the disclosure set forth herein. Therefore, it should be understood that this disclosure is not limited to the specific embodiments disclosed, and that modifications and other embodiments are also intended to be included within the scope of the appended claims. Furthermore, although the foregoing description and associated drawings describe exemplary embodiments in the context of certain example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions can be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, for example, different combinations of elements and / or functions different from those explicitly described above are also contemplated, as may be set forth in some of the appended claims. Although specific terminology is used herein, it is used only in a general and descriptive sense and not for limiting purposes.

Claims

1. A device for supporting an aircraft approaching a runway at an airport, comprising: Memory configured to store computer-readable program code; as well as A processing circuitry system configured to access the memory and execute the computer-readable program code to cause the device to at least: Receive a sequence of images of the airport captured by a camera mounted on the aircraft approaching the runway; and for at least one image in the sequence of images, The image is applied to a machine learning model trained to perform object detection and segmentation, wherein at least one of the runway on the airport or the runway sign on the runway is detected in the image, and wherein a mask is generated, the mask comprising pixel segments in the image assigned to the object category of the runway or the runway sign. The mask is applied to the corner detector to detect one or more points of interest on the mask, and thereby detect one or more points of interest on the runway or the runway marking in the image; Match one or more points of interest on the runway or runway sign in the image with one or more corresponding points on the airport or runway sign that have known local coordinates bounded by the runway. Perspective n-point estimation is performed using the one or more points of interest and the known local coordinates of the runway to determine the attitude estimate of the camera, and thereby determine the attitude estimate of the aircraft relative to the runway or the runway markings, wherein the attitude estimate is used for at least one of the guidance or control of the aircraft. Before the image is applied to the machine learning model, the image is cropped to reduce the camera's field of view, and only a portion of the image containing the runway or the runway marker is magnified. Access an earlier image in the image sequence, and an earlier mask, the earlier mask including the corresponding pixel segment in the earlier image assigned to the object category of the runway or the runway sign; as well as Identify the portion of the earlier masking in the earlier image, and thereby identify the portion of the airport where the runway is located. The cropping of the image by the device includes cropping the image into a portion of the image corresponding to the portion of the earlier image.

2. The apparatus of claim 1, wherein the image is a multi-channel image, and the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further convert the multi-channel image into a single-channel image.

3. The apparatus of claim 1 or 2, wherein the object detection and segmentation is at least single-class object detection and segmentation, wherein only the runway or the runway sign is detected in the image, and the mask includes only the pixel segments in the image assigned to the object category of the runway or the runway sign.

4. The apparatus of claim 1 or 2, wherein the object detection and segmentation is multi-class object detection and segmentation, wherein the runway and the runway sign are detected in the image, and the mask includes pixel segments in the image assigned to object categories of the runway and the runway sign, and the one or more points of interest include a first point of interest on the runway and the runway sign.

5. The apparatus of claim 1 or 2, wherein the processing circuitry is configured to execute the computer-readable program code to further at least: Determine the operational status of the aircraft; and Based on the operation state, a first machine learning model or a second machine learning model is selected as the machine learning model to which the image is applied, wherein the first machine learning model is trained to perform at least single-class object detection and segmentation, and the second machine learning model is trained to perform multi-class object detection and segmentation.

6. The apparatus of claim 1 or 2, wherein the machine learning model is further configured to determine a confidence interval or multidimensional confidence matrix associated with the detection of the runway or the runway markings, and the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further apply the mask to a confidence filter, the confidence filter being configured to pass the mask to the corner detector only if the confidence interval is higher than a threshold confidence interval dynamically set based on the operational state of the aircraft.

7. The apparatus of claim 1 or 2, wherein causing the apparatus to perform the perspective n-point estimation comprises causing the apparatus to determine the attitude estimate of the camera in camera coordinates and transforming the camera coordinates into local coordinates corresponding to the runway frame, the local coordinates being output for at least one of the guidance or control of the aircraft.

8. The apparatus of claim 1, wherein causing the apparatus to perform the perspective n-point estimation further comprises causing the apparatus to determine a confidence interval associated with the pose estimation, and the processing circuitry is configured to execute the computer-readable program code to cause the apparatus to further at least: Access one or more additional pose estimates and associated confidence intervals; Using the confidence interval and the associated confidence interval, perform sensor fusion of the attitude estimation and the one or more additional attitude estimations to generate an updated current attitude estimation; and The updated current attitude estimate is output for use in at least one of the guidance or control of the aircraft.

9. A method for supporting an aircraft to approach a runway at an airport, comprising: A system including a processor receives a sequence of images of the airport captured by a camera mounted on an aircraft approaching the runway on the airport. And for at least one image in the image sequence, The image is applied to a machine learning model trained to perform object detection and segmentation, wherein the runway or runway markings on the airport are detected in the image, and wherein a mask is generated, the mask comprising pixel segments in the image assigned to the object category of the runway or runway markings. The mask is applied to the corner detector to detect one or more points of interest on the mask, and thereby detect one or more points of interest on the runway or the runway marking in the image; Match one or more points of interest on the airport or runway sign in the image with corresponding points on the runway or runway sign that have known local coordinates bounded by the runway. Based on the one or more points of interest and the known local coordinates of the runway frame, determine the attitude estimate of the aircraft relative to the runway or the runway markings; Output the attitude estimate for use in at least one of the guidance or control of the aircraft during final approach; Before applying the image to the machine learning model, the image is cropped to reduce the camera's field of view, and only a portion of the image containing the runway or the runway marker is magnified. Access an earlier image in the image sequence, and an earlier mask, the earlier mask including the corresponding pixel segment in the earlier image assigned to the object category of the runway or the runway sign; as well as Identify a portion of the earlier mask in the earlier image, and thereby identify the portion of the airport where the runway is located. Cropping the image includes cropping the image into a portion of the image that corresponds to the portion of the earlier image.

10. The method of claim 9, wherein the image is a multi-channel image, and the method further comprises converting the multi-channel image into a single-channel image.

11. The method of claim 9, wherein the object detection and segmentation is at least single-class object detection and segmentation, wherein only the runway or the runway sign is detected in the image, and the mask includes only the pixel segments in the image assigned to the object category of the runway or the runway sign.

12. The method of claim 9, wherein the object detection and segmentation is multi-class object detection and segmentation, wherein the runway and the runway sign are detected in the image, and the mask includes pixel segments in the image assigned to object categories of the runway and the runway sign, and the points of interest include first points of interest on the runway and the runway sign.

13. The method of claim 9, further comprising: Determine the operational status of the aircraft; as well as Based on the operation state, a first machine learning model or a second machine learning model is selected as the machine learning model applied to the image. The first machine learning model is trained to perform single-class object detection and segmentation, and the second machine learning model is trained to perform multi-class object detection and segmentation.

14. The method of claim 9, wherein the machine learning model further determines a confidence interval or multidimensional confidence matrix associated with the detection of the runway or the runway markings, and the method further comprises applying the mask to a confidence filter, the confidence filter passing the mask to the corner detector only if the confidence interval is higher than a threshold confidence interval dynamically set based on the operational state of the aircraft.

15. The method of claim 9, wherein determining the attitude estimate of the aircraft relative to the runway or the runway marking includes performing a perspective n-point estimation, wherein performing the perspective n-point estimation includes determining the attitude estimate of the camera in camera coordinates and transforming the camera coordinates into local coordinates corresponding to the runway frame, the local coordinates being output for at least one of the guidance or control of the aircraft.

16. The method of claim 9, wherein determining the attitude estimate of the aircraft relative to the runway or the runway marking includes performing a perspective n-point estimation, wherein performing the perspective n-point estimation includes determining a confidence interval associated with the attitude estimate, and the method further includes: Access one or more additional pose estimates and associated confidence intervals; Using the confidence interval and the associated confidence interval, perform sensor fusion of the attitude estimation and the one or more additional attitude estimations to generate an updated current attitude estimation; as well as The updated current attitude estimate is output for use in at least one of the guidance or control of the aircraft.

Images