Systems and methods for detecting obstacles in aerial systems

Abstract

The invention relates to a system and method for detecting obstacles in an aerial system. An automatic detection and avoidance system provides a pilot with high fidelity knowledge of a physical state of an aircraft and notifies the pilot of any deviation in a desired state based on a predictive model. The automatic detection and avoidance system may include a processor and sensor equipment operably coupled to the processor to detect non-cooperative obstructions within a first airspace proximate to the aircraft. A sensor apparatus may include a radar that radially scans a first airspace and a camera that scans a second airspace within the first airspace.

Description

[0001] This application is a divisional application of Chinese patent application 201810774962.0, filed on July 16, 2018, entitled “System and method for detecting obstacles in an aerial system”. Technical Field

[0002] The present invention relates to the field of flight control systems, methods and apparatus; even more particularly, to systems, methods and apparatus for detecting and automatically navigating around stationary and / or moving obstacles. Background Art

[0003] Aircraft technology, including unmanned aerial vehicle ("UAV") technology, is a valuable tool for mission profiles involving intelligence, surveillance, reconnaissance, and equipment transfer. In operation, an aircraft may encounter both large and small obstacles within the aircraft's airspace, which obstacles may be fixed or moving and whose locations are not known in advance. Traditional forms of obstacle detection and avoidance within an aircraft rely on the pilot to provide the critical responsibility of looking outside the aircraft in order to ensure that the aircraft is not on a collision course with an obstacle, such as another aircraft. Existing technologies for preventing aircraft from colliding with obstacles, including the Global Positioning System ("GPS"), are generally inadequate because many obstacles cannot be identified (or quickly identified) via GPS equipment, and GPS accuracy performance varies widely across environments depending on altitude or terrain.

[0004] However, the commercial aviation industry has adopted the Traffic Collision Avoidance System ("TCAS") as a standard for collision avoidance, which allows cooperative aircraft to locate and avoid each other. As can be understood, a cooperative aircraft refers to an aircraft that can cooperate with cooperative sensors. For example, a cooperative aircraft can be equipped with a TCAS (TCAS II or earlier generation), such as a mode S or mode C transponder, ADS-B, or, alternatively, using other transmissions and interference messages (such as ADS-B). Although TCAS proposes a solution to the problem of detecting and avoiding obstacles for UAVs, TCAS can only achieve this goal if each UAV and obstacle contains a transponder. In other words, the cooperative target sends its position and heading (e.g., GPS position and velocity vector) to other aircraft via radio (e.g., using ADS-B or other methods), while non-cooperative obstacles do not send position and heading information to other objects (multi-rotor aircraft, general aircraft, birds, etc.). In addition, current flight control systems designed for detecting and avoiding non-cooperative obstacles use expensive radar arrays to track obstacle obstacles and are generally only used with large-scale aircraft.

[0005] Therefore, there is a need for a system to detect and avoid non-cooperative UAVs, aircraft, and obstacles while being affordable for use by both large and small aircraft. Additionally, there is a need for an open architecture system that enables rapid introduction of new capabilities, increased safety, and propagation of functionality without significant expense or recertification. A system for detecting and avoiding non-cooperative obstacle collision course in an aircraft, such as the system disclosed herein, addresses these needs and enables new capabilities to be introduced quickly with minimal cost or certification burden. Summary of the invention

[0006] The present invention relates to flight control systems, methods and apparatus; even more specifically, to systems, methods and techniques for detecting and automatically navigating around stationary and / or moving obstacles; and even more specifically, to sensor and vehicle systems, methods and techniques for detecting and automatically navigating around obstacles. As will be discussed, an automatic detection and avoidance system detects and avoids non-cooperative obstacles to provide significant benefits to a variety of operators in a variety of applications. By way of example and not limitation, the automatic detection and avoidance system can be applied to smaller autonomous aircraft where compliance with TCAS rules is uneconomical, impossible or undesirable, or as a substitute for air-to-air radar arrays on larger aircraft to detect and avoid non-cooperative obstacles that would otherwise go undetected.

[0007] According to a first aspect, an obstacle detection system for use in an aircraft comprises: a sensor device / sensor payload and a processor, the sensor device being used to detect a non-cooperative obstacle in a first airspace adjacent to the aircraft, the sensor device comprising a radar and a camera, the radar being used to radially scan the first airspace to generate radar information having a first resolution, the camera being used to image a second airspace within the first airspace to generate optical information at a second resolution higher than the first resolution; the processor being operably coupled to the sensor device, wherein the processor is configured to determine a position of the non-cooperative obstacle and to identify the non-cooperative obstacle based on the radar information and the optical information.

[0008] In some aspects, the camera is configured to pan and tilt.

[0009] In some aspects, the camera includes a long wave infrared sensor.

[0010] In certain aspects, the camera includes a visible-near infrared electro-optical (EO) sensor.

[0011] In certain aspects, the optical information includes at least one of a thermal cross section and an optical cross section.

[0012] In some aspects, the radar information includes radar cross section.

[0013] In certain aspects, the processor is configured to divide the digital representation of the first spatial domain into a plurality of radial sectors.

[0014] In certain aspects, the second airspace is located in one of the plurality of radial sectors.

[0015] In certain aspects, the radar information includes two-dimensional (2D) locations of non-cooperative obstacles in a first airspace, and the optical information includes azimuth locations of non-cooperative obstacles in a second airspace.

[0016] In some aspects, the radar is a mechanically rotating marine radar.

[0017] In certain aspects, the radar information includes a two-dimensional (2D) radial map of the first airspace.

[0018] In certain aspects, the 2D radial map is divided into a plurality of radial sectors.

[0019] In certain aspects, the processor is configured to identify a radial sector associated with a non-cooperative obstacle from the plurality of radial sectors associated with non-cooperative obstacles and to instruct the camera to scan the radial sector.

[0020] In certain aspects, the processor is configured to generate a predicted flight path for a non-cooperative obstacle based at least in part on the radar information and the optical information.

[0021] In certain aspects, the processor is configured to generate an obstacle-avoidance navigational rotue based at least in part on the predicted flight path to avoid collision with a non-cooperative obstacle.

[0022] In certain aspects, the processor is configured to generate one or more commands to actuate one or more flight controllers to follow an obstacle avoidance navigation path.

[0023] In certain aspects, the aircraft further includes a human machine interface operably coupled to the processor to provide an interface between the pilot and the aircraft.

[0024] In certain aspects, the human-machine interface is configured to enable a pilot to control and communicate with the processor.

[0025] In certain aspects, the human-machine interface is configured to display one or more parameters of the sensor equipment.

[0026] In some aspects, the human-machine interface includes a touch screen display.

[0027] In certain aspects, the human machine interface machine is configured to communicate commands between the processor and the pilot via a voice-based system.

[0028] In certain aspects, the first airspace provides a 360 degree field of view about the aircraft.

[0029] In certain aspects, the 360 ​​degree field resides in a plane parallel to the flight path defined by the aircraft.

[0030] According to a second aspect, a method for detecting and avoiding non-cooperative obstacles during operation of an aircraft includes: scanning a first airspace using a radar system to generate radar information having a first resolution; imaging a second airspace using a camera to generate optical information at a second resolution higher than the first resolution, wherein the second airspace is within the first airspace; tracking non-cooperative obstacles based at least in part on the radar information and the optical information; generating a predicted flight path based at least in part on the radar information and the optical information; generating an obstacle avoidance navigation route to avoid the non-cooperative obstacles; and communicating the obstacle avoidance navigation route to a flight control system of the aircraft.

[0031] In certain aspects, the aircraft is configured to automatically execute the generated obstacle avoidance navigation path.

[0032] In some aspects, the camera is configured to pan and tilt.

[0033] In some aspects, the camera includes a long wave infrared sensor.

[0034] In certain aspects, the method further comprises the step of dividing, using a processor, the first spatial domain into a plurality of radial sectors.

[0035] In certain aspects, the second spatial region is one of the plurality of radial sectors.

[0036] In certain aspects, the method further comprises the step of determining, using the camera, an azimuth position of the non-cooperative obstacle within the second airspace.

[0037] In some respects, the radar is a mechanically rotating marine radar.

[0038] In certain aspects, the radar information includes a two-dimensional (2D) radial map of the first airspace.

[0039] In certain aspects, the 2D radial map is divided into a plurality of radial sectors.

[0040] In certain aspects, the method further comprises the step of identifying, via a processor, a radial sector associated with a non-cooperative obstacle from the plurality of radial sectors associated with non-cooperative obstacles.

[0041] In certain aspects, the method further includes the step of generating, via the processor, a predicted flight path for the non-cooperative obstacle based at least in part on data received from the sensor equipment.

[0042] According to a third aspect, a sensor device for detecting non-cooperative obstacles in an airspace adjacent to an aircraft comprises: a radar that radially scans the airspace to provide a location of the non-cooperative obstacle; a first sensor that detects a thermal signature of the non-cooperative obstacle at the location; a second sensor that images the non-cooperative obstacle at the location; and a processor that is operably coupled to each of the radar, the first sensor, and the second sensor, wherein the processor is configured to determine the location of the non-cooperative obstacle using data from the radar and to classify the non-cooperative obstacle at the location using the thermal signature from the first sensor and the image from the second sensor.

[0043] In some aspects, the camera includes a long wave infrared sensor.

[0044] In certain aspects, the camera includes a visible-near infrared electro-optical (EO) sensor.

[0045] In some respects, the radar is a mechanically rotating marine radar.

[0046] Embodiments of the present invention relate to an obstacle detection system for use in an aircraft, the obstacle detection system may include a sensor device and a processor, the sensor device detects a non-cooperative obstacle in a first airspace adjacent to the aircraft, the sensor device includes: a radar that radially scans the first airspace to generate radar information with a first resolution, and a camera that images a second airspace within the first airspace to generate optical information at a second resolution higher than the first resolution; a processor is operably coupled to the sensor device, wherein the processor is configured to determine the position of the non-cooperative obstacle and identify the non-cooperative obstacle based on the radar information and the optical information. The camera may be configured to pan and tilt. The camera includes a long-wave infrared sensor. The camera includes a visible near-infrared electro-optical (EO) sensor. The optical information includes at least one of a thermal cross section and an optical cross section. The radar information may include a radar cross section. The radar information may include a two-dimensional (2D) position of the non-cooperative obstacle in the first airspace, and the optical information may include an azimuth position of the non-cooperative obstacle in the second airspace. The processor may be configured to generate a predicted flight path for the non-cooperative obstacle based at least in part on the radar information and the optical information. The processor may be configured to generate an obstacle avoidance navigation route based at least in part on the predicted flight path to avoid collision with non-cooperative obstacles. The first airspace can provide a 360-degree field of view about the aircraft. The 360-degree field of view can reside in a plane that can be parallel to a flight path defined by the aircraft.

[0047] Another embodiment of the present invention relates to a method for detecting and avoiding non-cooperative obstacles during operation of an aircraft, the method comprising scanning a first airspace using a radar system to generate radar information having a first resolution; imaging a second airspace using a camera to generate optical information at a second resolution that may be higher than the first resolution, wherein the second airspace may be within the first airspace; tracking a non-cooperative obstacle based at least in part on the radar information and the optical information; generating a predicted flight path based at least in part on the radar information and the optical information; generating an obstacle avoidance navigation path to avoid the non-cooperative obstacle; and communicating the obstacle avoidance navigation path to a flight control system of the aircraft. The aircraft may be configured to automatically execute the generated obstacle avoidance navigation path. The camera may include a long-wave infrared sensor configured to pan and tilt. The method may also include determining the azimuth position of a non-cooperative obstacle within the second airspace using the camera. The radar may be a mechanically rotating marine radar. The radar information may include a two-dimensional (2D) radial map of the first airspace. The method may also generate a predicted flight path for a non-cooperative obstacle via a processor based at least in part on data received from a sensor device.

[0048] Another embodiment of the invention relates to a sensor device that detects non-cooperative obstacles in an airspace adjacent to an aircraft, the sensor device comprising a radar that radially scans the airspace to provide a location of the non-cooperative obstacle; a first sensor that detects a thermal signature of the non-cooperative obstacle at the location; a second sensor that images the non-cooperative obstacle at the location; and a processor that is operably coupled to each of the radar, the first sensor, and the second sensor, wherein the processor can be configured to determine the location of the non-cooperative obstacle using data from the radar and to classify the non-cooperative obstacle at the location using the thermal signature from the first sensor and the image from the second sensor. The camera can include a long wave infrared sensor. The camera can include a visible near infrared electro-optical (EO) sensor. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] These and other advantages of the present invention may be readily understood with reference to the following specification and accompanying drawings, in which:

[0050] Figure 1a A block diagram illustrating example aircrew automation.

[0051] Figure 1b Illustration of Figure 1a Example flow of information data between subsystems.

[0052] Figure 1c A block diagram illustrating an example core platform.

[0053] Figure 2 A diagram illustrating an example core platform architecture.

[0054] Figure 3a A first example HMI / HMI is illustrated, illustrating an en-route application.

[0055] Figure 3b A second example human-machine interface is illustrated illustrating a procedure checklist and aircraft health alert screen.

[0056] Figure 3c to Figure 3e A third example human-machine interface is illustrated, illustrating an obstacle detection home screen.

[0057] Figure 4 A block diagram of an example aircraft condition monitoring system having a perception system is illustrated.

[0058] Figure 5a and Figure 5b An example primary actuation system is illustrated.

[0059] Figure 5cAn example secondary actuation system is illustrated.

[0060] Figure 6a Illustration of an example radar and the radar beam it produces.

[0061] Figure 6b Illustration of an example infrared camera and the infrared beam it produces relative to a radar beam.

[0062] Figure 7 An example fixed wing aircraft is illustrated having an equipment pod to house obstacle sensor equipment.

[0063] Figures 8a to 8c A schematic diagram illustrating an example hybrid field of view generated by a radar and infrared camera. DETAILED DESCRIPTION

[0064] Preferred embodiments of the present invention may be described below with reference to the accompanying drawings. In the following description, well-known functions or structures are not described in detail because they may obscure the present invention with unnecessary details. For this disclosure, the following terms and definitions may be applied.

[0065] As used herein, the terms "circuit" and "circuitry" refer to physical electronic components (i.e., hardware) and any software and / or firmware ("code") that can configure, be executed by, and or otherwise be associated with the hardware. As used herein, for example, a particular processor and memory may comprise a first "circuit" when executing a first set of one or more lines of code, and may comprise a second "circuit" when executing a second set of one or more lines of code.

[0066] As used herein, "and / or" refers to any one or more of the items connected by "and / or" in the representation. As an example, "x and / or y" refers to any element in the three-element set {(x), (y), (x, y)}. In other words, "x and / or y" refers to "one or both of x and y". As another example, "x, y and / or z" refers to any element in the seven-element set {(x), (y), (z), (x, y), (x, z), (y, z), (x, y, z)}. In other words, "x, y and / or z" refers to "one or more of x, y and z".

[0067] As used herein, the term "exemplary" refers to an example, instance, or illustration that serves as a non-limiting example. As used herein, the terms "such as" and "for example" set a list of one or more non-limiting examples, instances, or illustrations.

[0068] As used herein, when used to modify or describe a value (or range of values), the words "about" and "approximately" refer to a value or range of values ​​that is reasonably close. Therefore, the embodiments described herein are not limited to the values ​​and ranges of values ​​recited, but should include reasonable and feasible deviations.

[0069] As used herein, a circuit system or device is “operable” to perform a function whenever the circuit system or device includes the necessary hardware and code (if necessary) to perform that function, and regardless of whether the functionality of that function is disabled or not enabled (e.g., by user-configurable settings, factory trim, etc.).

[0070] As used herein, the terms "air vehicle" and "aircraft" refer to machines capable of flight, including but not limited to both conventional runway and vertical take-off and landing ("VTOL") aircraft, and also including manned and unmanned aerial vehicles ("UAVs"). VTOL aircraft may include fixed-wing aircraft (e.g., Harrier jet aircraft), rotary-wing aircraft (e.g., helicopters), and / or tilt-rotor / tilt-wing aircraft.

[0071] As used herein, the terms "communicate" and "in communication with" refer to (1) sending or otherwise conveying data from a source to a destination and / or (2) the communication media, systems, channels, networks, devices, wires, cables, fibers, circuits, and / or links that carry data to a destination.

[0072] As used herein, the term "database" refers to an organized entity of related data, regardless of the manner in which the data or its organized entity is represented. For example, the organized entity of related data may be in the form of one or more of a table, a map, a grid, a package, a datagram, a frame, a file, an email, a message, a document, a report, a list, or data presented in any other form.

[0073] As used herein, the term "processor" refers to processing devices, apparatuses, programs, circuits, components, systems and subsystems, whether implemented in hardware, tangibly embodied software, or both, and whether programmable or not. The term "processor" as used herein includes, but is not limited to, one or more computing devices, hardwired circuits, signal modification devices and systems, devices and machines for controlling systems, central processing units, programmable devices and systems, field programmable gate arrays, application specific integrated circuits, systems on chips, systems containing discrete components and / or circuits, state machines, virtual machines, data processors, processing facilities, and combinations of any of the above. The processor may be, for example, any type of general purpose microprocessor or microcontroller, a digital signal processing (DSP) processor, an application specific integrated circuit (ASIC). The processor may be coupled to or integrated with a memory device.

[0074] As used herein, the term "storage device" refers to computer hardware or circuitry to store information used by a processor. A memory device can be any suitable type of computer memory or any other type of electronic storage medium, such as, for example, read-only memory (ROM), random access memory (RAM), cache memory, compact disk read-only memory (CDROM), optoelectronic memory, magneto-optical memory, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), computer-readable media, etc.

[0075] Disclosed herein is an automatic detection and avoidance system configured to automatically detect and avoid obstacles during operation of an aircraft. The detection and avoidance system can be embodied in nearly any aircraft, including those configured with a pilot, an autopilot, or another automatic flight control system. For example, the automatic detection and avoidance system can be utilized in smaller autonomous aircraft where compliance with TCAS regulations is uneconomical, impossible, or undesirable, or as a replacement for air-to-air radar arrays on larger aircraft to detect and avoid non-cooperative obstacles that would otherwise go undetected.

[0076] The automatic detection and avoidance system will be generally described in conjunction with a crew automation system that is configured to function as a pilot's assistant (or co-pilot) or flight engineer. Commonly owned U.S. Patent Application No. 15 / 464,786, entitled “Aircrew Automation System and Method” and filed on March 21, 2017 by Jessica E. Duda et al. (“the '786 Application”) discloses an example crew automation system.

[0077] As explained in the 786 application, the crew automation system can be configured to operate the aircraft from takeoff to landing, automatically executing necessary flight and flight planning activities, checklists and procedures at the correct flight phase, while detecting and responding to emergencies. At the same time, the pilot (e.g., a human pilot or another operator) can be continuously informed through an intuitive human-machine interface operatively coupled to the crew automation system. That is, the crew automation system can provide real-time information and / or feedback to the pilot. For example, the crew automation system can indicate the status of the aircraft relative to the procedure being completed. The crew automation system can be configured to control the aircraft through robotic actuators, if necessary.

[0078] Unlike existing robotic autopilots and pilot assistance systems (which are intrusive) that require considerable installation expertise and are aircraft-specific, the crew automation system is able to employ a system architecture and knowledge acquisition system that enables rapid non-intrusive installation, which facilitates widespread use and enables the crew automation system to be quickly adapted for use in a variety of aircraft. Further, the data acquisition, perception system, and sensor equipment system of the crew automation system are not limited to GPS, acceleration, orientation, and heading, as is the case with existing robotic autopilots. In fact, the crew automation system exceeds the capabilities of existing data acquisition and perception systems by employing both independent sensors, instrument image data capture (e.g., temperature, altitude, radar, flap angle, etc.) to better capture aircraft performance, and measure, detect, or otherwise receive pilot input. Further, the design of the core platform and primary and secondary flight control actuation systems of the crew automation system makes it possible across a variety of aircraft. Therefore, unlike existing robotic autopilots or pilot assistance systems, the crew automation system can be temporarily installed and easily transferred from aircraft to aircraft without intrusive modifications to the aircraft. The crew automation system, through its modular design, further reduces the likelihood of designing point solutions that become obsolete as the aircraft evolves.

[0079] The combination of the subsystems of the crew automation system provides the pilot with high-fidelity knowledge of the physical state of the aircraft and informs the pilot of any deviations from the desired state based on, for example, predictive models. This state awareness can be directly translated into useful information for the pilot, such as warnings of developing emergency conditions, fuel state calculations, icing condition notifications, warnings of impending collisions with obstacles, notifications or detection of non-cooperative obstacles, etc. For example, the crew automation system can also act as a digital flight engineer, thereby informing the pilot by monitoring checklists, instrumentation, engine status, airspace, flight regime, etc.

[0080] This anterograde crew automation system, which can be non-invasively installed in a pre-existing aircraft, intuitively senses the state of the aircraft and, via other sensors, derives the aircraft state vector and other aircraft information and communicates any deviation from the desired aircraft state to the pilot or air traffic control tower. Although the crew automation system can be installed non-invasively (e.g., via a sensing system), it can alternatively be invasive. For example, the crew automation system can be electronically coupled to a cockpit instrument panel (e.g., via the reverse side of the instrument panel) via, for example, an aircraft state monitoring system. Alternatively, the crew automation system can be integrated and permanently installed during aircraft manufacturing. In conjunction with an actuation system, the crew automation system can further control the aircraft and automatically navigate the aircraft. Therefore, the various systems and subsystems of the crew automation system can be invasive, non-invasive, or a combination thereof.

[0081] System Level Architecture. In order to share the responsibilities and workload associated with performing flight activities, the crew automation system 100 should be able to perform actions that pilots routinely perform during flight, regardless of the make, model, or type of aircraft. Figure 1a to Figure 1c An example system architecture of a flight crew automation system 100 according to one aspect is shown. Figure 1a As illustrated in FIG, the processor-controlled central subsystem functions as a core platform 102 to connect one or more other subsystems via one or more interfaces. The subsystems can communicate with each other through software and / or hardware interfaces 156 using wired and / or wireless communication protocols and hardware. Figure 1b The diagram illustrates an example flow of information (eg, data) between various subsystems.

[0082] Each of the multiple subsystems of the crew automation system 100 may be modular, enabling the entire crew automation system 100 to be rapidly and substantially ported to another aircraft. For example, the various subsystems may be removably and communicatively coupled to each other via the core platform 102 using one or more software and / or hardware interfaces 156. However, in certain aspects, the crew automation system 100 or portions thereof may alternatively be integrated with the aircraft's systems, thereby directly employing sensors and indicators in the aircraft. For example, the crew automation system 100 or components thereof may be integrated into the aircraft during its design and manufacture.

[0083] The plurality of subsystems may include, for example, a perception system 106, an actuation system 108, a human-machine interface (“HMI”) system 104, a flight control system 116, and an obstacle sensor equipment 162, each of which may be operatively coupled to the core platform 102. In certain aspects, the need for the perception system 106 may be mitigated or eliminated via the use of another aircraft status monitoring system. For example, the crew automation system 100 may be coupled (e.g., communicatively or electronically) to an instrument panel, or otherwise integrated with an aircraft or its existing systems. However, as desired, such integration may require some degree of modification to the aircraft or its wiring.

[0084] The crew automation system 100 and / or the core platform 102 may also include or be operably coupled to a knowledge acquisition system 114 and a communication system 122. The modular configuration further enables an operator to remove / disable unnecessary systems or modules or add / install additional systems or modules. For example, when the crew automation system 100 is configured to only provide information to the pilot via the HMI system 104 (i.e., without the ability to control the aircraft), the actuation system 108 may be removed or disabled to reduce weight, cost, and / or power consumption. Accordingly, depending on the configuration, the crew automation system 100 may be configured with fewer or additional modules, components, or systems without departing from the spirit and scope of the present invention.

[0085] In operation, the flight control system 116 derives the state of the aircraft based on information data from another subsystem (e.g., the aircraft state monitoring system 112, the perception system 106, the obstacle sensor equipment 162, etc.), and directs another subsystem (e.g., the actuation system 108, the flight control system 116, the flight control system 116, etc.) to operate in a manner (e.g., dynamically) to maintain the stability of the aircraft. In practice, the flight control system 116 can receive vehicle mode commands and configuration data from the core platform 102 while sending state and command information generated by the flight control system 116 to the core platform 102. For example, the core platform 102 can be configured to communicate one of more commands to the flight control system 116 of the aircraft based at least in part on flight condition data available from the aircraft state monitoring system 112, the perception system 106, the obstacle sensor equipment 162, and / or a combination thereof.

[0086] The flight control system 116 may include or communicate with existing flight control devices or systems, such as those employed in fixed-wing aircraft and rotary-wing aircraft. The communication system 122 enables the crew automation system 100 to communicate with other devices, including remote or telescopic devices, via, for example, a network. The communication system 122 may receive communication commands and configuration data from the core platform 102, while sending status and response information from the communication system 122 to the core platform 102.

[0087] Core platform 102. Figure 2 An architectural diagram illustrating an example core platform 102. To enable a vehicle-agnostic crew automation system 100, the core platform 102 may be a processor-based computer system to provide or otherwise facilitate / serve middleware that can be targeted to a specific aircraft or configuration through an initial transition and setup phase. In other words, the mission control system 110 may provide an operating system 206 that provides services to a set of operating applications 202 and outputs signals to one or more of a set of hardware interfaces 204 or HMI systems 104, while collecting and recording data required by those enabled applications.

[0088] The core platform 102 acts as a primary autonomous agent and decision maker that utilizes its acquired knowledge base to synthesize inputs from the perception system 106, the aircraft state monitoring system 112, the obstacle sensor equipment 162, and the HMI system 104 to determine the overall aircraft system state. The core platform 102 can process the inputs from the various sensor suites and aggregate the resulting information into an understanding of the current aircraft state. The resulting information can be compared to aircraft specific files that contain the crew automation system 100's understanding of the pilot's intent, system health, and understanding of appropriate aircraft procedures as they relate to the crew automation system's 100 state estimate. The resulting state knowledge and related recommendations can be passed to the human pilot via the HMI system 104, or in some aspects, to the flight control system 116 and / or actuation system 108 to achieve autonomous operation. For example, the core platform 102 may communicate information reflecting a detected non-cooperative obstacle relative to the aircraft on a collision path to a flight control system (e.g., the flight control system 116 , which may be an existing flight control system of the aircraft) for initiating an obstacle avoidance navigation path generated by the core platform 102 , where the non-cooperative obstacle on the collision path may be detected by the obstacle sensor equipment 162 .

[0089] The core platform 102 may also include a flight data recorder, for example to provide performance review capabilities and provide robustness to in-flight resets. The crew automation system 100 may further generate a log of a given flight for later analysis, which may be used to facilitate pilot training, which may provide detailed training and operational flight missions. The log may be used in conjunction with, for example, flight operations quality assurance analysis, maintenance analysis, etc.

[0090] As illustrated, the core platform 102 may include a mission control system 110 and a flight controller 118, each of which is configured to communicate with each other and other subsystems via one or more software and / or hardware interfaces 156 (which may be a combination of hardware (e.g., permanent or removable connectors) and software). The core platform 102 is capable of housing / hosting various software processes stored to memory devices that track aircraft, cooperative obstacles, non-cooperative obstacles, and program states, as well as any modules for trend analysis (predictive warnings) and machine learning routines. In some aspects, the crew automation system 100 and / or the core platform 102 may use computer buses and specifications (e.g., as interfaces) to facilitate discovery of hardware components of subsystems within the crew automation system 100 without requiring physical device configuration or user intervention in resolving resource conflicts. Such a configuration may be referred to as "plug and play." Thus, a user may easily add systems or subsystems (eg, modules) to or remove systems or subsystems (eg, modules) from the crew automation system 100 via the core platform 102 without requiring substantial modification or integration work.

[0091] The output of the core platform 102 may be used to provide messages to the HMI system 104. These messages may indicate, for example, checklist progress, initiation of emergencies, issuance of warnings, locations of non-cooperative obstacles, potential obstacle avoidance navigation paths, etc. The hardware and various computers may also be ruggedized and co-housed with other devices, such as a perception computer. As discussed below, the core platform 102 may be operably coupled to a global positioning system ("GPS") / inertial navigation system ("INS") system 154, an obstacle sensor rig 162, and a power management system (e.g., a 28 VDC power supply).

[0092] The mission control system 110 generally includes a mission manager 132, a standard interface 130 (e.g., a STANAG interface), a situation awareness manager 158, and other operating components 120 (e.g., hardware and software controllers and / or interfaces), each of which is communicatively coupled to one another via one or more data buses 124. For example, one of the operating components 120 may be a path manager 160. The path manager 160 may use data from an obstacle sensor rig 162 to track both non-cooperative obstacles and cooperative obstacles to generate a predicted flight path for each of the non-cooperative obstacles and cooperative obstacles. Based at least in part on the predicted flight path, the path manager 160 may generate an obstacle avoidance navigation path for the aircraft to avoid or otherwise mitigate collisions with non-cooperative and / or cooperative obstacles. When generating an obstacle avoidance navigation route, the route manager 160 may also take into account terrain and any aircraft constraints, which may be dictated by the aircraft configuration (e.g., multi-rotor aircraft versus fixed wings). For example, a multi-rotor aircraft may be able to transition from level flight to hovering flight more quickly. Other aircraft constraints may be defined in the operations documentation for a given aircraft.

[0093] The open architecture of the core platform 102 allows for incorporation of additional data received from the system via the data bus 124. In some aspects, the mission control system 110 can be coupled to one or more cockpit instruments of the aircraft via a vehicle system interface to collect flight condition data. In other aspects, the mission control system 110 can collect flight condition data via an aircraft state monitoring system 112 through an aircraft state interface, which can collect or generate flight condition data via a direct connection to the aircraft, the perception system 106, and / or the obstacle sensor equipment 162.

[0094] As illustrated, mission control system 110 may be operably coupled with secondary actuation system 108b (e.g., when autonomous operation is desired), perception system 106, obstacle sensor equipment 162, and HMI system 104, which includes human-machine interface 126 (e.g., software and / or hardware that transmits input from and displays information to the pilot) and ground station 128. Mission control system 110 may communicate with flight controller 118 via mission manager 132.

[0095] The flight controller 118 may include, for example, an autopilot manager 134 and a vehicle manager 136. The vehicle manager 136 may be generally responsible for navigating and determining the position and state of the aircraft. The vehicle manager 136 may be coupled to a state estimation module 142 that uses information received from the perception system 106 via the perception module 138, from the obstacle sensor equipment 162, and from the GPS / INS system 154 via the navigation module 140 to determine an estimated state of the aircraft.

[0096] The autopilot manager 134 may generally be responsible for controlling the flight of the aircraft based on information received, for example, from the vehicle manager 136 and the mission control system 110. The autopilot manager 134 controls, among other things, the flight control system 152, which may be new or pre-existing (and include the flight controller 150), as well as the crew automation actuation module 144 and the aircraft actuation module 146. The crew automation actuation module 144 may control the primary actuation system 108a, while the aircraft actuation module 146 may control the aircraft controller 148 (e.g., various flight surfaces and actuators). The autopilot manager 134 may be configured to receive and implement one or more routes from the route manager 160, including an obstacle avoidance navigation route.

[0097] In certain aspects, components of flight controller 118 may overlap with certain components of flight control system 116. For example, in certain aspects (e.g., where redundancy is not desired and non-intrusive integration is possible), core platform 102 may leverage certain existing aircraft software and / or hardware, thereby avoiding the need for additional hardware, such as certain flight controller 118 components and / or GPS / INS system 154.

[0098] Open Architecture. The core platform 102 serves as a central subsystem or interface for the crew automation system 100 that connects and controls the remaining subsystems (e.g., as separate applications) in an open architecture. The remaining subsystems include, for example, the flight control system 116 (including any flight planning capabilities), the HMI system 104, the actuation system 108 (e.g., primary and secondary actuation systems that provide autonomous operation when desired), the perception system 106, the knowledge acquisition system 114, the obstacle sensor equipment 162, and other subsystems 236. Thus, control of other crew automation system 100 hardware can be provided via separate applications dedicated to specific pieces of hardware, which enables rapid integration of new systems or other external flight planning support technologies.

[0099] The architecture of the core platform 102 allows for rapid portability and extensibility when transitioning to a new aircraft or introducing new flight plan features / capabilities. Thus, applications may be used to enable the crew automation system 100 to obtain information specific to the aircraft or otherwise as needed, or to provide new capabilities. For example, transition and setup can be handled by various applications running within the core platform 102 or other subsystems, representing an ever-growing library of aircraft-specific functions and crew automation system 100 capabilities that can be exchanged based on flight plan, aircraft, or crew requirements. In some aspects, the transition process may be supported by a software application external to the crew automation system 100, such as a program editor.

[0100] Aircraft data structures 208. Operating system 206 operates as middleware, interconnecting operable applications 202, hardware interfaces 204, and other subsystems such as knowledge acquisition system 114. Operating system 206 may employ aircraft data structures 208, which may include knowledge database 210, program database 212, and state database 214.

[0101] The aircraft data structure 208 facilitates the vehicle agnostic crew automation system 100 by enabling the core platform 102 to develop a complete understanding of the aircraft's systems, their configuration, and the procedures necessary to maintain safe operation, as well as all other knowledge and expertise that a certified pilot of that aircraft is expected to possess. The aircraft data structure 208 may be populated by the knowledge acquisition system 114 (discussed below) containing necessary information about the aircraft currently being operated (e.g., flight control models, operating procedures, aircraft systems, etc.), data received from internal state sensors, and other subsystems or sensors (e.g., obstacle sensor equipment 162).

[0102] The aircraft data structure 208 can be populated and tuned to a specific aircraft during a knowledge acquisition phase (e.g., during initial setup) so that it contains all the information necessary to operate the aircraft. For example, when transitioning to a new aircraft, the knowledge acquisition system 114 can perform predetermined activities to determine the aircraft's layout (e.g., controls / readouts, such as the layout of cockpit instruments), performance parameters, and other characteristics. The predefined activities can include, for example: (1) generating an aircraft system model that informs the crew automation system 100 about which systems are on it and their configurations, actuation limits, etc.; (2) program compilation that informs the crew automation system 100 how to operate the aircraft in normal and abnormal conditions, further including the compilation of checklists; (3) aerodynamic models that inform the crew automation system 100 how the aircraft flies and what performance to expect for which aircraft configurations; and (4) information about mission operations.

[0103] The core platform 102 is able to combine this information with data from a set of internal state sensors, which also improves redundancy and system robustness, allowing the crew automation system 100 to generate highly accurate estimates of aircraft state and system state, and identify deviations from expected behavior. During flight operations, the data structure is dynamically updated with real-time data collected by, among other things, the crew automation system 100, the perception system 106, the obstacle sensor equipment 162, the HMI system 104, and the crew automation system 100 internal state sensors. Once the aircraft data structure 208 for a given aircraft is populated, the aircraft data structure 208 can be retained in the aircraft library and used for all other aircraft of the same make and model, and available to the aircraft crew automation system 100 for all other aircraft of the same make and model. As additional data is generated and / or collected by the crew automation system 100, the aircraft data structure 208 can be further refined.

[0104] Operational applications 202. The core platform 102 may provide a plurality of operational applications 202 for the crew automation system 100. Examples of such operational applications 202 may include, but are not limited to, normal flight operations applications 216, anomaly detection applications 218, incident operations applications 220, intelligence, surveillance, and reconnaissance (“ISR”) applications 222 (e.g., ISR tracks), trend identification applications 238, or other flight plan specific activity applications 224, such as the aerial refueling application 316 and / or the sensor equipment monitoring application 336.

[0105] Assuming no unexpected events, the normal flight operations application 216 enables the crew automation system 100 to fly the predetermined flight plan from takeoff to landing. The normal flight operations application 216 is dedicated to the continuous execution of normal flight activities required by a particular flight phase. The predetermined flight plan may be modified in flight due to undesirable interference such as weather, air traffic control orders, air traffic, etc. For example, the predetermined flight plan may be replaced by an obstacle avoidance navigation route or other route based at least in part on sensor data from the obstacle sensor equipment 162 from the route manager 160.

[0106] The anomaly detection application 218 employs machine learning techniques to monitor the aircraft state, cluster and classify sensor inputs in order to detect the presence of abnormal conditions and identify whether an emergency has occurred. The anomaly detection application 218 is configured to compare the sensed state to a set of thresholds defined in the operational documentation for the particular aircraft (e.g., never exceeding a predetermined airspeed, engine temperature, etc.). The anomaly detection application 218 may also compare the sensed state to additional information available to the crew automation system 100, such as information received from the obstacle sensor equipment 162, and generate an alert or other message in response to meeting a predetermined or dynamically determined threshold (e.g., a warning threshold, etc.).

[0107] In the case of an emergency situation, the emergency operation application 220 executes the necessary predetermined checklists, procedures and actions specified by the emergency operation application 220 to maintain the safe operation of the aircraft or safely transfer the flight. It is worth noting that if a deviation from the expected performance is observed, the pilot can be warned of an abnormal state, thereby mitigating or avoiding potential errors. If the aircraft is susceptible to specific operating errors (e.g., pilot-induced oscillations), the crew automation system 100 can identify and mitigate such events. If an anomaly is detected, the emergency operation application 220 notifies the pilot via the HMI system 104 and interacts with the pilot, and ultimately executes the necessary (one or more) procedures to respond to the anomaly. Finally, the ISR application 222 and other flight plan-specific activity applications 224 can provide instructions, algorithms or information to perform mission-related operations.

[0108] The trend identification application 238 provides trend analysis developed using machine learning based on, for example, the knowledge acquisition system 114. In some aspects, the trend identification application 238 may provide data or otherwise trigger the anomaly detection application 218. For example, if the trend identification application 238 detects an undesirable trend, the trend may be marked as an anomaly and reported to the anomaly detection application 218 (e.g., when a non-cooperative obstacle is detected on a collision course with the aircraft).

[0109] Hardware interfaces 204. Various information regarding operable applications 202 is communicated between core platform 102, primary actuation system 108a, secondary actuation system 108b, perception system 106, obstacle sensor equipment 162, aircraft status monitoring system 112, HMI system 104, and other subsystems 236 via, for example, primary actuation interface 226, secondary actuation interface 228, aircraft status interface 230, HMI interface 232, and other interfaces 234.

[0110] Human / machine interface (HMI) system 104. HMI system 104 provides a control and communication interface for a pilot (e.g., a human pilot, whether onboard or remote). HMI system 104 may be configured to operate as a flight plan manager, enabling the pilot to direct the crew automation system 100. HMI system 104 can combine elements of a glass cockpit, an unmanned aerial vehicle (UAV) ground station, and an electronic flight bag (EFB) to enable effective, efficient, and delay-tolerant communication between the pilot and the crew automation system 100. Generally, an EFB is an electronic information management device that allows the crew to perform various functions that are traditionally accomplished by using paper references. HMI system 104 may include a human-machine interface 126, which may be based on a touch screen graphical user interface ("GUI") and / or a voice recognition system. Human-machine interface 126 may be implemented, for example, as a tablet computer, laptop computer, smart phone, or a combination thereof. Depending on pilot preference, human-machine interface 126 can be fixed near the pilot (e.g., on a joystick—because checklists are often on a joystick, or on a knee strap). The human machine interface 126 may be removably coupled to the cockpit, or in some aspects, employ an integrated display (eg, an existing display) within the cockpit.

[0111] Figure 3a An example human machine interface 126 having a single screen touch interface and a voice recognition system is illustrated. The HMI system 104 serves as a primary channel for communication between the pilot and the crew automation system 100, enabling the pilot to command tasks and receive feedback or instructions from the crew automation system 100 to change the task allocation between the pilot and the crew automation system 100 and select which operational applications 202 are currently activated for the crew automation system 100. Figure 1bAs illustrated in FIG. 1 , for example, the HMI system 104 can receive status information from the subsystems of the crew automation system 100 via the core platform 102, while sending one or more mode commands generated by the HMI system 104 or input by the pilot to the core platform 102. The pilot can be remote (e.g., on the ground or on another aircraft) or onboard (i.e., in the aircraft). Thus, in some aspects, the HMI system 104 can be remotely assisted over a network via the communication system 122.

[0112] Human-machine interface 126. Figures 3a to 3e As illustrated in FIG, the human-machine interface 126 may employ a tablet-based GUI and a voice recognition interface to enable voice communications. The goal of the human-machine interface 126 is to enable the pilot to interact with the core platform's 102 knowledge base in a manner similar to the way the pilot interacts with a human flight engineer or co-pilot.

[0113] The human-machine interface 126 can display the current state of the crew automation system 100 (its current settings and responsibilities) as well as which operational applications 202 are currently installed, which operational applications are running, and if they are active, what actions the operational applications 202 are performing. The GUI display of the human-machine interface 126 can also be night vision goggle compatible so that it is visible regardless of the pilot's glasses. The speech recognition system can be used to replicate the same type of verbal communication used by human crew members when running through checklists and communicating on the flight deck. In some aspects, speech recognition may be limited to the same standards of coded communication used by the pilot team to minimize the possibility that the system cannot recognize commands or change to an inappropriate operating mode. The speech recognition system can be configured to learn / recognize the voice of a given pilot through a sound training protocol. For example, a pilot can speak a predetermined script so that the speech recognition system can be trained through the pilot's dialect.

[0114] The human-machine interface 126 may provide details of the status and / or various operations, including: the entire crew automation system 100 via the crew automation status application 302, the perception system 106 via the perception status application 304, the obstacle sensor equipment 162 via the sensor equipment monitoring application 336, the autopilot (where applicable) via the autopilot status application 306, the GPS / INS system 154 via the GPS status application 308, and any other application or system status information 310. The display of the human-machine interface 126 may be customized by the pilot. For example, the pilot may wish to add, reorganize, or delete certain display icons and / or operational applications 202, which may be accomplished by select and drag manipulation or through the crew automation settings application 312. The human-machine interface 126 may further inform the pilot about the operating status of the aircraft and provide instructions or suggestions to the pilot.

[0115] As illustrated, the human-machine interface 126 may provide a toolbar with various selectable tabs, such as a route tab 328, a procedure tab 330, a calibration tab 332, and an application tab 334. For example, when the pilot selects the application tab 334, the human-machine interface 126 may display various operable applications 202 installed on the crew automation system 100 (e.g., the core platform 102), including, for example, the normal flight operations application 216, the continuous operations application 220, the crew automation setup application 312, the gauge application 314, the sensor equipment monitoring application 336, and the aerial refueling application 316. However, additional mission applications may be included to facilitate the execution of desired mission operations by the crew automation system 100.

[0116] Selecting the crew automation settings application 312 enables the pilot to change, reassign, or otherwise edit the settings of the crew automation system 100 and / or install the operational applications 202. Selecting the gauges application 314 causes the human-machine interface 126 to display various operating conditions of the aircraft, including, for example, position, direction, speed, altitude, pitch, yaw, etc. Various operating conditions of the aircraft, which may be collected from the perception system 106, the obstacle sensor equipment 162, or another sensor, may be displayed as alphanumeric characters or graphical dials (e.g., according to the pilot's preference settings). Selecting the air refueling application 316 icon causes the crew automation system 100 to execute a predetermined protocol for facilitating or coordinating an air refueling operation. For example, when the air refueling application 316 is selected, the crew automation system may coordinate with another aircraft to facilitate refueling and execute the necessary checklists for the same operation (e.g., ensuring aircraft position, airspeed, fuel hatch opening, etc.).

[0117] When the pilot selects the route tab 328, the human-machine interface 126 may display a regional map 326 with an icon 322 that represents the current position of the aircraft relative to its various waypoints 320 along the flight path. Selecting (e.g., tapping, clicking, etc.) the icon 322 causes a dialog window 324 on the display to provide various operating conditions of the aircraft (e.g., identification, altitude, speed, heading, etc.). The regional map 326 may be saved, exported, rotated, or translated using the map control window 318. The regional map 326 may be saved or exported (e.g., via the communication system 122) as a static image or a data set (or database). When the pilot selects the calibration tab 332, the human-machine interface 126 may display the calibration of the aircraft, which may further enable the pilot to modify the calibration. The regional map 326 may be configured to display one or more routes 352 generated by the route manager 160, including the original route 352a and, where applicable, the obstacle avoidance navigation route 352b (or other deviation) to avoid obstacles 344.

[0118] HMI system 104 can provide an intuitive display and interface, including checklist verification and health alerts from core platform 102 and predictions of aircraft status (e.g., fuel consumption and estimated remaining range), as well as failure predictions and yaw alerts (e.g., "left engine EGT is 5 degrees above normal and rising" and "imminent collision with non-cooperative detected"). Thus, when the pilot selects procedure tab 330, such as Figure 3b, the pilot can review and monitor the checklist items, as well as review any health alerts. In fact, the function of the HMI system 104 is to facilitate checklist monitoring and / or execution, marking items as completed when the sensing system 106 senses that the items are completed, and providing warnings to the pilot when the items are not completed, such as based on information previously imported from, for example, the pilot operating handbook ("POH"). The crew automation system 100 also monitors system health, compares the current system state to the system expected based on the POH and other knowledge sources, and guides appropriate responses to unexpected events. In some aspects, as the checklist actions are executed and the HMI system 104 automatically proceeds to the correct checklist as appropriate, the pilot or core platform 102 can confirm the checklist actions. The HMI system 104 can give visual and auditory alerts to draw the pilot's attention to unnoticed checklist items, instruments that display values ​​outside of normal ranges, or events predicted as the aircraft proceeds through the flight plan, which can be entered as a series of waypoints (for example). For example, as illustrated, a task list may be provided next to an indicator indicating whether the task has been completed, is being completed, or needs to be completed (e.g., including a completed "check (√)" icon, an "in progress" icon, and a "to be completed" icon). Similarly, a list of health hazards may be provided, along with one or corresponding icons to indicate that one or more operating conditions are out of range. For example, if the fuel is low, a low fuel indicator may be provided next to the low fuel icon.

[0119] Selecting the sensor equipment monitoring application 336 icon causes the human machine interface 126 to display an obstacle detection home screen 338, such as in Figure 3c 3. An example obstacle detection home screen 338 is illustrated in FIG. The obstacle detection home screen 338 may display a number of status / alert windows including, among other things, an available sensor status window 338a, a detect / avoid alert window 338b, and an obstacle location window 338c.

[0120] The available sensor status window 338a may display, among other things, a list of the various available sensors coupled to the obstacle sensor equipment 162. The status (e.g., operational, faulty, service due, calibration required, etc.) may be provided adjacent to its corresponding sensor name and / or icon. The detect / avoid alert window 338b may display one or more alerts based at least in part on data from the obstacle sensor equipment 162. The alerts may provide information about any obstacles (e.g., cooperative and non-cooperative obstacles) within the field of view of the aircraft, including the presence and location of any detected obstacles. The alerts may be automatically organized according to their relative importance to one another. To this end, each alert type may be assigned a weight (or rank) so that the alerts are listed and categorized based on a hierarchy. For example, an imminent threat (e.g., an imminent collision threat—such as an obstacle within a predetermined distance 342a) may be listed first, followed by an intermediate threat (e.g., an obstacle beyond the predetermined distance 342a, but within a second predetermined distance 342b), and finally a general status alert (e.g., an obstacle within the field of view of the aircraft). In some aspects, the alerts may be color coded such that an imminent threat is a first color (e.g., red, possibly accompanied by an audible sound), an intermediate threat is a second color (e.g., yellow), and a general status alert is a third color (e.g., green, black, etc.). The obstacle location window 338c may provide a graphical representation 340 of the alert provided by the detect / avoid alert window 338b (as well as other information). The operator may select the graphical representation 340 via the human-machine interface 126 to enlarge the obstacle location window 338c and / or the graphical representation 340, an example of which is shown in FIG. Figure 3d The diagram in the figure shows the following figure.

[0121] refer to Figure 3d, the obstacle location window 338c may be enlarged to display the graphical representation 340 in detail. For example, the graphical representation 340 may illustrate all obstacles 344 detected (e.g., by the obstacle sensor equipment 162) within the field of view 342c of the aircraft 346. The obstacles 344 may be, for example, cooperative obstacles (e.g., cooperative aircraft) or non-cooperative obstacles (e.g., non-cooperative aircraft and other flying objects, such as birds). Based at least on the proximity of the obstacle to the aircraft 346 (in terms of vertical / altitude and horizontal distance), each of the obstacles 344 within the field of view 342c may be shown as one of an upcoming threat obstacle 344a, an intermediate threat obstacle 344b, and / or a non-threat obstacle 344c. For example, if obstacle 344 is within predetermined distance 342a of aircraft 346, an impending threat obstacle 344a may be identified, while if the obstacle is beyond predetermined distance 342a of aircraft 346 but within a second predetermined distance 342b, an intermediate threat obstacle 344b may be identified. Obstacles 342 that are within field of view 342c but not within second predetermined distance 342b (or at a sufficiently different altitude from aircraft 346) may be identified as non-threat obstacles 344c. Figure 3d The graphical representation 340 of FIG. 3 is generally illustrated in terms of proximity in two dimensions (e.g., an XY plane defined by an X-axis and a Y-axis), but the graphical representation 340 may be illustrated similar to a three-dimensional depiction (X-axis, Y-axis, and Z-axis) to better illustrate detected obstacles 344 approaching aircraft 346, an example of which is shown in FIG. Figure 3e 3D view, the operator may select (e.g., click to toggle) 2D / 3D icon 348. When in a three-dimensional view, the operator may further translate and / or rotate graphical representation 340 using pan / rotate icon 350, or simply engage graphical representation 340 by tapping and dragging to move until the operator has achieved the desired view. In certain aspects, any blind spots (i.e., airspace not monitored by the sensor) may be illustrated as shaded to indicate to the operator that potential obstacles may exist in the airspace of the shaded area.

[0122] In addition to proximity, the graphical representation 340 may provide various conditions of the obstacles 344. For example, for each detected obstacle 344, the graphical representation 340 may display, among other things, identification (e.g., tail number, if known), altitude, speed, heading, status (e.g., cooperative or non-cooperative), etc. The conditions may also be used as a factor in determining the threat status of the aircraft 346 for a given obstacle 344. For example, the core platform 102 may determine (based at least in part on data from the obstacle sensor equipment 162) that an obstacle 344 (which should be a non-threat obstacle 344c based on the current position) may become an impending threat obstacle 344a (or an intermediate threat obstacle 344b) based on the speed and / or heading of the non-threat obstacle 344c over a predetermined time period (e.g., a short time period, such as 1 to 10 minutes, or about 1 minute). In this situation, core platform 102 may appropriately escalate obstacle 344 from non-threat obstacle 344 c to impending threat obstacle 344 a (or intermediate threat obstacle 344 b ) regardless of the proximity of the current position of obstacle 344 to aircraft 346 .

[0123] Task Assignment. The HMI system 104 can enable the pilot to limit the activities (if any) performed by the crew automation system 100. The HMI system 104 can define the task assignment between the pilot and the crew automation system 100, their responsibilities and the information communication between the two, thereby acting as a partner to the pilot. Therefore, the crew automation system 100 can operate in a purely advisory role (i.e., without any control of the aircraft), a fully autonomous role (i.e., controlling the flight controls without pilot intervention), or an advisory role with the ability to control the flight controls, depending on the configuration. The HMI system 104 can further be designed to enable the pilot to enter a transition phase in which the pilot specifies various aspects of the flight operation for which the crew automation system is responsible. For example, the HMI system 104 can display a task list in which the pilot can select whether the crew automation system 100 or the pilot is responsible for a given task on the list. The task list can be provided to the HMI system 104 from a program editor described below. Once the aircraft data structure 208 has been populated and refined so that the pilot has greater trust in the crew automation system 100, the pilot may allow the crew automation system 100 to perform additional actions, transitioning the pilot from the primary mode to the supervisory mode (i.e., a fully autonomous role). In this supervisory mode, pilot interaction may be at a high, goal-based level, with the HMI system 104 supporting these tasks as well as allowing the operator to gain insight into other levels of troubleshooting. As described above, in some aspects, all tasks may be performed by the pilot, leaving the crew automation system 100 to serve an advisory role.

[0124] Mode awareness. A risk when employing any automation system is that mode confusion may occur to the pilot (e.g., the pilot ignores a task, thinking that the automation system will handle it). The HMI system 104 avoids this mode confusion by first generating the correct functionality and the above-mentioned distribution of tasks between the crew automation system 100 and the pilot. In fact, the HMI system 104 allows the pilot to directly command and configure the crew automation system 100 via the human-machine interface 126, and displays the information necessary for the pilot to understand what actions the crew automation system 100 is taking to ensure mode awareness. In other words, mode awareness generally refers to the state in which the mode of the system matches the operational mode desired by the operator. The human-machine interface 126 can display the necessary information to ensure that the pilot is always aware of the mode in which the crew automation system 100 is operating. Additionally, the HMI system 104 serves as a human-machine interface for various task applications (e.g., the operational applications 202).

[0125] Aircraft state monitoring system 112. Aircraft state monitoring system 112 collects, determines, or otherwise senses real-time aircraft state. As described above, aircraft state monitoring system 112 may sense real-time aircraft state through, among other things, direct connection to the aircraft and / or perception system 106 (e.g., integrated with the aircraft or otherwise hardwired). Aircraft state monitoring system 112 may further be coupled to obstacle sensor equipment 162, either directly or via core platform 102, to obtain information reflecting any obstacles 344 in the airspace proximate to the aircraft.

[0126] When using perception system 106, aircraft status monitoring system 112 may include a dedicated controller (e.g., a processor) or a controller 402 that shares perception system 106. For example, perception system 106 may employ a combination of visual systems, acoustic systems, and recognition algorithms to read or understand flight condition information displayed by cockpit instruments. Example cockpit instruments include, for example, an altimeter, an airspeed indicator, a vertical speed indicator, one or more compass systems (e.g., a magnetic compass), one or more gyroscopic systems (e.g., an attitude indicator, a heading indicator, a turn indicator), one or more flight indicator systems, one or more navigation systems (e.g., a very high frequency omnidirectional range system (“VOR”), a non-directional radio beacon (“NDB”), etc. The perception system 106 may include a processor and one or more optical sensors (e.g., three or more lightweight machine vision cameras) trained on the instrument panel to maximize pixel density, glare robustness, and redundancy. The one or more optical sensors may be connected to the perception computer via a hardwired connection (such as Ethernet). The one or more optical sensors should be mounted to have a line of sight to the instrument panel to reduce pilot obstruction.

[0127] The flight condition data sensed by the perception system 106 and / or the aircraft status monitoring system 112 may be encoded and provided to the core platform 102 in real time. The open architecture of the core platform 102 allows for the incorporation of additional data received via the data bus 124 to enhance the flight condition data generated by the perception system 106 or the obstacle sensor equipment 162. Figure 1b As illustrated in the figure, for example, the aircraft state monitoring system 112 and / or the perception system 106 can receive commands and configuration data from the core platform 102 and send to the core platform 102 status and flight condition information (e.g., flight condition data) collected by the perception system 106 or flight condition information otherwise collected by the aircraft state monitoring system 112.

[0128] Figure 4The diagram illustrates an example perception system 106 that is operably coupled, inter alia, to the core platform 102 (which is coupled to other subsystems, such as the flight control system 116), the GPS / INS system 154, and the obstacle sensor equipment 162. The perception system 106 visually and / or acoustically monitors, inter alia, cockpit instruments to generate flight condition data that can be used to derive aircraft status from a cockpit layout that can range from basic analog aircraft instruments to a highly integrated glass cockpit avionics suite. In addition to deriving physical status information (such as airspeed and altitude), the perception system 106 can also monitor instruments for aircraft systems, such as fuel gauges and radios, and provide secondary feedback on the status and positioning of the actuation system 108.

[0129] As illustrated, the perception system 106 may include a perception controller 402 operatively coupled to a database 404 and a plurality of sensors such as a camera 410 (for a visual system), a microphone 408 (for an acoustic system), and / or other sensors 406 (e.g., temperature sensors, position sensors, inertial sensors, etc.). For example, the perception controller 402 may be a processor configured to feed back flight condition data (or otherwise indicate) to the core platform 102 based on receiving and manipulating information received from the plurality of sensors, the database 404, and external components such as the GPS / INS system 154 and the obstacle sensor equipment 162.

[0130] Vision System. The perception system 106 may employ a monocular or stereoscopic vision system, possibly including motion capture markers, to continuously monitor the status of the aircraft by reading what is displayed on the cockpit instruments. In some aspects, by comparing information about the scene from two vantage points, 3D information can be extracted by examining the relative positions of obstacles in the two panels. The vision system can be used to accurately monitor instruments (e.g., glass gauges, physical steam gauges, etc.) and switches, and their locations in various lighting conditions and cockpit layouts and sizes. The use of stereoscopic vision systems and / or markers also provides sensing to prevent collisions between any robotic components and the pilot.

[0131] The vision system may employ a set of high definition, stereo cameras and / or LIDAR laser scanners. The system is capable of recognizing data from all flight instruments and deriving the status of switch knobs and gauges that display the status of specific aircraft systems (e.g., remaining fuel). It is also capable of recognizing the status of panels with sufficient resolution to detect small changes due to pilot actions. Machine vision algorithms on the computers of the perception system 106 "read" instruments (gauges, lights, wind correction angle panels, primary flight displays or individual elements of a multifunction display in a glass cockpit) and mechanical items (such as throttles, trim settings, switches, and circuit breakers), thus providing real-time cockpit status updates to the core platform 102.

[0132] The perception system 106 is capable of deriving aircraft status from cockpit layouts ranging from basic simulated aircraft instruments to highly integrated "glass cockpit" avionics suites. Through the visual system, the requirement for data feeds from the aircraft is avoided, thereby allowing / increasing portability across aircraft. However, where possible, the crew automation system 100 can also be coupled to the data feed of the aircraft (e.g., through a data port). In addition, using the application method described for the core platform 102, different underlying operational applications 202 can be used to address and understand different cockpit layouts. For example, the crew automation system 100 can use a gauge application 314 to derive the value displayed on the instrument, whether it is a graphical dial (e.g., an analog "steam" gauge or its digital representation) or a glass cockpit. The method will also enable the crew automation system 100 to run an operational application that monitors, in particular, weather radar, traffic displays, and terrain maps displayed in the cockpit.

[0133] In order to make the flight crew automation system 100 portable, the process of quickly learning new cockpit layouts and encoding the location and scaling of instruments or subtle differences in units is addressed by the perception system 106 design. For example, during the initial knowledge acquisition phase, the location and scaling of instruments and switches can be coded and verified for a specific aircraft, reducing the real-time task of extracting the location or number of graphic dials (circular dials) (glass cockpit), whether it is a graphic dial gauge, whether it is a CRT display or LCD, etc. The piece-wise planar structure of the cockpit instruments enables the perception system 106 to analyze the image (e.g., using homography methods) and record it on the pre-mapped data generated during the initial knowledge acquisition phase. Accordingly, live images can be recorded and compared with previously annotated models, greatly simplifying the interpretation of the data.

[0134] Actuation system 108. When necessary, actuation system 108 performs actions commanded via core platform 102 to direct the flight and overall operation of the aircraft. Actuation system 108 of crew automation system 100 performs actions commanded by core platform 102 to direct the flight and overall operation of the aircraft without interfering with activities performed by the pilot. For example, Figure 1b As illustrated in FIG. 1 , the actuation system 108 may receive actuation commands and configuration data from the core platform 102 , while sending to the core platform 102 status and response information generated by the actuation system 108 .

[0135] Manned aircraft cockpits are designed for a human reach envelope, and therefore, all cockpit controls are reachable by comparably sized robotic / mechanical manipulators. However, manipulators capable of actuating every single switch, knob, stem, and button on every single possible cockpit with the rapid execution required for emergency operations in high gravity (G) and vibration environments would be expensive, heavy, and more intrusive than desired for the crew automation system 100.

[0136] To more effectively achieve portability across aircraft, the crew automation system 100 may separate the actuation of primary flight controls (stick / yoke, joystick, side-stick or collective, rudder pedals, brakes, and throttles) from the actuation of secondary flight controls (e.g., switches, knobs, rockers, fuses, etc.). This approach reduces the likelihood of designing a single point solution that becomes obsolete as the aircraft evolves. Therefore, the crew automation system 100 may employ a primary actuation system 108a and a secondary actuation system 108b to physically control actuators in the cockpit. More specifically, the primary actuation system 108a may actuate the primary flight controls, while the secondary actuation system 108b may actuate the secondary flight controls without interfering with the pilot's use of these controls. The primary actuation system 108a and the secondary actuation system 108b are configured to jointly actuate all standard controls present on today's flight decks during flight operations.

[0137] As discussed below, the primary actuation system 108a focuses on actuating primary flight controls (cog / stick, joystick, sidestick or collective pitch stick, rudder pedals, brakes, and throttles), while the secondary actuation system 108b focuses on actuating controls that are not easily accessible to the primary actuation system 108a, such as secondary flight controls (e.g., switches, knobs, rockers, fuses, etc.).

[0138] Primary actuation system 108a. The primary actuation system 108a focuses on a set of controls necessary for safe operation of the aircraft. Figure 5a and Figure 5b As shown in FIG. 1 , the primary actuation system 108 a includes a frame 516 with an articulated arm 502 (e.g., a robotic appendage or “arm”) and a stick / coupling actuator 510 that actuates the primary flight controls (joystick, control column, sidestick or collective stick, rudder pedals, brakes, and throttles) and other easily accessible controls. The actuator may be one or more of a linear (straight line) actuator, a rotary (circular) actuator, or an oscillating actuator, which may be driven by one or more of electrical, pneumatic, and / or hydraulic techniques.

[0139] The frame 516 may be sized and shaped to fit within a standard aircraft seat. To this end, the footprint of the frame 516 should be the same size or smaller than the average human "sitting" footprint. The actuation system 108 may be manufactured using lightweight metals, metal alloys, and / or composite materials.

[0140] A joystick actuator 510. The joystick actuator 510 can be coupled and engaged with an existing joystick 514 of an aircraft using a joystick holder 512. The joystick holder 512 can be sized and shaped so that it is universal and can engage various forms of joysticks and / or control wheels. The joystick actuator 510 can be configured to move the joystick 514 forward / backward, left, right, and to positions therebetween. The joystick holder 512 can further include one or more actuators for actuating buttons and / or switches located on the joystick 514.

[0141] Articulated arm 502. The actuator-controlled articulated arm 502 may be sized, shaped, and configured to occupy the space typically occupied by the co-pilot's arm, thereby ensuring portability across aircraft. To enable movement with multiple degrees of freedom ("DOF"), the articulated arm 502 may include multiple arm segments (whether linear, curved, or angled) joined using multiple hinged or pivot joints 506. The articulated arm 502 may include a clamp 504 at its distal end. The clamp 504 may be coupled to the articulated arm 502 via a multi-DOF connection. The base of the articulated arm 502 may be rotatably and slidably coupled to the frame 516 via a movable base 508. For example, the articulated arm 502 may be coupled to an upper base 508a, which is slidably coupled to a lower base 508b, which may be fixed to the frame 516. The upper base 508a can slide relative to the lower base 508b using, for example, a combination of rails and ball bearings. In some aspects, the upper base 508a can slide relative to the lower base 508b along the X-axis and the Y-axis.

[0142] The articulated arm 502 can be equipped with encoders (e.g., dual 18-bit encoders) for each of its degrees of freedom to ensure accurate positioning of the articulated arm 502. An internal clutch can be provided at each articulated or pivot joint 506 so that the articulated arm 502 can be overpowered by the pilot when necessary without damaging the articulated arm 502. In this case, the crew automation system 100 can determine the position or location of the articulated arm 502 using the encoders.

[0143] The gripper 504 may be configured to couple or otherwise engage, for example, a throttle lever, etc. The gripper 504 may also provide force and pressure detection, allowing the crew automation system 100 to estimate how to grasp the flight control actuator and adjust the motion to throw it correctly. Once the motion is executed, the same feedback may be used to determine if the desired switch configuration has been achieved. In some aspects, the articulated arm 502 may be equipped with electronics (e.g., homing equipment) that enable it to find and hit obstacles.

[0144] Secondary actuation system 108b. Unlike primary flight controls (which are typically located in the same vicinity across makes and types of aircraft), secondary flight controls (e.g., avionics, switches, knobs, rockers, toggles, covered switches, fuses, etc.) are not consistently located or spatially contained from aircraft to aircraft.

[0145] The secondary actuation system 108b focuses on actuating controls that are not easily accessible by the primary actuation system 108a. For example, some switches may even be located directly above the captain's head on the top panel, making it difficult to manipulate them with a robotic arm (especially in turbulent flight conditions). Therefore, some actuators may be assigned to the primary actuation system 108a described above, while other actuators may be assigned to a self-contained secondary actuation system 108b.

[0146] The secondary actuation system 108b may be provided in the form of an adaptable XY-plotter or gantry system that is mounted directly on the panel of interest and calibrated to the specific panel being operated on. The secondary actuation system 108b is preferably universal and scalable. Figure 5c An example XY-plotter is illustrated. The XY-plotter may include a square frame with rails 520 that serve as the plotter, a rotatable multi-tool 528 with multiple interfaces (e.g., switch actuators 532 and knob actuators 530) that can manipulate controls of interest, and a control system that moves the multi-tool 526 within the frame along the set of Y-axis rails 522 and X-axis rails 524.

[0147] In use, the plotter moves the multi-tool 528 into position, selects the correct manipulator interface, and manipulates the secondary flight control of interest. For example, the multi-tool 528 can flip a binary switch and / or an override switch using the switch actuator 532, and can twist a knob using the knob actuator 530. The switch actuator 532 and / or the knob actuator 530 can be coupled to the multi-tool 528 via an articulated or rotating member, such as a rotatable switch arm 534.

[0148] When not in use, the multi-tool 526 can be returned to its original position (e.g., automatically navigated to a remote corner) to prevent obstruction of the panel. The multi-tool 526 will be equipped with sensors (e.g., proximity sensors) so that it can get out of the way when it detects the pilot's hand. During the initial setup of the plotter on a new aircraft, the location, type, and positioning of the secondary flight control panel can be encoded. Once a particular secondary flight control panel is encoded, the configuration can be saved to the aircraft data structure 208 and loaded when the crew automation system 100 is installed on the same aircraft or the same type of aircraft. In some aspects, additional actuators can be provided to actuation controllers located in, for example, the footwell of the cockpit, such as foot pedals (e.g., brake and / or rudder pedals).

[0149] Obstacle sensor equipment 162. Obstacle sensor equipment 162 may employ a plurality of sensors to identify and monitor obstacles external to the aircraft. In one embodiment, obstacle sensor equipment 162 may be coupled to core platform 102 directly, or via another system, such as aircraft status monitoring system 112, flight control system 116, or an existing aircraft system. Obstacle sensor equipment 162 or components thereof may be further configured to communicate wirelessly with other systems of the aircraft or crew automation system 100. Figure 1b As illustrated in , for example, obstacle sensor equipment 162 may receive commands and configuration data from core platform 102 while sending obstacle information about any cooperating and non-cooperating obstacles collected by obstacle sensor equipment 162 to core platform 102 .

[0150] As described above, the obstacle sensor equipment 162 can be operably connected to the core platform 102 to enable the route manager 160 of the crew automation system 100 to generate an obstacle avoidance navigation route to avoid one or more obstacles detected by the obstacle sensor equipment 162, for example, based at least in part on the obstacle information. In addition, the obstacle sensor equipment 162 can transmit the collected obstacle information to the aircraft status monitoring system 112 to warn the operator (e.g., via the human-machine interface 126) of possible collisions, obstacle locations, or other parameters thereof. In one aspect, the obstacle sensor equipment 162 can simultaneously employ a radar sensor 412 and a camera (e.g., an infrared camera 414—a camera with an infrared sensor, a visible near infrared EO sensor 416, or other optical sensor 418) to monitor the airspace adjacent to the aircraft to detect cooperative and non-cooperative obstacles within its field of view along its trajectory, etc.

[0151] The obstacle sensor equipment 162 integrates multiple sensing modalities into a single package for tracking both cooperative and non-cooperative targets, implements the latest avoidance algorithms, and defines an open architecture so that future sensing modalities or avoidance algorithms can be easily integrated. The obstacle sensor equipment 162 utilizes multiple sensing modalities across the electromagnetic spectrum to determine information related to non-cooperative targets in the airspace. The obstacle sensor equipment 162 employs multiple sensors for non-cooperative sensing, including a radar sensor 412, a long-wave infrared (thermal) sensor 414, and a visible near-infrared electro-optical (EO) sensor 416. The radar sensor 412 scans the airspace for objects (e.g., metal objects), the long-wave infrared (thermal) sensor 414 scans the airspace for thermal signatures, and the visible near-infrared electro-optical (EO) sensor 416 scans the airspace to identify and help classify objects.

[0152] The obstacle sensor equipment 162 can assist the pilot as another set of "eyes and ears" for external perception applicable to all forms of aircraft. In one embodiment, for example, the obstacle sensor equipment 162 provides additional situational awareness to the physical pilot (whether in the air with the aircraft or on the ground). In another embodiment, the obstacle sensor equipment extends beyond the input of the physical pilot and can be directly integrated with the fly-by-wire system, allowing it to take over control of the aircraft during a loss of link scenario. Therefore, while the obstacle sensor equipment 162 is primarily discussed in conjunction with the crew automation system 100, the obstacle sensor equipment 162 can be provided as an add-on system to nearly any aircraft (e.g., a Group 2-3 unmanned aircraft system) to perform detection and avoidance, thereby enabling the aircraft to be operated within the national airspace.

[0153] Radar sensor 412 . Figure 6aAn example radar sensor 412 and the radar beam 600a it generates are illustrated. The radar sensor 412 can be mechanically rotated to provide a 360 degree field of view. The radar sensor 412 can be, for example, an active electronically scanned array, a passive electronically scanned array, a metamaterial electronically scanned array radar, a weather radar, or a marine radar. To facilitate use with small aircraft, the radar sensor 412 is preferably compact, lightweight, and low cost. Suitable radar sensors 412 include marine radars, such as the Furuno DRS4D-NXT Solid-state Doppler Radar, which is 16 pounds and approximately 24" (L) x 24" (W) x 9" (H). The Furuno DRS4D-NXT radar provides a field of view of 2° (V) x 25° (H), adjustable antenna rotation speeds of 24, 36, and 48 RPM, and a range of up to 36 miles.

[0154] Radar sensor 412 may be configured to monitor the airspace near the aircraft (e.g., a circular airspace around aircraft 700) for cooperative and non-cooperative obstacles. As illustrated, when in a fixed position, radar sensor 412 provides a relatively narrow horizontal beamwidth (W) at Y°. Horz. ), but provides a wide vertical beamwidth (W Vert .), resulting in a radial cone of airspace monitored by the radar. Specifically, depending on the specifications of the marine radar, the radar sensor 412 can provide a 2° (H) (e.g., Y°=360° divided into 2°) × 25° (V) (e.g., Z°=360° divided into 25°) field of view. As can be appreciated, the beam size of the radar beam 600a from the radar sensor 412 increases as a function (e.g., a linear function) of the distance from the radar sensor 412 to the obstacle. Specifically, with reference to Figure 6a , the cross-sectional dimensions of radar beam 600a at distance Y 604 are greater than the cross-sectional dimensions at distance X 602. For example, assuming a 2° (H)×25° (V) field of view, the cross-sectional dimensions at six miles from radar sensor 412 (i.e., distance X=6 miles) may be 2,200 feet (H)×14,000 feet (V) in voxels, while the beam dimensions at 36 miles (i.e., distance Y=36 miles) may be 13,000 feet (H)×84,000 feet (V). Note that the ratio of the beam dimensions in terms of horizontal (H) width to vertical (V) width remains substantially constant along the distance.

[0155] In order to provide a larger coverage area, the radar sensor 412 can be configured to rotate using a mechanically rotating infrastructure. For example, the mechanically rotating infrastructure can be coupled to the radar sensor 412 via a drive shaft to provide a mechanically rotating radar system to scan the airspace in a radial pattern around the aircraft. The rotating radar sensor 412 provides a full 360° coverage around the aircraft (in the XY plane) and a 25° azimuth field of view (in the XZ plane). The radar sensor 412 can rotate continuously at a predetermined rotation speed, such as 20 to 80 revolutions per minute (RPM), more preferably 40 to 60RPM, and most preferably 48RPM (e.g., 0.8Hz). Although the radar sensor 412 has a poor resolution compared to other technologies, the significant advantages of the radar sensor 412 are its range and relatively low cost.

[0156] The radar sensor 412 may be housed in a dome or other structure to protect the radar equipment. The geometry of the dome may be aerodynamic to reduce drag when traveling through the air. The dome is preferably made of a material that is transparent to radio waves and prevents contaminants (e.g., ice, sleet, dust, debris, etc.) from accumulating directly on the radar equipment (such as the surface of a radar antenna). In the case of a rotating / rotating radar dish, the dome also protects the antenna from debris and rotational irregularities caused by wind. In operation, the radar sensor 412 may combine the collected radar data with data collected by other sensors (or data sources) that may be used for the obstacle sensor equipment 162 in order to communicate with the core platform 102.

[0157] Cameras / Optical Sensors. As described above, obstacle sensor equipment 162 further employs one or more optical sensors (e.g., cameras) configured to pan and tilt, such as infrared camera 414 that scans airspace for thermal signatures and visible near infrared electro-optical (EO) sensor 416 that scans airspace to identify and help classify objects. Thus, while infrared camera 414 will be primarily described, other optical sensors 418 may similarly be used in addition to or in place of infrared camera 414, including, among others, ultraviolet, visible, near infrared, short wave infrared, mid wave infrared, long wave infrared (LWIR), bolometers, electro-optical cameras, laser radar (LIDAR), LED projection, structured light, multi-view reconstruction, and the like.

[0158] Figure 6bThe diagram illustrates an example infrared camera 414 and the infrared beam 600b it produces relative to the radar beam 600a. The infrared camera 414 can be a long wave infrared (LWIR) camera. An advantage of the infrared camera 414 is its thermal imaging capability. To facilitate use with a small aircraft, the infrared camera 414, like the radar sensor 412, is preferably compact, lightweight, and low cost. Suitable infrared cameras 414 include the FLIR model M-612L Thermal Night Vision System, which is 9 pounds and approximately 7" (L) x 7" (W) x 12" (H). The FLIR model M-612L provides a 640 x 480 VOx microbolometer sensor, a 50 mm focal length with a 12° (H) x 9° (V) (NTSC) field of view, and a zoom capability.

[0159] Like radar sensor 412, infrared camera 414 can be configured to monitor the airspace around the aircraft for both cooperative and non-cooperative obstacles, but at a higher resolution and shorter range. As illustrated, when in a fixed position, infrared camera 414 provides a horizontal beam width (W) at Y°. Horz. ), the horizontal beam width is wider than the vertical beam width at Z° (W Vert. ). Specifically, according to the specifications of the infrared camera, the infrared camera 414 can provide a 12° (H) (e.g., Y°=360° divided into 12°) × 9° (V) (e.g., Z°=360° divided into 9°) field of view, but other fields of view are available, such as 24°×18°, 25°×20°, etc. For example, based on the 12° (H) × 9° (V) field of view, the cross-sectional size of the infrared beam 600b at 6 miles (e.g., distance X=6 miles) can be 7000 feet (H) × 5200 feet (V). The cross-sectional size of the infrared beam 600b increases linearly as a function of the distance from the infrared camera 414, while the resolution decreases with distance. However, the infrared camera 414 provides a significantly greater resolution than the radar sensor 412. For example, at 6 miles, the infrared camera 414 is able to provide a pixel coverage of 11 feet × 11 feet per pixel, and the resolution of the infrared camera 414 is approximately 500 times greater than that of the radar sensor 412 at the same distance.

[0160] Compared to the radar beam 600a at distance X 602, the infrared beam 600b of the infrared camera 414 at distance X 602 is approximately half the size along the vertical axis (Z axis). In order to compensate for the narrower field of view along the vertical axis, the infrared camera 414 can be configured to translate and tilt. For example, the infrared camera 414 can be configured to tilt up and down (e.g., + / -90°) to expand the vertical field of view, and also configured to translate left (port) and right (starboard) (i.e., rotate 360° to provide continuous translation) to expand the horizontal field of view. The infrared camera 414 can translate continuously or stepwise (e.g., incremental lock-step), each step being a radial sector. In some aspects, the infrared camera 414 can continuously translate left / right and tilt up / down to cover the uncertain voxels (e.g., 25°×2°) of the radar sensor 412. In fact, the infrared camera 414 can provide a measurable field of view that is equal to or greater than the field of view specified by the Federal Aviation Administration (FAA). Thus, while infrared camera 414 suffers from poor range and a narrow vertical field of view (eg, making it difficult to provide 360 ​​degree coverage), an advantage of infrared camera 414 is its high resolution.

[0161] As with radar sensor 412, infrared camera 414 (or another camera type, as the case may be) may be housed in a dome or other structure to protect the camera equipment, which may include optically transparent portions to facilitate operation of infrared camera 414. Radar sensor 412 and infrared camera 414 may share a dome (e.g., a single equipment pod), or be located in separate domes that may be co-located or separately located on the aircraft. In operation, infrared camera 414 may combine collected infrared camera data with data collected by other sensors (or data sources) available to obstacle sensor equipment 162 for communication with core platform 102.

[0162] By way of illustration, an aircraft equipped with obstacle sensor equipment 162 may (1) detect non-cooperative obstacles at a known distance and velocity at a first resolution via radar sensor 412, and (2) image the non-cooperative obstacles at a second resolution (i.e., a higher resolution) via a camera (e.g., infrared camera 414, visible near infrared EO sensor 416, or other optical sensor 418). A processor, whether integrated with obstacle sensor equipment 162 or as part of core platform 102, may combine information from radar sensor 412 with the camera to identify non-cooperative obstacles. For example, the processor may combine the radar cross section from radar sensor 412 with the optical cross section from visible near infrared EO sensor 416 and the thermal cross section from infrared camera 414.

[0163] As understood by those of ordinary skill in the art, radar cross section generally refers to a measure of the ability of a target (i.e., a non-cooperative obstacle) to reflect radar signals in the direction of the radar sensor 412 (e.g., at the obstacle sensor equipment 162). In other words, the radar cross section provides a measure of the ratio of the backscattered power per steradian (unit solid angle) in the direction of the radar (from the target) to the power density intercepted by the target. Accordingly, the optical cross section refers to a value describing the maximum light flux reflected back from the light source from the non-cooperative obstacle, while the thermal cross section reflects the thermal imaging measurement data from the infrared camera 414 describing the non-cooperative obstacle. In some aspects, the obstacle sensor equipment 162 can be configured to identify non-cooperative obstacles using incomplete information. For example, if only the radar sensor 412 detects a non-cooperative obstacle, the processor coupled to the obstacle sensor equipment 162 can instruct the aircraft to create a large cylindrical avoidance zone, while the information from the camera can be used to limit the avoidance zone to a small spherical avoidance zone.

[0164] Figure 7 The diagram illustrates a perspective view of an example aircraft 700 having a fuselage 702, one or more wing panels 704 (or other flight surfaces), a tail 706, an equipment pod 708, and one or more propulsion devices 710 (e.g., a jet engine, one or more propellers axially driven by an engine or electric motor, etc.). Depending on the type of sensor, the obstacle sensor equipment 162 can be coupled to the aircraft externally or internally. For example, the obstacle sensor equipment 162 can be configured within the equipment pod 708 of the aircraft 700. Although Figure 7 Aircraft 700 is illustrated as a fixed wing aircraft, but the present disclosure is not limited to a particular aircraft configuration and may be applied to nearly any aircraft configuration, including multi-rotor VTOL aircraft.

[0165] Equipment pod 708 (which may house, among other things, obstacle sensor equipment 162 or portions thereof) can be rotatably and turnably coupled to fuselage 702 (or other structural member) via a gimbal system. For example, equipment pod 708 may be coupled to the front end of fuselage 702 to make it easier to orient equipment pod 708 forward to monitor obstacles along the flight path or trajectory of aircraft 700. Obstacle sensor equipment 162 may be coupled to aircraft 700 in a non-invasive manner to allow for easy removal and / or relocation to another aircraft. Alternatively, obstacle sensor equipment 162 may be installed during manufacture of aircraft 700, and thus, obstacle sensor equipment 162 may be permanently attached or integrated with aircraft 700. For example, one or more thrusters 710 may be positioned on fuselage 702 (e.g., a pushrod configuration as illustrated), on wing panels 704, or elsewhere on aircraft 700. Although the aircraft 700 is illustrated as having a single propeller 710, it should be understood that additional propellers 710 may be provided. For example, one or more propellers 710 may be provided on each wing panel 704.

[0166] Figures 8a to 8c A schematic diagram illustrating an example hybrid field of view generated by radar sensor 412 and infrared camera 414 for aircraft 700 is shown. As illustrated, radar sensor 412 and infrared camera 414 of obstacle sensor equipment 162 may be used in conjunction to more effectively detect and track non-cooperative obstacles 344 within the hybrid field of view. As can be appreciated, the hybrid field of view benefits from the 360-degree long-range scanning capability of radar sensor 412 and the ability of infrared camera 414 to accurately detect non-cooperative obstacles 344.

[0167] Figure 8aIllustration of two-dimensional (2D) radial map 800 represents a top plan view of the blended field of view. In one aspect, radar sensor 412 may be configured to generate a two-dimensional (2D) radial map 800 of the airspace adjacent to aircraft 700 without requiring azimuth resolution. 2D radial map 800 may be a circular airspace about aircraft 700 (i.e., aircraft 700 is located at the center of 2D radial map 800). The size of 2D radial map 800 may be indicated by the range of radar sensor 412. For example, if radar sensor 412 has an effective range of 36 miles (distance Y 604 = 36 miles), the circular airspace around the aircraft may have a radius of 36 miles. In operation, radar sensor 412 can be configured to identify obstacle 344 within 2D radial map 800, and if radar sensor 412 detects obstacle 344, infrared camera 414 may be pointed to obstacle 344 to provide a higher resolution observation (e.g., image) of obstacle 344. A processor operably coupled to infrared camera 414 may then compare the higher resolution image of obstacle 344 to a known shape / image database (e.g., a lookup table). For example, particularly in surveillance situations, the shape / image database may assist in classifying the type and / or threat level of obstacle 344. This technique may also be used to scale the threat level of an object where multiple obstacles 344 are detected to determine which obstacles are more dangerous and should be prioritized and avoided / attacked if necessary (e.g., a bird is less dangerous than an aircraft).

[0168] Figure 8b The diagram illustrates a side view of the hybrid field of view. If radar sensor 412 detects obstacle 344 within the radius of 2D radial map 800, infrared camera 414 may aim in the direction of obstacle 344 to provide higher resolution monitoring once obstacle 344 is within range of infrared camera 414 (e.g., within distance X 602, which may be an effective range of infrared camera 414). Figure 8b , infrared camera 414 may be tilted up and down to determine and track the azimuthal position (e.g., within a vertical field of view) of obstacle 344. For example, if infrared camera 414 provides a 25° azimuthal field of view, infrared camera 414 may be tilted 12.5° upward and 12.5° downward (relative to the line of flight).

[0169] Figure 8cThe top plan view of the 2D radial map 800 divided into a plurality of radial sectors 802 is illustrated. In order to locate the obstacle 344 within the airspace of the hybrid field of view, the 2D radial map 800 can be divided into a predetermined number of radial sectors 802 (i.e., N radial sectors). For example, each radial sector 802 can be represented as two straight edges (i.e., the same length as the radius of the circular 2D radial map) and a straight (or curved) edge. The predetermined number of radial sectors 802 can be indicated by an operating parameter of a sensor (e.g., infrared camera 414) of, for example, the obstacle sensor equipment 162. For example, in the above example, the infrared camera 414 provides a 12° field of view in the horizontal direction, and therefore, 360° can be divided into 30 radial sectors (i.e., N=360° / horizontal field of view=360° / 12°=30). Therefore, the infrared camera 414 can be configured to translate between N positions within the 360-degree 2D radial map 800. More specifically, infrared camera 414 need only scan designated radial sector 802 to determine the precise azimuth position of obstacle 344 .

[0170] As explained above, radar sensor 412 provides excellent range and is therefore able to more quickly identify distant obstacles 344 within 2D radial map 800. However, infrared camera 414 provides higher accuracy and resolution, but has a more limited range and field of view. Therefore, radar sensor 412 can be used to locate obstacles 344 within radial sectors 802 of 2D radial map 800 and guide infrared camera 414 to the radial sector 802 where obstacle 344 is located. For example, if radar sensor 412 detects an impending threat obstacle 344a within first radial sector 802a (e.g., within first predetermined distance 342a), infrared camera 414 can be translated from its current position to first radial sector 802a to perform further analysis and observation of impending threat obstacle 344a. Likewise, given the extended range provided by radar sensor 412, radar sensor 412 may detect intermediate threat obstacle 344b that may be located outside of the effective range of infrared camera 414 (e.g., between second predetermined distance 342a and second predetermined distance 342b). For example, if intermediate threat obstacle 344b is detected within second radial sector 802b, infrared camera 414 may translate from its current position to second radial sector 802b once threat obstacle 344b is within the effective range of infrared camera 414 to perform further analysis and observation of intermediate threat obstacle 344b.

[0171] Between the radar data from the radar sensor 412 and the camera data from the infrared camera 414, the obstacle sensor equipment 162 is able to provide data reflecting, among other things, the current location (in three dimensions), trajectory, and physical characteristics (e.g., size and shape) of each obstacle 344. In addition, the obstacle sensor equipment 162 can determine an identification (e.g., tail number, if known), status (e.g., cooperative or non-cooperative), etc. The location and various operating conditions of each obstacle 344 are then communicated to the core platform 102 and / or the flight control system 116 for appropriate action by the pilot or crew automation system 100. As described above with reference to Figure 3c to Figure 3e As discussed, the obstacle sensor equipment 162 and any obstacles 344 can be monitored by the human-machine interface 104 via the sensor equipment monitoring application 336. The sensor equipment monitoring application 336 can be configured to access all information collected by the radar sensor 412 and the infrared camera 414, as well as monitor the health of the sensors. The sensor equipment monitoring application 336 can also be configured to track air traffic and generate a predicted future path to be displayed to the pilot. For example, based on various radial azimuth distance obstacle measurements, the route manager 160 can generate an obstacle avoidance navigation route to avoid one or more obstacles detected by the obstacle sensor equipment 162. The sensor equipment monitoring application 336 can also be configured to prompt the pilot to select the generated obstacle avoidance navigation route to be executed by the flight control system 116.

[0172] Knowledge acquisition system 114. The knowledge acquisition system 114 collects and / or generates the necessary knowledge base to enable the crew automation system 100 to determine aircraft specific information. This includes knowledge of aircraft performance characteristics, limitations, checklists, procedures (including emergency procedures), and criteria that define aircraft emergencies. Data can be derived from a combination of coded data (e.g., from manuals, pilot briefings, pilot operating manuals) and data obtained in flight (e.g., via sensors) that supports offline machine learning and trend analysis. The data to be coded can be loaded in a .xml (or .xmlx) format that describes the content of a process and the flow of tasks within and between processes.

[0173] like Figure 1bAs illustrated in FIG, for example, the knowledge acquisition system 114 may receive actionable commands from the core platform 102 while sending configuration data and status and response information generated by the knowledge acquisition system 114 to the core platform 102. The operation of the knowledge acquisition system 114 may generally be divided into three processes, including, for example, aircraft system modeling, program compilation, and aerodynamic modeling. The aircraft system modeling process provides the crew automation system 100 with information about available onboard systems and how the onboard systems are configured, actuation limitations, and the like. The program compilation process provides the crew automation system with information about the operation of the aircraft under normal and abnormal conditions. For example, program compilation may include the compilation of checklists. Finally, the aerodynamic modeling process provides the crew automation system 100 with information about the flight of the aircraft and what performance is expected for a given aircraft type and configuration.

[0174] During the knowledge acquisition phase, conditions must also be established under which conditions a situation is considered an abnormal or emergency event. These conditions are frequently discrete, such as an engine overspeed or exceeding an airspeed limit. Using machine learning, the crew automation system 100 is able to fine-tune its aerodynamic and control models by observing a series of in-flight strategies flown by pilots. This information includes flight dynamics data, operational limitations, procedures, aircraft systems and layouts, and other relevant data. In addition to written information, the crew automation system 100 can also compile information based on past events and the experience of more experienced pilots. Machine learning enables the knowledge acquisition process to be performed efficiently and quickly.

[0175] Using the perception system 106 and actuation system 108 of the crew automation system 100, the instruments and controls in the aircraft cockpit or a real simulator are monitored as the pilot goes through the motions of a typical flight profile. Observing the actions of the pilot allows the crew automation system 100 to learn directly from the pilot and simulate smooth expert control for a given operation. This process benefits from the fact that flight operations are highly structured, because what to do in a given situation (machine learning) can then codify how to do something.

[0176] Population of the aircraft data structure 208 may be accomplished using extensible markup language ("XML"). More specifically, an XML data structure may be employed that contains a set of fields and data trees that, when populated, allow the core platform 102 to configure and operate the aircraft. In certain aspects, the crew automation system 100 may employ natural language interpretation of flight documents and / or software tools to enable humans to efficiently and accurately enter data.

[0177] In certain aspects, a set of aircraft agnostic features can be generated and encoded. For example, procedures such as landing gear retraction, engine factory procedures on multi-engine aircraft, and stall recovery are similar across many types of aircraft and require only minimal modification to a specific airframe. Additionally, basic airframe limitations (such as speeds never to be exceeded) need only be entered as specific numbers and can be entered from the flight guide within a nominal period of time.

[0178] Program editor. Aircraft specific information can be collected during the transition phase using, for example, written documents (e.g., pilot operating manuals, maintenance guides, etc.) and by direct monitoring of aircraft operations. The output of this knowledge acquisition process is an aircraft data structure 208, which is described above with respect to the core platform 102. Included in the aircraft data structure 208 may be operating procedures, available systems and their designs, cockpit layouts, and all other information necessary for safe operation of the aircraft. In some aspects, the crew automation software development kit can allow software / flight control engineers to detail, code, and unit test one aircraft subsystem (e.g., electrical or hydraulic) every day. The crew automation software development kit can provide tools for converting flight guide programs into state machines compatible with Matlab state flow and Simulink, which can then automatically code programs in C language for inclusion in the core platform 102. The crew automation software development kit can also generate unit-level test code and interfaces for testing the core platform 102. For example, the program editor can provide a task list in which the pilot can select whether the crew automation system 100 or the pilot is responsible for a given task on the list.

[0179] Knowledge Acquisition for Flight Controls. The first step in knowledge acquisition for flight controls uses the Athena VortexLattice ("AVL") method to generate a mathematical model in the form of dimensionless stability derivatives, which is used and refined during the pilot's flight. Once the primary flight control mechanisms are calibrated, the System ID Trainer application can be used to execute a sequence of flight maneuvers designed to identify specific stability derivatives. The data is automatically processed into updated stability derivatives for use in the controller. The controller can use an automatic tuner. The same updated stability derivatives are used in a 6-DOF simulation as a verification step that the controller is fully executed before flight. An additional benefit of performing knowledge acquisition for flight controls is that it enables a large amount of formal procedural knowledge to be refined and consolidated. Although the procedure lists the individual steps, the details of how to perform these steps can be ignored (for example, how long to wait between steps or how to increase the throttle significantly).

[0180] Reverse engineering of aircraft flight performance characteristics. Aircraft performance characteristics that can be measured by onboard data acquisition units are generally considered to be proprietary to aircraft and avionics manufacturers. This information can be used for flight simulation, aircraft health monitoring, aircraft development, etc. Currently, third parties who want to use onboard data acquisition are limited by its proprietary nature. This limitation is only partially overcome using independent aircraft sensor suites. These commercially available sensor suites only measure a small portion of the data available through cockpit instruments and pilot inputs. However, since the crew automation system 100 uses various sensors to determine the flight performance characteristics of the aircraft, it effectively reverse engineers the performance characteristics of the air vehicle. The crew automation system 100 collects aircraft information through independent sensors, data capture through images of cockpit instruments, and input controls.

[0181] Examples. Aspects of the present disclosure may be illustrated by the following example flight plan, which illustrates how the crew automation system 100 interacts with the pilot, executes the flight plan, performs flight operation tasks, responds to unexpected events during system engagement and takeoff, flight plan engagement, and anomaly detection and handling. However, the present teachings should not be limited to those used in this example.

[0182] System Engagement and Takeoff. The pilot enters the left seat of the aircraft, fastens his seat belt, comfortably positions the human machine interface 126 at his side, and activates the crew automation system 100 application. The application starts and runs a series of power diagnostics and mechanical interface power and calibration. A message may be displayed on the human machine interface 126 confirming successful tests and asking the pilot to confirm engagement of the crew automation system 100. The pilot selects a flight plan for the day via the application tab 334. The crew automation system 100 may be used for checklist monitoring. The pilot selects engine start and the crew automation system 100 may begin a sequence of engine start actions, requiring final confirmation before the actual start. At the same time, the pilot may call the tower for clearance and receive a flight plan for the training area.

[0183] When the engine start is complete, the crew automation system 100 may report success to the pilot and report, for example, "Ready to taxi" (audibly or via the human-machine interface 126). The pilot requests a taxi clearance, and upon hearing it, the crew automation system 100 transcribes the taxi clearance and displays it to the pilot for confirmation. The pilot then clicks the "Taxi via Clearance" button on the application, and the crew automation system 100 taxis to the assigned runway as the pilot monitors traffic. While at the runway threshold, the pilot verbally commands the crew automation system 100 to perform a pre-takeoff check (via a checklist), and the system completes all necessary checks, prompting the pilot to manually double-check critical items, such as flight controls. For example, the crew automation system 100 may monitor the human operator's execution of the checklist and output "Checklist Complete" or identify a flight plan or error.

[0184] After receiving further clearance, the pilot then commands the crew automation system 100 to direct the aircraft to line up, wait, and then finally take off. The crew automation system 100 pushes the throttle forward via the primary actuation system 108a, visually checks the engine and cockpit indicators via the sensor system 106, calls out the speed via the HMI system 104, and rotates at a speed appropriate for the current weight, balance, and density altitude. The pilot places his hand on the control stick / joystick 514 to confirm the crew automation system 100 input and maintain its muscle memory. The crew automation system 100 confirms the aircraft performance according to the current conditions and reports any deviation from the expected climb rate. During the climb, the pilot's workload is reduced by the crew automation system 100, allowing more head-up time (i.e., eyes forward, not on the instrument) to look for traffic in busy airspace. The crew automation system 100 can also provide experienced pilot suggestions for a given checklist, aircraft, or position. For example, at a particular airport, the crew automation system 100 may indicate airport specific cues to the human operator, such as "steep departure angle from this runway."

[0185] The flight plan is engaged. At the top of the climb, the crew automation system 100 levels the aircraft and adjusts trim and power settings while heading to the first waypoint of the flight plan. During cruise, the crew automation system 100 continues to visually monitor all cockpit displays, constantly comparing engine and aircraft performance to expected values ​​and alerting the pilot to any deviations.

[0186] The aircraft arrives at the training area and begins flight planning for the day. However, during flight planning, the aircraft enters towering cumulus clouds where instrument meteorological conditions ("IMC") conditions are below freezing temperatures. The pilot requests and receives permission from ground via an Internet Relay Chat ("IRC") chat window on the human-machine interface 126 to climb to 24,000 feet to cross the weather. In certain aspects, the crew automation system 100 requests permission from ground.

[0187] Anomaly Detection and Handling. After a period of time, the crew automation system 100 may detect that, assuming a climb, for these pitch and power settings, the indicated airspeed is slowly deviating from its modeled airspeed, indicating a lower than expected value. This indicates that the pitot heater has failed and the pitot tube has iced over. The pilot has flown the aircraft for less than 100 hours and is unaware that this model of pitot heater is known to be unreliable. The pilot has not yet noticed that the airspeed indicator is trending below nominal.

[0188] However, the flight crew automation system 100 recognizes that the airspeed data is anomalous to the rest of the flight data and its internal flight dynamics model and audibly alerts the pilot of an "airspeed indicator failure." Although the pilot recognizes that the airspeed information is currently unreliable, he is unsure whether the aircraft is flying faster or slower than the indicator shows.

[0189] Plotting on a database of previous anomalies, the crew automation system 100 presents a set of procedure options and highlights the lowest safe altitude for the area (e.g., 8,000 feet). The pilot selects the most conservative option, which results in wings level, pitch, and power down to a lower altitude (e.g., 10,000 feet). The crew automation system 100 eases off power, drops pitch slightly, and begins the descent. When descending to 15,000 feet, the pitot tube comes online again. Once stabilized at 10,000 feet, the crew automation system 100 keeps the aircraft straight and level while the pilot assesses the situation before returning to the flight plan.

[0190] In the competition of the day's flight plan, the crew automation system 100 can perform an automatic landing procedure. For example, the crew automation system 100 can navigate the aircraft to a predetermined waypoint, where the aircraft can begin its initial descent. During the descent, the crew automation system 100 can monitor flight conditions and locate the runway. In the final approach, the crew automation system 100 may slow down the aircraft and eventually land the aircraft. If the crew automation system 100 determines that landing is not feasible (e.g., obstacles or unacceptable flight conditions), the crew automation system 100 can start a missed approach routine or other contingency routine. For example, the crew automation system 100 can retry landing at the same location or navigate the aircraft to an alternative landing location. An example system for landing an aircraft at an alternative landing site is disclosed by co-owned U.S. Patent Application No. 2015 / 0323932, entitled "Autonomous Cargo Delivery System".

[0191] The crew automation system 100 and derived technologies can be applied to a wide range of aircraft and flight simulators. Flight performance characteristics derived from aircraft flight testing can be used to improve the fidelity of flight simulators used to train pilots. Providing flight simulators with access to actual aircraft performance data is of great value to flight simulator operators. Another benefit of the crew automation system 100 is that it can integrate flight performance characteristics when the aircraft is modified for a special flight plan, such as adding sensors and antennas that can affect aerodynamic performance and flight handling qualities (e.g., aircraft development). In addition, the data captured by the crew automation system 100 can be used for aircraft health monitoring, using predictions to sense maintenance needs.

[0192] The crew automation system 100 enhances the safety and utility of commercial aviation operations while providing significant savings in manpower operating costs. For example, the crew automation system 100 may be applied to long-haul air cargo companies to increase the safety and efficiency and cost savings of this advanced pilot assistance technology. In addition, the final state machine, for example, may be used as a training tool for pilots in flight, or as a safety system to provide a second set of eyes in a traditional single-pilot aircraft. The portion of the human-machine interface 126 simplifies flight operations for all pilots, even multiple crew operations.

[0193] The above-referenced patents and patent publications are hereby incorporated by reference in their entirety. Although various embodiments have been described with reference to particular arrangements of components, features, etc., these embodiments are not intended to be exhaustive of all possible arrangements or characteristics, and indeed many other embodiments, modifications, and variations may be ascertainable to those skilled in the art. Therefore, it should be understood that the present invention may therefore be practiced otherwise than as specifically described above.

Claims

1. An obstacle detection system for use in an aircraft (346, 700), the obstacle detection system comprising: a sensor (406) device that detects a non-cooperative obstacle (344) in a first airspace adjacent to the aircraft (346, 700), the sensor (406) device comprising a radar and a camera (410), the radar radially scanning the first airspace to generate radar information having a first resolution, the camera imaging a second airspace within the first airspace to generate optical information at a second resolution higher than the first resolution; and A processor is operably coupled to the sensor (406) equipment, wherein the processor is configured to determine a location of the non-cooperative obstacle (344) and to identify the non-cooperative obstacle (344) based on the radar information and the optical information.

2. The obstacle detection system of claim 1, wherein the camera (410) is configured to pan and tilt.

3. The obstacle detection system of claim 2, wherein the camera (410) comprises a long wave infrared sensor (406).

4. The obstacle detection system according to claim 2 or claim 3, wherein the camera (410) comprises a visible near infrared electro-optical sensor (406), i.e. a visible near infrared EO sensor (406).

5. The obstacle detection system of claim 2, claim 3 or claim 4, wherein the optical information comprises at least one of a thermal cross section and an optical cross section.

6. An obstacle detection system according to claim 1, claim 2, claim 3, claim 4 or claim 5, wherein the radar information includes a two-dimensional position, i.e., a 2D position, of the non-cooperative obstacle (344) in the first airspace, and the optical information includes an azimuth position of the non-cooperative obstacle (344) in the second airspace.

7. An obstacle detection system according to claim 1, claim 2, claim 3, claim 4, claim 5 or claim 6, wherein the processor is configured to generate a predicted flight path for the non-cooperative obstacle (344) based at least in part on the radar information and the optical information.

8. The obstacle detection system of claim 1, claim 2, claim 3, claim 4, claim 5, claim 6 or claim 7, wherein the first airspace provides a 360 degree field of view about the aircraft (346, 700).

9. The obstacle detection system of claim 8, wherein the 360 ​​degree field of view resides in a plane parallel to a flight path defined by the aerial vehicle (346, 700).

10. A method for detecting and avoiding non-cooperative obstacles (344) during operation of an aircraft (346, 700), the method comprising: Scanning a first airspace using a radar system to generate radar information having a first resolution; imaging a second airspace using a camera (410) to generate optical information at a second resolution higher than the first resolution, wherein the second airspace is located within the first airspace; tracking the non-cooperative obstacle based at least in part on the radar information and the optical information (344); generating a predicted flight path based at least in part on the radar information and the optical information; generating an obstacle avoidance navigation route to avoid the non-cooperative obstacle (344); as well as The obstacle avoidance navigation path is communicated to a flight control system of the aircraft (346, 700).

Images