Method and system for contextually cascading display, auditory, and voice alerts

Abstract

A method and system for monitoring auditory and message alerts generated by systems within the cockpit and received in an Internet of Things (IoT) cockpit of an aircraft during flight, the method comprising: receiving a plurality of alerts including at least one of an auditory alert or a message alert; applying a first natural language processing (NLP) process to the auditory alert to convert the auditory alert into a text alert that is structurally consistent with the message alert for aggregation with the message alert to form a cascaded message alert; and identifying a context of the cascaded message alert by applying a second NLP process in its entirety to the cascaded message alert and then tagging the cascaded message alert to associate the tagged message with a display element, wherein the tagged message is a cascaded message.

Description

Technical Field

[0001] The present invention relates generally to alerts with cockpit displays and, more particularly, to methods and systems for contextually concatenating display, auditory, and voice alerts in a cockpit display system. Background Art

[0002] As the increase in air traffic results in more aircraft leaving and arriving at airports, there is also an associated increase in communications between aircraft and between aircraft and ground controllers to ensure smooth operation, efficiency and safety of air traffic in the air and on the ground. In order to meet such greater demands (especially with respect to safety and efficiency), pilots keep an assessment of any abnormal or relevant flight information during flight with the help of relevant information and alerts on different systems. These alerts or messages may occur when various aircraft systems detect certain conditions or when ground air traffic controllers or airline operations send relevant information to the aircraft via a communication channel. The published messages and alerts can be displayed on the display system using intuitive technology, or the published messages and alerts can be audibly expressed by ground controllers or airline operations through an audio channel, or the published messages and alerts can be communications between cockpit crew members. In addition, different cockpit systems can be used to independently evaluate the specific conditions of the messages to be displayed or audibly expressed by voice.

[0003] While the system may display or audibly voice such information, such systems fail or do not collate messages or audible alerts and do not display information from these multiple sources by making a determination or assessment of the applicability of a specific message prior to any display on the cockpit display. Furthermore, the system does not identify conflicting system alerts, advisories, or information. Therefore, the responsibility for making such assessments, identifications, and determinations falls upon the pilot or flight crew and their ability to aggregate information referenced from multiple systems to use this information to articulate a complete picture of a particular situation and to judiciously identify conflicts to comply with a specific system advisory in an instance.

[0004] Therefore, it is desirable to address these deficiencies in understanding the messages and auditory alerts received by a pilot with an improved alert system and method that allows cascading messages and alerts from different systems, both displayed and auditory, based on the context of the flight and presenting an integrated view of the various related messages and alerts to the pilot. The present disclosure addresses at least this need. Summary of the invention

[0005] A method and system for cascading display, auditory, and text messages and alerts corresponding to the same context is described that can identify related auditory, text messages and / or announcements and present the alert information to the pilot in a contextual manner, which increases the likelihood that the pilot will understand the complete view of the specific situation related to the received alert.

[0006] In an exemplary embodiment, a method of monitoring auditory and message alerts generated by systems within a cockpit received in an Internet of Things (IoT) cockpit of an aircraft during flight includes: receiving a plurality of alerts including at least one of an auditory alert or a message alert, wherein the auditory and message alerts are from entities including: an internal system of the aircraft, air traffic control (ATC) communications, and internal and external communications of the aircraft; applying a first natural language processing (NLP) process to the auditory alerts to convert the auditory alerts into text alerts that are structurally consistent with the message alerts for aggregation with the message alerts; Forming a cascade of message alerts, wherein NLP is applied with context-based attributes to determine the context of the audible alert at least by matching a set of context attributes previously derived for a particular aircraft with those locally stored therein; and identifying the context of the cascaded message alert by applying a second NLP process in its entirety to the cascaded message alert and then tagging the cascaded message alert to associate the tagged message with the display element by matching the context of the cascaded message derived by the second NLP process from a tag of the tagged message with a particular display element from a plurality of display elements, wherein the tagged message is a cascaded message.

[0007] In another exemplary embodiment, a cockpit display system for use onboard an aircraft, the cockpit display system comprising: a source of sensory and message alerts; a processor communicatively coupled to the source of sensory and message alerts and a cockpit display, the processor configured to process the sensory and message alerts to: receive a plurality of alerts including at least one of a sensory alert or a message alert, wherein the sensory and message alerts are from an internal system of the aircraft, from internal communications, or from external communications; apply a first natural language processing (NLP) process to the sensory alerts to convert the sensory alerts into text alerts that are structurally consistent with the message alerts for use with the message alerts; aggregating together to form a cascade of message alerts, wherein NLP is applied with context-based attributes to determine the context of the sensory alert at least by matching a set of context attributes previously derived for a particular aircraft with those locally stored therein; and the context of the cascaded message alert is determined by applying a second NLP process in its entirety to the cascaded message alert and then tagging the cascaded message alert to associate the tagged message with the display element by matching the context of the cascaded message derived by the tag of the tagged message through the second NLP process with a particular display element from a plurality of display elements, wherein the tagged message is a cascaded message.

[0008] In an embodiment, a cockpit display system for use onboard an aircraft, the cockpit display system comprising: a source of sensory and message alerts; a processor communicatively coupled to the source of sensory and message alerts and a cockpit display, the processor configured to process the sensory and message alerts to: receive a plurality of alerts including at least one of a sensory alert or a message alert, wherein the sensory and message alerts are from an internal system of the aircraft or from air traffic control (ATC) communications; apply a first natural language processing (NLP) process to the sensory data alerts to convert the sensory alerts into text alerts that are structurally consistent with the message alerts for aggregation with the message alerts; together to form a cascading message alert type, wherein NLP is applied with context-based attributes to determine the context of the sensory alert at least by matching a set of context attributes previously derived for a specific aircraft with those locally stored therein; and the context of the cascading message alert is determined by applying a second NLP process in its entirety to the cascading message alert and then tagging the cascading message alert to associate the tagged message with the display element by matching the context of the cascading message derived by the tag of the tagged message through the second NLP process with a specific display element from a plurality of display elements, wherein the tagged message is a cascade message.

[0009] In yet another exemplary embodiment, a method for presenting information on a display system for use onboard an aircraft, the method comprising: receiving from a source of aircraft flight alerts; and at a processor communicatively coupling the source of aircraft flight alerts and a display device, the processor being configured to process the aircraft flight alerts to receive a plurality of flight alerts including at least one of a set of auditory alerts, announcement alerts, or message alerts, wherein the auditory, announcement, and message alerts are from an internal system of the aircraft, from internal communications, or from external communications, the processor further comprising a speech recognition processor to: apply a first natural language processing (NLP) process to the auditory alerts to convert the auditory alerts or announcement alerts into a structured speech recognition signal; A text alert consistent with the message alert is used to be aggregated with the message alert to form a cascaded message alert, wherein NLP is applied with context-based attributes to determine the context of the auditory alert or announcement alert at least by matching a set of context attributes previously derived for a specific aircraft with those locally stored therein; and the context of the cascaded message alert is determined by applying a second NLP process to the cascaded message alert and then marking the cascaded message alert to associate the tagged message with the display element by matching the context of the cascaded message derived by the tag of the tagged message through the second NLP process with a specific display element from a plurality of display elements, wherein the tagged message is a cascaded message.

[0010] This Summary is provided to describe selected concepts in a simplified form that are further described in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0011] Furthermore, other desirable features and characteristics of the system and method will become apparent from the subsequent detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing background. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention will hereinafter be described in conjunction with the following drawings, wherein like numerals represent like elements, and wherein:

[0013] Figure 1 is a block diagram of a system suitable for use in an aircraft according to an exemplary embodiment described herein;

[0014] Figure 2 is a block diagram of a system suitable for use in an aircraft according to an exemplary embodiment described herein;

[0015] Figure 3 is suitable for use with Figure 1 A flowchart of an exemplary method for use with a system;

[0016] Figure 4 is a first example of a controller clearance processed by the exemplary embodiments described herein;

[0017] Figure 5 is a second example of a controller clearance processed by the exemplary embodiments described herein;

[0018] Figure 6 is a third example of a controller clearance processed by the exemplary embodiments described herein; and

[0019] Figure 7 is a fourth example of a controller clearance handled by the exemplary embodiments described herein. DETAILED DESCRIPTION

[0020] The following detailed description is merely exemplary in nature and is not intended to limit the invention or the application and use of the invention. As used herein, the word "exemplary" means "used as an example, instance or illustration". Therefore, any embodiment described herein as "exemplary" is not necessarily to be interpreted as being preferred or advantageous over other embodiments. All embodiments described herein are provided to enable those skilled in the art to make or use exemplary embodiments of the present invention and do not limit the scope of the invention as defined by the claims. In addition, there is no intention to be bound by any express or implied theory presented in the aforementioned technical field, background technology, content of the invention or the following detailed description.

[0021] Although the exemplary embodiments described herein refer to displaying information on an aircraft, the invention may also be applied to other vehicle display systems, such as displays used by off-site controllers (e.g., ground controllers) or even other vehicle displays and warning systems (such as automobiles, etc.). Moreover, message displays and audible warnings are merely used as exemplary embodiments in which notifications are made to the pilot. However, it is contemplated that other different modal notifications may be used, including but not limited to visualization of differently colored or formatted data presented, notifications by light, vibration notifications, tactile feedback and gestures, etc.

[0022] Increasing information automation places a greater burden on flight personnel to obtain and interpret information related to the aircraft. In addition, with these increased burdens, there may be an increase in human error, which is attributed to multiple factors including the following: uncoordinated cockpit system messages, poor external visibility, fatigue, lack of position awareness, misunderstanding ATC clearances, multiple or conflicting ATC clearances, or inadvertent flight deck distractions still occur, leading to accidents and airframe losses. Therefore, accidents and near miss scenarios attributed to conflicting instructions or permissions are not uncommon and are proven or may be proven in aircraft operations almost every day. That is, because of these factors, pilots or flight personnel cannot or may not derive a complete picture when receiving a set of non-sequential elements, and it is impossible to properly combine a set of received and displayed messages or auditory alarms from the sources of several systems in an appropriate coherent manner for understanding a specific situation.

[0023] While reference is made to communications with respect to ATC communications, it is contemplated that such communications are not limited to ATC communications but are instead limited to a variety of ground controller communications between aircraft and ground controllers and between aircraft and aircraft, including Aircraft Operations Control (“AOC”). Furthermore, references to message displays and auditory alerts are used as exemplary embodiments in which notifications are made to the pilot. However, the use of such examples should not be construed as restricting other different modal notifications, including but not limited to differently colored or formatted visualizations of presented data, notifications formed from light and vibration notifications, and hand gestures, among others.

[0024] The Next Generation (NextGen) overhaul of the U.S. airspace system and the partner Single European Sky ATM Research (SESAR) overhaul of the European airspace system have created various track-based mechanisms to improve air traffic management on these continents. That is, electronic instrument displays continue to advance in sophistication, achieving higher and higher levels of information density and, therefore, presenting greater amounts of visual information to be perceived and understood by operators (e.g., pilots). It is important that the visual display provides the correct cognitive mapping between what the operator is trying to achieve and the information available to accomplish the task.

[0025] In addition, electronic flight bag (EFB) solutions, which include electronic display systems for use on a flight deck or by cabin crew members, are becoming increasingly popular. For example, an EFB device can display a variety of aviation data or perform basic calculations (e.g., performance data, fuel calculations, takeoff and landing calculations, etc.). In the past, some of these functions had to be entered manually using paper references, or based on data provided to the flight crew through the airline's flight dispatch function. Therefore, EFB solutions increase the burden placed on the flight crew to interpret the displayed information when it is displayed. However, there is a lack of integration of accurate ATC instructions and other information displays (such as moving maps and electronic charts).

[0026] Thus, in response to potential information overload or differential analysis of received information, an advanced connected cockpit environment of an IOT connected cockpit may implement an autonomous system that allows or facilitates monitoring of relevant current cockpit system auditory and message alerts and may allow intelligent cascading of auditory alerts or ATC / AOC voice transcribed data into their corresponding contextual display system messages as needed to achieve conflict resolution and improve overall aircraft safety and performance. Moreover, it should be appreciated that the cascaded message may be annunciated by displaying the cascaded message on the relevant display system or playing it back to the pilot as an auditory alert, or any other relevant mode of annunciation may be selected based on the context of the message, flight phase, and human factors considerations.

[0027] Therefore, it is desirable that the method and system monitor the current cockpit auditory message or announcement, determine the nature of the auditory message, see if a contextual cascade can be completed, identify the contextual message, convert the context into an alert message, and tag it to an existing display or graphical element, and provide a symbology for replaying the alert message as an auditory alert along with the cascaded message. Such a system would increase pilot efficiency, situational awareness, and improve safe terminal area operations by automatically processing transcribed voice instructions and system auditory alerts from multiple sources, and identifying relevant instructions that affect the aircraft. The system would utilize all the benefits of voice communications, auditory and display alerts, voice transcription, and provide relevant and intuitive instructions to the pilot regarding possible conflicts or intrusion threats for safe and continued flight mission execution.

[0028] In an exemplary embodiment, in an accident where an aircraft in taxiing collides with another aircraft that has been cleared to land. This accident may be caused by any of the aircraft not knowing the status of the other aircraft. For example, in a specific instance, an aircraft (designated aircraft "A") may have misunderstood the direction of a taxi outclearance at a specific airport and began to enter the same runway on which another aircraft (e.g., designated aircraft "B") was, and the other aircraft had given a valid landing clearance, but did not communicate on the same control tower ("TWR") frequency. This other aircraft (aircraft "B") is or will proceed with a touchdown maneuver from an approach in the opposite direction or opposite end of the runway in use. In the view of aircraft "A" that has stopped with the nose of the aircraft protruding onto the runway, aircraft "B" applied maximum manual braking and stopped just before reaching aircraft "A". In other words, because of air traffic control ("ATC") and airport procedures and aircraft "A" crew actions, the actions of each of the parties contributed to the risk created.

[0029] Therefore, in view of the above types of scenarios and the need to prevent such unsafe scenarios, it is desirable for a system to monitor auditory alerts and messages from various cockpit systems, determine the nature of the auditory alert / response, determine whether context addition can be completed for the auditory response, identify the context message and tag the message to a display element, provide symbology to output the cascaded message as an audible signal to the pilot to prevent miscommunication and subsequent risk as a result of aircraft operation.

[0030] Certain terms are used with respect to various embodiments of the present disclosure. For example, a display unit is an aircraft onboard device that provides a user interface for a specific avionics system onboard the aircraft, which may include a flight management system (FMS), a communication management function (CMF), an assigned spacing target (ASG), point-by-point realization (ABP), broadcast automatic dependent surveillance (ADS-B), broadcast flight information service (FIS-B), or other avionics systems. The flight crew can use the FMS interface to perform tasks associated with flight planning, navigation, guidance, and performance. The processor can be coupled to the display unit to format the communication of data transmissions originating from one or more avionics systems onboard the aircraft, and it is directed to the cockpit display onboard the aircraft.

[0031] Figure 1 A block diagram depicts an exemplary aircraft navigation and control system in accordance with the disclosed embodiments. Figure 12 is a functional block diagram of an avionics display system 20 according to an exemplary embodiment. The avionics display system 20 includes at least one processor 22 and at least one monitor 24 (operably coupled to the processor 22). During operation of the avionics display system 20, the processor 22 drives a graphics module 27, which, in conjunction with the processor 22 driving the monitor 24, generates a graphic display 26 that visually provides information to the pilot and flight crew about the aircraft and neighboring aircraft within a predetermined proximity of the host aircraft. The processor 22 includes a speech recognizer 11 and can be used to provide the functionality of the speech recognizer 11. The speech recognizer 11 can convert received voice transmissions into text and perform appending of text and concatenation of text messages with system messages. The speech recognizer 11 can perform natural language processing ("NLP") of received audio or voice transmissions and monitoring of broadcasts and transmissions for conversion to text and further processing by the processor 22. The speech recognizer 11 can perform or assist the processor 22 in performing functions related to contextual interpretation and aggregation of received broadcast and transmitted voice transmissions. These transmissions may include transmissions from ATC and aviation operating control ("AOC") to the aircraft or other aircraft, as well as broadcasts. The graphic display 26 may include a visual representation of one or more flight characteristics related to neighboring aircraft, as described more fully below. The processor 22 may generate the graphic display 26 in a two-dimensional format (e.g., as a lateral or vertical profile display) or in a hybrid format (e.g., in a picture-in-picture or split-screen arrangement) and may be incorporated into all units capable of displaying TCAS data; such as a primary flight display, a multifunction display, and an interactive navigation display. The processor 22 may generate auditory and audio messages to an audio output device 40 that provides information to the pilot and flight crew about the aircraft and neighboring aircraft within a predetermined proximity of the host aircraft. An audio capture device 42 may also be included connected to the processor 22. This audio capture device 42 may capture auditory alerts from other systems of the aircraft or from pilot communications for processing by the processor 22.

[0032] The processor 22 may include or be associated with any appropriate number of single microprocessors, flight control computers, navigation equipment, memory, power supplies, storage devices, interface cards, and other standard components known in the art. In this regard, the processor 22 may include or be associated with any number of software programs (e.g., avionics display programs) or instructions designed to implement the various methods, process tasks, calculations, and control / display functions described below, for example, the processor 22 may be included in a flight management computer (FMC) of the type typically deployed in a flight management system (FMS). The processor 22 may implement functions associated with parsing, transcribing, aggregating, and appending auditory and text messages received by various inputs to or from the internal systems of the aircraft. The processor 22 with the speech recognizer 11 may include an application and solution for a natural language processing (NLP) process for executing the input of the auditory alarm to convert the auditory alarm into a text alarm. The conversion may include an auditory alarm that is structurally consistent with the message alarm for aggregating with the message alarm to form a cascaded message alarm type. NLP may be applied based on application of the processor 22 with the speech recognizer 11 to determine the context of the audible alert with context-based attributes by matching a set of context attributes previously derived for a particular aircraft with those stored locally in the memory or storage 55 of the processor 22 .

[0033] Image generating devices suitable for use as monitors 24 include various analog (e.g., cathode ray tubes) and digital (e.g., liquid crystal, active matrix, plasma, etc.) display devices. In some embodiments, monitor 24 may take the form of a head-down display (HDD) or a head-up display (HUD) included in an electronic flight instrument system (EFIS) of an aircraft. Monitor 24 may be set at various locations throughout the cockpit. For example, monitor 24 may include a primary flight display (PFD) and reside at a central position within the pilot's primary field of view. Alternatively, monitor 24 may include a secondary flight panel display (such as an engine instrument and crew advisory system (EICAS) display), which is mounted at a location convenient for observation by the aircraft crew but typically resides outside the pilot's primary field of view. In further embodiments, monitor 24 may be worn by one or more members of the flight crew.

[0034] The processor 22 includes one or more inputs operatively coupled to one or more air traffic data sources. During operation of the avionics display system 20, the air traffic data sources continuously provide navigation data related to any nearby aircraft to the processor 22. Figure 1In the exemplary embodiment illustrated in FIG. 1 , the air traffic data source includes a wireless transceiver 28 and a navigation system 30, which are operatively coupled to first and second inputs of the processor 22, respectively. The navigation system 30 includes an onboard radar 32 and various other onboard instruments 34, such as a radio altimeter, a barometric altimeter, a global positioning system (GPS) unit, etc. In a preferred embodiment, the navigation system 30 may be included in an FMS; and the onboard radar 32 may be included in a terrain awareness and warning system (TAWS), such as an enhanced ground proximity warning system (EGPWS).

[0035] In some embodiments, the graphic display 26 may provide output from the aircraft's onboard radar 32. For example, the graphic display 26 may provide a top-down view, a horizontal view, or any other view of weather conditions, objects, and / or specific terrain detected by the aircraft's onboard radar 32. The view of weather conditions may include a monochrome or color graphical representation of the weather. The graphical representation of weather conditions may include an indication of the altitude (or altitude coordinates) of those objects or an altitude relative to the aircraft.

[0036] The avionics display system 20 provides display information to the pilot or flight crew in a highly intuitive manner. For this exemplary embodiment, the avionics display system 20 includes a processor 22 connected to a database 14, a flight management system 16, a navigation system 30, a graphics module 27, and a graphics display 26. In addition, it is contemplated that although in Figure 1 The avionics display system 20 appears to be arranged as an integrated system, but the present disclosure is not intended to be so limited and may also include arrangements whereby one or more of the processor(s) 22, database 14, flight management system 16, navigation system 30, graphics module 27, and graphics display 26 are separate components or subcomponents of another system located onboard or external to the aircraft.

[0037] Also, for example, the avionics display system 20 may be arranged as an integrated system (e.g., an aircraft display system, a primary flight display system, etc.) or as a subsystem of a more comprehensive aircraft system (e.g., a flight management system, a navigation and control system, a targeting and control system, a collision warning and / or avoidance system, a weather avoidance system, etc.). Furthermore, the present disclosure is not limited to aircraft displays, and may also be implemented for electronic displays of other types of vehicles (such as, for example, spacecraft navigation displays, ship navigation displays, submarine navigation displays, train navigation displays, motor vehicle navigation displays, etc.).

[0038] The database 14 may also include, for example, a terrain database, which may include the locations and heights of natural terrain obstacles, such as mountains or other elevated ground areas, and also the locations and heights of man-made obstacles, such as radio towers, buildings, bridges, etc. As another example, the processor 22 may retrieve and / or receive airport and runway location data, as well as other types of high priority target data (e.g., locations of incoming traffic to be avoided, constructed waypoints, obstacles in the flight path of the aircraft, etc.) from an appropriate source of such data, such as, for example, an onboard flight management system database (e.g., a component of the flight management system 16), an onboard navigation database (e.g., a component of the navigation system 30), an onboard sensor 36 or an onboard radar 32, or an external database (e.g., via a data communication uplink).

[0039] The aircraft may be any air vehicle that uses the flight management system 16 as a primary user interface for flight personnel to interact with the avionics display system 20 onboard the aircraft. The aircraft may be implemented as an airplane, a helicopter, a spacecraft, a hovercraft, etc. The one or more avionics systems may include a flight management system (FMS), an aircraft interface device (AID), etc. The data obtained from the one or more avionics systems may include, without limitation: flight plan data, aircraft status data, weather data, braking system data, fuel and weight data, runway analysis data, aircraft performance data, etc.

[0040] The memory 55 may be external to the processing unit and operatively coupled to the processing unit, or instead integrated into the processor 22. In one embodiment, the processor and the memory of the processor 22 reside in an application specific integrated circuit ("ASIC"). The memory 55 may store data, such as various software or firmware, that supports the operation of the processor 22 and other components included in the avionics display system 20 (such as graphics systems, sensor systems, and sources of aircraft status data). In addition, the memory 55 may store one or more onboard databases or be connected to the database 14. Onboard the aircraft, the database 14 may include a navigation database, a terrain database, a weather database, a historical trend database, and / or a runway database, such as an enhanced ground proximity warning system ("EGPWS") runway database.

[0041] Processor 22 and graphics module 27 cooperate to display, render, or otherwise communicate one or more graphical representations, synthetic displays, graphic icons, visual symbology, or images associated with the operation of avionics display system 20 on display device 26. Embodiments of avionics display system 20 may utilize existing graphics processing techniques and technologies in conjunction with graphics module 27. Graphics module 27 is suitably configured to support well-known graphics techniques and display technologies, including (i) synthetic vision, (ii) enhanced vision, (iii) combined vision, and (iv) compressed attitude.

[0042] The display 26 may include any one or more image generating devices capable of producing one or more navigation displays of the type described herein. As a point of emphasis, the term "display device" includes display devices (image generating devices) fixed to the cockpit of an aircraft (A / C), as well as electronic flight bags ("EFBs") and other portable display devices (which can be carried by pilots into the cockpit of the A / C and perform the functions described below). For example, to name just a few examples, the display 26 may implement one or more of a multifunction display (MFD), a three-dimensional MFD, a primary flight display (PFD), a synthetic vision system (SVS) display, a vertical position display (VSD), a horizontal position indicator (HSI), a traffic awareness and avoidance system (TAAS) display, and a three-dimensional TAAS display. In addition, the display may be implemented using multiple types of displays 26, each of which may implement one or more of these different non-limiting displays. Regardless of the number or specific type of display used to implement the display 26, it is noted above that the display 26 responds to the image reproduction display commands it receives to reproduce various images. The image reproduced by the display 26 will depend on, for example, the type of display implemented.

[0043] When performing the above functions, the display 26 may also take into account input data received via the user input interface device 21. In this regard, the user input interface may include any number and type of input devices suitable for receiving pilot input, which may be distributed throughout the cockpit of the aircraft (A / C) and may be included in other systems or subsystems. In one embodiment, the user input interface takes the form of or includes an alphanumeric keypad of the FMS.

[0044] As the data and information is received, the processor 22 is configured to continuously process the information to identify the predicted tracking angle, roll, pitch and yaw of the aircraft A / C and generate a symbology representing the same. The symbology is an aircraft orientation cue to be displayed on the display 26, and the dimensions of the symbology vary predictably to indicate the predicted tracking angle, roll and yaw, respectively.

[0045] Within other onboard instruments 34 , each sensor may include one or more sensor techniques, devices, instruments (such as onboard radar, radar altimeter, global positioning system (GPS)), and software sufficient to detect and provide aircraft A / C status data (including speed, position, location, remaining fuel, faults, status, and detected weather and temperature).

[0046] Figure 2 A block diagram depicts an exemplary aircraft navigation and control system in accordance with the disclosed embodiments. Figure 2 Includes an Automatic Dependent Surveillance-Broadcast (ADS-B) receiver 205, which is an integral component of the Next Generation National Airspace Strategy, where the aircraft receives Flight Information Service-Broadcast (FIS-B) and Traffic Information Service-Broadcast (TIS-B) data as well as other ADS-B data (such as direct communications from nearby aircraft at the traffic computer 210). The traffic computer 210 receives the ADS-B and generates target aircraft state parameters to the aircraft interface device (AID) 215. The traffic computer 210 may also include a processor for a speech recognizer 211 and may provide speech recognition functionality to convert voice transmissions into text. In addition, the speech recognizer 211 may include semantic context interpretation for voice transmissions and solutions for appending and aggregating to other text messages or flight data. For example, the functions and solutions of the speech recognizer 211, together with the functions and solutions of the traffic computer 210, may be used to match auditory and text messages, concatenate auditory and text messages, and append or aggregate pairs of auditory transformed messages with text messages in a similar or identical structure. Furthermore, the AID 215 receives flight data, weather, wind, and inputs received from system wide information management / air traffic control (SWIM / ATC) data sources 220 . The aircraft interface device (AID) 215 is coupled to the FMC 225 and sends speed data to the autopilot 230 .

[0047] The ADS-B receiver 205 is considered an air traffic data source because the ADS-B receiver 205 receives navigation data from an external control source and relays this data to the traffic computer 210. For example, the ADS-B receiver 205 may receive Traffic Information Service-Broadcast (TIS-B) data from an external control source. In a preferred embodiment, the ADS-B receiver 205 receives Traffic Collision Avoidance System (TCAS) data and may receive Automatic Dependent Surveillance-Broadcast (ADS-B) data from neighboring aircraft. This data, as well as other such external source data, is formatted to include air traffic information, which may be used to determine the current location of neighboring aircraft and the presence and location of air traffic.

[0048] In addition, TCAS is an air traffic system that detects and tracks aircraft near the host aircraft. TCAS includes a processor, an antenna, a traffic display (such as an LMAP display, a VSD, etc.), and a device for controlling an aviation display system (such as Figure 1 ). The processor and antenna detect and track other aircraft by interrogating their transponders and tracking these potential intruders on the display. The TCAS processor analyzes the transponder replies for a determined range, bearing, and relative altitude. If the system determines that a potential hazard exists, it issues a visual and audible advisory to the crew. The visual advisory takes the form of symbols on one or more traffic displays; for example, an LMAP display and a VSD.

[0049] Figure 3 A flow chart of message transmission and audible alerts for the communication system of the present disclosure is disclosed. Figure 3 In the system, the system will monitor auditory alarms and messages from various cockpit systems, determine the nature of the auditory alarm / response, determine whether context addition can be completed for the auditory response, identify the context message and mark the message to the display element. The monitored cockpit systems may include, but are not limited to, cockpit display systems, flight management systems, EGPWS, TCAS, weather or radar systems, EFB equipment, etc. Moreover, it is envisioned that in an IoT environment, many of the participating systems can be connected to each other and additional systems connected or interconnected can be easily added or removed. In addition, the system will provide a symbol system to output cascaded messages to the pilot as audible signals. Context-based cascading is performed with the intention of providing more information to the pilot. The message is intuitive and sufficient to provide the pilot with additional situational awareness and support his quick decision-making.

[0050] exist Figure 3This system is illustrated in a flowchart of FIG. 310 , where initially at task 310 , the processor of the communication system monitors air traffic control (“ATC”) and air and aeronautical operations control (“AOC”) communications and auditory alerts sent to the cockpit of the aircraft. That is, the ATC / AOC or service can issue reports and alerts to the pilot in the cockpit in the form of message display alerts and / or audible auditory alerts on the internal system. In an exemplary embodiment, some examples of such reports or auditory alerts include reports and alerts of the following actions: landing of the aircraft, takeoff of the aircraft, altitude change of the aircraft, and taxiway clearance of the aircraft for a specific runway, etc. These reports and auditory alerts can be issued by an air traffic controller (ATC) or aeronautical operations control (“AOC”) using a voice channel or CPDLC uplink message used to send communications. In addition, emerging techniques can also allow remotely piloted aircraft or even technology, where a controller on the ground is monitoring a specific aircraft or a group of aircraft, and in such a scenario, relevant messages and alerts for a specific flight can be on a system remote to the aircraft.

[0051] In an exemplary embodiment, these multiple systems may include smart runways, smart landings, AMM, and synthetic vision systems. However, it is contemplated that there may be additional or different or fewer multiple systems and therefore the monitoring of reports and auditory alarms is not limited in number or in nature to the multiple systems and types of systems used to generate reports or auditory alarms. In other words, in task 325, the process may transcribe features and extract the features from the auditory alarms and communications using a natural language process that is independent of the original system that generated the auditory alarms and communications. Therefore, even if a particular system uses a different protocol or platform to issue an auditory alarm or communication.

[0052] Next, in Figure 3 In the present invention, it is expected that the system will perform the steps of intelligently cascading the data of the auditory alert or ATC / AOC voice transcription into their corresponding contextual display system messages. The system will monitor the current cockpit auditory message or announcement, determine the nature of the auditory message, see if the context addition can be done, identify the context message, convert the context into an alert message and tag it to the existing display or graphic element, and provide symbology for replaying the alert message as an auditory alert along with the cascaded message.

[0053] That is, at task 315, speech elements may be extracted from a database and features at task 320 may be received at a feature database to provide a taxonomy and elemental structure for transcription and feature extraction of auditory alerts and communications at task 325. In an exemplary embodiment, natural language processing ("NLP") may be used to process auditory alerts to, for example, convert auditory alerts to text. The converted text will be in the same or consistent structure of the message alert for subsequent aggregation with the message alert to form a cascaded message alert type. At task 330, a task-based monitoring process and context determination (such as determining permissions as an example) are performed. At task 335, a parsing step of the message is performed to determine the context message and the need or how many cascades are necessary. For example, another NLP may be applied during the parsing step with context-based attributes to determine the context of the auditory alert by matching a set of attributes previously derived for a particular aircraft with those stored locally. At task 340, a determination is made to determine whether cascading is required. If not, the process stops at task 345. Alternatively, if cascading is required, a determination is made whether a source has the context of the message at task 350. If the determination is that there is no source with the context of the message, flow stops at task 345. If a determination is made that a source has the context message, flow proceeds to task 355.

[0054] In an exemplary embodiment, the context of the cascaded message alert can be determined and identified by applying the second NLP process in its entirety to the cascaded message alert and then marking the cascaded message alert to associate the marked message with the display element by matching the context of the cascaded message derived by the mark of the marked message through the second NLP process with a specific display element from a set of display elements, wherein the marked message is a cascaded message. At task 355, the context message is converted to a context message for an audible signal using an application solution. At task 360, a monitoring step is performed to ascertain the completion of the current auditory alarm. In other words, a subsequent auditory alarm will not be issued until the current auditory alarm is completed. At task 365, the converted audible information is attached to the message alert. At task 370, the audible signal is marked with a message in the display element, and a symbol adjacent to the message alert and allowing the playback of the complete audio signal is provided on the screen. Therefore, the pilot can select the symbol for the playback of the complete audio signal when viewing the display using the user interface tool. Additionally, the monitoring step at task 360 can include message alerts near the area in which the aircraft is flying by monitoring ATC / AOC communications for auditory or message alerts broadcast or transmitted near the aircraft that are contextually related to the content of ATC / AOC communications for auditory or message alerts that the aircraft has received. Additionally, in task 335, during the parsing step, a determination of the context of auditory alerts that have been broadcast or transmitted near the aircraft can be performed. For example, by applying an NLP process, auditory alerts that are contextually related to auditory or message alerts received by the aircraft can be determined.

[0055] Figure 4 is a first example of an ATC / AOC clearance processed by the exemplary embodiments described herein. Figure 4A use scenario 400 of a cascade of messages on a smart runway system to reduce runway incursions is illustrated. For example, a runway awareness and advisory system ("RAAS") determines that an aircraft 410 is approaching runway 11 at 415 on a path 405 for takeoff or cross. However, the controller has cleared runway 11 for another aircraft (not shown) that is approaching the controller to land. The system has monitored all relevant ATC / AOC, ground controller, internal and external communications and will therefore determine that it is unsafe to cross runway 11. When the RAAS issues an "approaching runway 11" message 425, the system will determine related messages that can be cascaded with the "approaching runway 11" message 420 on the display system. In this case, the system polls the clearance monitoring system and identifies the "runway not cleared" message 425 to be attached based on the context. That is, the "runway not cleared" message is a cascade message from an ATC controller communication 455 that the system determines is contextually related to the aircraft 410 operation approaching runway 11. As soon as the "Approaching Runway 11" message 420 announcement is complete, the system will continue to announce the "Runway Not Cleared" alert message along with the "Approaching Runway 11" message. Therefore, the announced message will be a two-part message concatenated together of the first part of the "Approaching Runway 11" message 420 and the second part of the "Runway Not Cleared" message 425. The two parts of the message will be announced together seamlessly. In addition, the two messages appended together of the "Approaching Runway 11, Runway Not Cleared" message are marked to the Airport Moving Map (AMM) display element using symbology 445 to review and re-hear the cascaded message. Therefore, an option 450 to play back the complete cascade of messages and sources is available to the pilot when selected.

[0056] Figure 5 is a second example of an ATC clearance handled by the exemplary embodiments described herein. Figure 5A use scenario 500 of cascading messages on a smart runway system to reduce runway conflicts is illustrated. For example, when RAAS determines that an aircraft 510 is approaching runway 11 to take off or cross over. However, runway 11 is also partially used as a taxi path and has been cleared by the controller for another aircraft 505 that has just landed on the runway designated RW16R marked 530. The system monitors ATC communications and receives the taxi path of traffic of interest or related to the operation of aircraft 510 and in this case determines that runway 11 marked 515 is safe to cross over or take off but only for the next 2 minutes. When RAAS issues an "approaching runway 11" message, the system determines that this related message "approaching runway 11" can be cascaded 555 with the "approaching runway 11" message 520 on the display system. In this case, the system polls the clearance monitoring system and identifies the "next 1 minute available" message 525 to be attached based on the context with the "approaching runway 11" message 520. As soon as the "Approaching Runway 11" message annunciation is complete, the system annunciates the "Available in Next 1 Minute" message 525 alert along with the displayed alert message in a continuous uninterrupted operation. Additionally, the aggregated message of the two parts of the "Approaching Runway 11 Available in Next 1 Minute" message is annotated to the Airport Moving Map (AMM) display element using symbology 545 to review and re-hear the cascaded message. Thus, an option 550 to play back the complete cascaded message and source is available to the pilot when selected.

[0057] Figure 6 is a third example of an ATC clearance handled by the exemplary embodiments described herein. Figure 6A use scenario 600 of cascading messages on a smart runway system for lift off distance is illustrated. For example, when RAAS determines that an aircraft 610 is aligned on runway three-four left (labeled runway 630) for takeoff. A system monitoring the FMS Takeoff and Landing Data ("TOLD") message and the runway distance announced by the airport can determine if the available runway length is sufficient for a safe takeoff. When the RAAS issues an "on runway three-four left" message, the system will determine related messages that can be cascaded 655 with the "on runway three-four left" message 620 on the display system. In this case, the system polls the clearance TOLD and airport systems and identifies the "3500 feet requiring 6000 feet" message 625 to be attached based on the context. Once the "on runway three-four left" message 620 is completed, the proposed system will continue to announce the "3500 feet requiring 6000 feet" message 620 along with the "on runway three-four left" alert. Additionally, the "3500 feet on runway three-four left requiring 6000 feet" message is annotated to the airport moving map (AMM) display element using symbology 645 to review and re-hear the cascaded message. Thus, an option 650 to play back the complete cascade of messages and sources is available to the pilot when selected.

[0058] Figure 7 is a fourth example of an ATC clearance handled by the exemplary embodiments described herein. Figure 7 Illustrated is a use scenario 700 of a cascade of messages on a smart runway system for braking distance. As aircraft 710 is landing, the system determines that runway 730 is wet and braking is insufficient and appends message 720 after hearing 4000 feet message 725 to increase braking and provides an indication in the display along with a symbol playing that message.

[0059] Those skilled in the art will recognize that the various illustrative logic blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or a combination of the two. Some of the embodiments and implementations are described above with respect to functions and / or logic block components (or modules) and various processing steps. However, it should be appreciated that such block components (or modules) can be implemented by any number of hardware, software, and / or firmware components configured to perform a specified function. In order to clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been generally described above with respect to their functions.

[0060] Such function is implemented as hardware or software depending on the specific application and design restrictions imposed on the whole system. The technician can implement the function in a variable manner for each specific application, but such implementation decision should not be interpreted as causing a deviation from the scope of the present invention. For example, the embodiment of the system or parts can adopt various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, look-up tables, etc., which can implement multiple functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will recognize that the embodiments described herein are only exemplary embodiments.

[0061] The various illustrative logic blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but as an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0062] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein can be specifically implemented directly in hardware, in a software module executed by a controller or a processor, or in a combination of the two. The software module can reside in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. As an alternative, the storage medium can be integrated with the processor. The processor and the storage medium can reside in an ASIC.

[0063] In this document, relational terms such as first and second, etc. may be used merely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Numerical ordinals such as "first", "second", "third", etc. merely express different singles among a plurality and do not imply any order or sequence (unless specifically defined by the claim language). The sequence of text in any of the claims does not imply that the process steps must be performed in a temporal or logical order according to such sequence, unless it is specifically defined by the claim language. The process steps may be interchanged in any order without departing from the scope of the present invention, as long as such interchange does not contradict the claim language and is not logically meaningless.

[0064] In addition, depending on the context, words such as "connected" or "coupled to" used in describing the relationship between different elements do not imply that a direct physical connection must be made between these elements. For example, two elements may be connected to each other physically, electronically, logically, or in any other way through one or more additional elements.

[0065] Although at least one exemplary embodiment has been proposed in the above detailed description of the present invention, it should be recognized that there are a large number of changes. It should also be recognized that one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability or configuration of the present invention in any way. More specifically, the above detailed description will provide a convenient road map for implementing the exemplary embodiments of the present invention for those skilled in the art. It is to be understood that various changes can be made in the function and arrangement of the elements described in the exemplary embodiments without departing from the scope of the present invention as set forth in the appended claims.

Claims

1. A method of monitoring auditory and message alerts received in an Internet of Things (IoT) cockpit of an aircraft during flight and generated by a system within the cockpit, the method comprising: receiving a plurality of alerts including at least one of an auditory alert or a message alert, wherein the auditory and message alerts are from communications including: an internal system of the aircraft, air traffic control (ATC) communications, and internal and external communications of the aircraft; applying a first natural language processing (NLP) process to the aural alert to convert the aural alert into a text alert that is structurally consistent with the message alert, wherein the first natural language processing (NLP) process is applied with context-based attributes to determine at least a context of the aural alert by matching a set of contextual attributes previously derived for the particular aircraft with attributes stored locally therein; Aggregate text alerts with message alerts to form cascaded message alerts; as well as The context of the cascading message alert is identified by applying a second natural language processing (NLP) process to the cascaded message alert as a whole and then marking the cascaded message alert so as to associate the marked message with the display element by matching the context of the cascaded message derived by the second natural language processing (NLP) process from the mark of the marked message with a specific display element from a plurality of display elements, wherein the marked message is the cascaded message.

2. The method according to claim 1, further comprising: Playback of the message is enabled by selection by a user of a particular display element for performing a playback operation with the cascaded message replacing the non-concatenated original message. The method of claim 1 , wherein the particular display element is positioned adjacent to the display of the tagged message.

4. The method according to claim 1, further comprising: monitoring for communications of an auditory or message alert that is contextually related to communications of an auditory or message alert received by an aircraft, wherein the auditory or message alert may be indirectly sent to the aircraft, and Auditory or message alerts received by the aircraft are aggregated for presentation to a user on a display based on the determination of the monitored indirectly transmitted auditory or message alerts.

5. The method according to claim 1, further comprising: Monitoring for communications of audible or message alerts broadcast, transmitted or sent via a data link in an area proximate to the aircraft that are contextually related to the content of communications of audible or message alerts that have been received by the aircraft.

6. The method according to claim 5, further comprising: A context of an audible alert that has been broadcast, transmitted or sent via a data link in an area near the aircraft is determined by applying first and second natural language processing (NLP) processes to determine an audible alert that is contextually related to an audible or message alert received by the aircraft.

7. The method according to claim 6, further comprising: The auditory or message alerts received by the aircraft are cascaded based on a determination of the context of the contextually relevant auditory or message alerts broadcast, transmitted or sent via a data link in an area proximate to the aircraft.

8. The method according to claim 7, further comprising: The auditory or message alert received by the aircraft is appended based on a determination of contextually relevant auditory or message alerts broadcast, transmitted or sent via a data link in an area near the aircraft.

9. A cockpit display system for use on an aircraft, the cockpit display system comprising: Sources of sensory and message alerts; a processor communicatively coupled to a source of sensory and messaging alerts and a cockpit display, the processor configured to process the sensory and messaging alerts to: receiving a plurality of alerts including at least one of a sensory alert or a message alert, wherein the sensory and message alerts are from an internal system of the aircraft, from internal communications, or from external communications; applying a first natural language processing (NLP) process to the sensory alert to convert the sensory alert into a text alert that is structurally consistent with the message alert; aggregating the text alert with the message alert to form a cascaded message alert, wherein a first natural language processing (NLP) process is applied with context-based attributes to determine at least the context of the sensory alert by matching a set of contextual attributes previously derived for the particular aircraft with attributes locally stored therein; as well as The context of the cascading message alert is determined by applying a second natural language processing (NLP) process to the cascaded message alert as a whole and then marking the cascaded message alert to associate the tagged message with the display element by matching the context of the cascade message derived by the second natural language processing (NLP) process from the tag of the tagged message with a specific display element from a plurality of display elements, wherein the tagged message is the cascade message.

10. The display system according to claim 9, further comprising: The processor is further configured to: Playback of the message is enabled by selection by a user of a particular display element that performs a playback operation with the cascaded message replacing the non-concatenated original message.

Images