Systems and / or methods for semantic parsing of air traffic control audio

Abstract

The method S200 can include receiving an audio utterance from an air traffic controller at an aircraft S210, converting the audio utterance to text, determining a command from the text using a question and answer model S240, and optionally regulating the aircraft based on the command S250. The method is for automatically interpreting flight commands from air traffic control (ATC) streams.

Description

[0001] Cross Reference to Related Applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 090,898, filed October 13, 2020, which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0003] The present invention relates generally to the field of aviation, and more particularly to a new and useful semantic parser in the field of aviation. BRIEF DESCRIPTION OF DRAWINGS

[0004] Figure 1 is a schematic representation of a variant of the system.

[0005] Figure 2 is a graphical representation of a variant of the method.

[0006] Figure 3 is a graphical representation of a variant of the method.

[0007] Figure 4 is a graphical representation of an example of training an ASR model in a variant of the method.

[0008] Figure 5 is a graphical representation of an example of training a language model in a variant of the method.

[0009] Figure 6 is a graphical representation of an example of training a question / answer model in a variant of the method.

[0010] Figure 7 is a schematic representation of an example of the system.

[0011] Figure 8 is a graphical representation of an example of a domain expert evaluation tool in a variant of the method.

[0012] Figure 9 is a graphical representation of a variant of the method.

[0013] Figures 10A-10D are graphical representations of first, second, third, and fourth variants of the system, respectively.

[0014] Figures 11A-11C are first, second, and third examples of tree-based query structures, respectively.

[0015] Figure 12 is a graphical representation of a variant of the system and / or method.

[0016] DESCRIPTON OF PREFERRED EMBODIMENTS

[0017] The following description of preferred embodiments of the present application is not intended to limit the application to these preferred embodiments, but rather to enable any person skilled in the art to make and use the application.

[0018] 1. SUMMARY

[0019] The method (examples of which are shown in Figure 2 ) can include performing inference S200 using a system; and can optionally include training system components S100. The method is for automatically interpreting flight commands from air traffic control (ATC) radio communication streams. The method can additionally or alternatively be used for training and / or updating natural language processing systems based on ATC communications.

[0020] Performing inference S200 can include receiving audio utterances from air traffic control at an aircraft S210, converting the audio utterances to a predetermined format S215, determining commands S240 using a question and answer model, and optionally regulating the aircraft based on the commands S250 (examples of which are shown in Figure 3 ). The method is for automatically interpreting flight commands from air traffic control (ATC) streams. The flight commands can be: automatically used to control an aircraft flight; presented to a user (e.g., a pilot, a remote teleoperated robotic arm); relayed to an auto-pilot system in response to a user (e.g., a pilot) confirmation; and / or otherwise used.

[0021] In illustrative examples, the method can receive an ATC audio stream, convert the ATC audio stream to ATC text, and provide the ATC text (as reference text) and a predetermined set of queries (each query associated with a different flight command parameter) to an ATC-regulated question and answer model (e.g., an ATC-regulated BERT) that analyzes the ATC text to obtain query answers. The query answers (e.g., responses of the question and answer model) can then be used to select subsequent queries and / or fill in command parameter values that can be used for direct or indirect aircraft regulation. The ATC audio stream can be converted to ATC text using an ATC-regulated integrated sentence boundary detection and automatic speech recognition model (SBD / ASR model) and an ATC-regulated language model, where utterance hypotheses (e.g., sentence hypotheses, utterances of individual speakers, etc.) can be selected for inclusion in the ATC text based on joint scores from the SBD / ASR model and the language model.

[0022] S200 can be performed using system 100 comprising a speech-to-text module and a question and answer (Q / A) module (e.g., cooperatively forming a semantic parser). The system is used to interpret audio air traffic control (ATC) audio as flight commands, and can optionally regulate an aircraft based on a set of flight commands.

[0023] System 100 is preferably installed to, on, in, and / or configured to operate with any suitable vehicle (e.g., the system can comprise the vehicle). Preferably, the vehicle is an aircraft, but can alternatively be a watercraft, a land-based vehicle, a spacecraft, and / or any other suitable vehicle. The system can be integrated with any suitable aircraft, such as a rotorcraft (e.g., a helicopter, a multicopter), a fixed-wing aircraft (e.g., an airplane), a VTOL, a STOL, a lighter-than-air aircraft, a multicopter, and / or any other suitable aircraft. However, the vehicle can be an autonomous aircraft, an unmanned aerial vehicle (UAV), a manned aircraft (e.g., primary aircraft regulation performed by a pilot, by an unskilled operator), a semi-autonomous aircraft, and / or any other suitable aircraft. Hereinafter, the term "vehicle" can refer to any suitable aircraft, and the term "aircraft" can likewise refer to any other suitable vehicle.

[0024] The system is preferably equipped on an autonomous aircraft configured to regulate the aircraft according to a set of flight commands using a flight processing system without user (e.g., pilot) intervention. Alternatively, the system can be equipped on a semi-autonomous vehicle and / or a human-operated vehicle as a flight aid. In a first variant, the system can display ATC commands to a user (e.g., pilot) and / or relay ATC commands to an autopilot system in response to confirmation by the user (e.g., pilot).

[0025] The term "tune" as referenced with respect to neural networks, language models, or other aspects can be understood to involve adjusting model parameters using training data. Thus, an ATC-tuned network can be understood to have parameters tuned based on ATC audio and / or ATC-specific semantic training data (as opposed to a network specialized for a particular radio frequency band).

[0026] 2. Benefits

[0027] Variants of the technology can provide several benefits and / or advantages.

[0028] First, variations of the system and method can impart higher semantic parsing accuracy than traditional systems by utilizing a multi-query (or repeated question and answer) approach (e.g., through a neural network (e.g., BERT)) because existing deep neural network models have high intrinsic accuracy when responding to these types of questions.

[0029] Second, variations of the technology utilizing a multi-query approach that asks natural language questions of the neural network (e.g., “Expected message for DAL 456?”; “Subject?”; “Heading values?”; etc.) can improve the explainability and / or auditability of the semantic parser. In such variations, when a user rejects a command, the particular module / model / query of the system can be identified as a point of failure, which can be used to further train / improve the system. In some variations, the multi-query approach can additionally enable portions of the semantic parser to train based on partial and / or incomplete labeled responses (e.g., which can be sufficient to answer a subset of the queries used to extract commands from ATC transcripts). For example, training data can be used when a value and / or aircraft tail number is not identified and / or validated in a training dataset.

[0030] Third, variations of the technology can enable semantic parsing of ATC discourse without the use of grammar rules or syntax— which can take a significant amount of time to develop, execute slowly, and produce inaccurate results (particularly when handling edge case scenarios or unusual speech patterns). In an example: ATC controllers and pilots often shorten phrases and / or deviate from standard speech templates as a conversation between an ATC and an aircraft continues— which can severely impact the utility of grammar / syntax-based NLP methods. In variations, the system and / or method can convert unformatted audio, syntactically inconsistent (non-standardized) audio, and / or non-uniform audio data or corresponding ATC transcripts into standardized / formatted data inputs (e.g., as can be accepted / interpreted by certified aircraft processors). In variations, the standardized inputs can be used to certify aircraft systems in a deterministically testable manner. For example, the technology can be used to convert an arbitrary amount of audio signals into a substantially finite set of commands (e.g., with values corresponding to a predetermined set of aircraft command parameters with bounded ranges that can be deterministically tested and / or repeatably demonstrated).

[0031] Fourth, variations of this technology can include approaches that must be rooted in computer technology to overcome specific problems arising in the field of computer networks. In one example, the technology could automatically convert audio into a computer-readable format that can be interpreted by an aircraft processor. In another example, the technology could enable control of partially and / or fully autonomous systems based on communication with an ATC operator. In such examples, the system / method could function as a substitute for an incapacitated pilot (e.g., for a manned aircraft) and / or replace an onboard pilot (e.g., for an unmanned aircraft).

[0032] Fifth, variations of this technology can achieve high-speed and / or high-accuracy natural language processing (NLP) of air traffic control (ATC) utterances by utilizing neural network models (e.g., pre-trained models) pre-trained on other datasets and then tuned to ATC-specific semantics. These ATC-tuned models can improve the system's speed / accuracy in noisy, multi-speaker ATC channels. These ATC-tuned models also preserve a broad "common sense" understanding of pre-existing models and avoid overly biasing the system towards conventional ATC language, thus enabling the system to effectively respond to edge cases or speech patterns that are not frequently encountered in ATC communications.

[0033] However, variations of this technology may additionally or alternatively provide any other suitable benefits and / or advantages.

[0034] 3. System

[0035] System 100 ( Figure 1 An example of system 100 shown in the figure may include: a voice-to-text module 120 and a question and answer (Q / A) module 130 (e.g., a collaborative "semantic parser"). The system may optionally include a communications subsystem 110 and a flight processing system 140. However, system 100 may additionally or alternatively include any other suitable set of components. System 100 is used to determine flight commands 106 based on audio input 102 (e.g., received ATC radio transmissions), which may be used for vehicle guidance, navigation, and / or control.

[0036] The audio input 102 can include a single utterance (e.g., a statement), multiple utterances (e.g., over a predetermined window (e.g., 30 seconds), within a continuous audio stream, over a rolling window), periods of silence, a continuous audio stream (e.g., over a particular radio channel, such as based on a current aircraft location or a dedicated ATC communication channel), and / or any other suitable audio input. In a first example, the audio input can be provided as a continuous stream. In a second example, a continuous ATC radio frequency stream can be stored locally, and a rolling window of a particular duration (e.g., the last 30 seconds, a dynamic window sized based on a previous utterance detection, etc.) can be analyzed from the continuous radio frequency stream.

[0037] The audio input is preferably in the form of a digital signal (e.g., via an A / D converter and / or a radio transmission of a wireless communication chipset), however can be in any suitable data format. In a particular example, the audio input is a radio stream in digital format from an ATC station. In a variation, the system can receive radio communications directly from an ATC tower and translate the communications into commands that can be interpreted by the flight processing system. In a first "human in the loop" example, a user (e.g., a pilot in command, a non-skilled operator, a remote moderator, etc.) can confirm and / or validate the commands before the commands are sent to and / or executed by the flight processing system. In a second "autonomous" example, the commands can be sent to and / or executed by the flight processing system without the direct involvement of a human. However, the system 100 can otherwise appropriately determine commands from the audio input.

[0038] The system 100 is preferably mounted to, mounted on, integrated into, and / or configured to operate with any suitable vehicle (e.g., the system can include the vehicle). The system 100 is preferably specific to a vehicle (e.g., the module is specifically trained for the vehicle, the module is trained on a vehicle-specific dataset), however can be generic across multiple vehicles. The vehicle is preferably an aircraft (e.g., a cargo plane, an autonomous aircraft, a passenger plane, a manually piloted aircraft, a manned aircraft, an unmanned aircraft, etc.), however can alternatively be a watercraft, a land-based vehicle, a spacecraft, and / or any other suitable vehicle. In a specific example, the aircraft can include exactly one pilot / PIC, where the system can be used as a backup or failsafe in the event of the sole pilot / PIC incapacitation (e.g., an autonomous co-pilot, enabling remote verification of aircraft regulations, etc.).

[0039] The system 100 can include any suitable data processors and / or processing modules. Data processing for various system and / or method elements is preferably local on the aircraft, but can additionally or alternatively be distributed among remote processing systems (e.g., for primary and / or redundant processing operations) (e.g., at a remote validation site, at an ATC data center, on a cloud computing system, and / or at any other suitable location). Data processing for the speech-to-text module and the Q / A module can be centralized or distributed. In a particular example, data processing for the speech-to-text module and the Q / A module can occur at a processing system separate from the flight processing system (e.g., not performed by the FMS or FCS processing system; the speech-to-text module and the Q / A module can be decoupled from the FMS / FCS processing; Figure 12 are shown in the first variation, but can additionally or alternatively occur at the same computing node and / or within the same (certified) aircraft system. Data processing can be performed at redundant endpoints (e.g., redundant on-board / aircraft endpoints), or can be singular for various instances of the system / method. In a first variation, the system can include a first natural language processing (NLP) system including a speech-to-text module and a Q / A module that can be used with a second flight processing system including a flight processing system and / or a communication system (e.g., an ATC radio). In a second variation, the aircraft can include a unified “on-board” processing system for all runtime / inference processing operations. In a third variation, remote (e.g., cloud) processing can be used for speech-to-text operations and / or Q / A response generation. However, the system 100 can include any other suitable data processing system / operations.

[0040] The system 100 can optionally include a communication subsystem for converting ATC communications (e.g., radio signals) into audio input that can be processed by the ASR module. Additionally or alternatively, the communication subsystem can be configured to transmit responses to the ATC. The communication subsystem can include antennas, radio receivers (e.g., ATC radio receivers), radio transmitters, A / D converters, filters, amplifiers, mixers, modulators / demodulators, detectors, wireless (radio frequency) communication chipsets, and / or any other suitable components. The communication subsystem includes ATC radios, cellular communication devices, VHF / UHF radios, and / or any other suitable communication devices. In a particular example, the communication subsystem is configured to perform S210. However, the communication subsystem can include any other suitable components, and / or otherwise suitably establish communication with air traffic control (ATC).

[0041] The speech-to-text module of the system 100 is used to convert audio input (e.g., ATC radio signals) into speech hypotheses 104 in the form of text (e.g., ATC transcripts) and / or alphanumeric characters. The speech hypotheses are preferably a stream of text (e.g., a dynamic transcript), but can alternatively be a text document (e.g., a static transcript), a string of alphanumeric characters (e.g., ASCII characters), or in any other suitable human-readable and / or machine-readable format. The speech-to-text module is preferably onboard the aircraft, but can additionally or alternatively be remote. The speech-to-text module is preferably an ATC-tuned speech-to-text module that includes one or more models pre-trained on ATC audio data, but can additionally or alternatively include one or more general-purpose models / networks and / or models / networks pre-trained on generalized training data (e.g., natural language speech not associated with ATC communications).

[0042] The speech-to-text module can include an integrated automatic speech recognition (ASR) module 122, a sentence boundary detection (SBD) module 124, a language module 126, and / or other modules and / or combinations thereof. In a particular example, the speech-to-text module can include an integrated ASR / SBD module 125. The speech-to-text module (and / or its sub-modules) can include a neural network (e.g., a DNN, CNN, RNN, etc.), a cascade of neural networks, a combination network, a Bayesian network, a Markov chain, predetermined rules, a probability distribution, an attention-based model, a heuristic algorithm, a probabilistic graphical model, or other models. The speech-to-text module (and / or its sub-modules) can be a tuned version of a pre-trained model (e.g., pre-trained using different training data for another domain or use case), a trained version of a previously untrained model, and / or otherwise constructed.

[0043] In variations, a sub-module of a speech-to-text module (e.g., ASR module and / or SBD module) can ingest an audio input (e.g., audio stream, audio clip) and generate a set of language hypotheses (e.g., weighted or unweighted), which can be used as an intermediate data format, e.g., which can be used to audit the speech-to-text module, audit sub-modules / models therein, and / or select a single utterance hypothesis. The set of language hypotheses can include overlapping / alternative hypotheses for an audio segment, or can be singular (e.g., a single hypothesis for a separate audio segment or time period). The set of language hypotheses can include: utterance hypotheses (e.g., utterance hypothesis candidates), letters, word-segment streams, phonemes, words, sentence fragments (e.g., in text format), word sequences (e.g., phrases), sentences, speaker changes, utterance breaks (e.g., starts, stops, etc.), and / or any other suitable hypotheses. In variations in which an audio stream includes multiple speakers / utterances, the set of language hypotheses can additionally include utterance boundary hypotheses, which can distinguish between multiple speakers and / or identify the start and termination of utterances, with associated weights and / or speaker hypotheses (e.g., identify a label of a particular speaker, identify a label of a particular aircraft / tower). Additionally or alternatively, the utterance boundary hypotheses can identify utterance boundaries and / or changes in speakers without identifying individual speakers. Each language hypothesis preferably includes a relevant weight / score associated with the utterance (and / or utterance boundary), which is assigned according to a relative confidence (e.g., statistical; e.g., determined using an ASR model, SBD model, and / or language model; etc.). The set of language hypotheses is preferably ordered, sequential, and / or time-stamped in association with a time of receipt, but can be otherwise suitably related.

[0044] However, the speech-to-text module can generate, store, and / or output any other suitable set of hypotheses. For example, the language hypotheses can include multiple utterance hypotheses, where a single utterance hypothesis can be selected based on the set of generated utterance hypotheses. As a second example, a subset (e.g., complete set) of language hypotheses with corresponding weights / scores can be output by the speech-to-text module.

[0045] The speech-to-text module can include an ASR module for extracting language hypotheses from the audio input. Using the audio input, the ASR module can determine a sequence of language hypotheses, e.g., letters, token streams, phonemes, words, sentence fragments (e.g., in text format), word sequences (e.g., phrases), sentences, and / or any other suitable language hypotheses (e.g., with corresponding weights). The ASR module is preferably a neural network (e.g., Wav2Letter, Kaldi, Botium, etc.), but can alternatively be any other suitable model. In an example, a pre-trained neural network can be fine-tuned for ATC audio and / or trained using ATC audio (e.g., with associated transcripts). In a second example, the ASR module can include an ASR model trained by S110 and / or S120. In a particular example, the ASR module is configured to perform S220 of the method. The ASR module can optionally include an integrated SBD module. In variations where the ASR module outputs low-level language components (e.g., phonemes, speech, etc.), the system can optionally include a secondary transformation module (e.g., phoneme-to-word transformation) that converts the low-level language components to language components compatible with the language module and / or other system modules.

[0046] The speech-to-text module can include an SBD module for identifying utterance boundaries and / or speaker changes for multi-utterance audio input. Using the audio input, the SBD module can determine a sequence of language hypotheses, e.g., utterance boundary hypotheses, speaker hypotheses (e.g., identifying a label for a particular speaker, identifying a label for a particular aircraft / tower), and / or any other suitable hypotheses. The SBD module is preferably integrated with the ASR module (as shown in the example in Figure 10A Figures 10C-10D Figure 10B ​​The SBD module is preferably a neural network (e.g., Wav2Letter, Kaldi, Botium, etc.), but can alternatively be any other suitable model. In an example, a pre-trained SBD neural network can be adjusted for ATC audio and / or trained using ATC audio (e.g., with associated transcripts). In a second example, the SBD neural network can be trained separately from the ASR module (e.g., using a different training set, using a training set that includes periods of radio silence and / or audio artifacts, etc.). In a third example, the SBD model can be adjusted for ATC audio and / or trained using ATC audio, e.g., trained to recognize silent speakers and / or utterance boundary characters (e.g., transitional speakers, transitional audio artifacts). However, the speech-to-text module can include any other suitable SBD module.

[0047] The language module of the speech-to-text module is used to select an utterance hypothesis based on a set of language hypotheses, which can then be passed into the Q / A module. The language module receives a set of language hypotheses (e.g., a set of phonemes, words, sentences, etc.) from the ASR module and returns an utterance hypothesis (e.g., a sentence, a series of language hypotheses, etc.) associated with a single utterance. The language module preferably determines the utterance hypothesis solely from the language hypotheses, but can alternatively or additionally ingest audio input and / or other auxiliary data. The auxiliary data can include: aircraft ID, contextual information (e.g., vehicle state, geographic location, ATC control tower ID and / or location, etc.), weather data, and / or any other suitable information. The utterance hypothesis is preferably text (e.g., a string of text or an utterance transcript), but can alternatively be a set of phoneme indices, audio, or any suitable data format.

[0048] The language module preferably selects the utterance hypothesis from the set of language hypotheses by weighting the likelihood of various “sound-based” language interpretations in the context of the entire utterance and / or ATC language mode. In a first variant, the language module assigns a language weight / score to each utterance hypothesis using a neural network language model tuned for ATC language (e.g., LSTM network, CNN, FairSeq ConvLM, etc.) (e.g., a neural network trained using ATC transcripts, etc.; e.g., a language model trained according to S140). In a second variant, the language module assigns a language weight / score according to a grammar-based language model (e.g., according to a set of heuristic algorithms, grammar rules, etc.). In a third variant, the language module can be tightly integrated with the ASR module. In examples, the language model can be used during search, during a first pass, and / or during re-ranking. However, the language module can assign weights / scores in any other suitable manner. In particular examples, the language module is configured to perform S230 of the method.

[0049] In examples, the speech-to-text module transforms the ATC audio stream into a natural language text transcript provided to the Q / A module, preserving the syntax (e.g., arbitrary, inconsistent, non-uniform syntax) as conveyed by the ATC speaker.

[0050] Alternatively, the speech-to-text module can include a neural network trained (e.g., using audio data labeled with audio transcripts) to output utterance hypotheses (e.g., one or more series of linguistic components separated by utterance boundaries) based on audio input. However, the speech-to-text module can include only an automatic speech recognition module, only a language module, and / or otherwise be structured.

[0051] However, the system can include any other suitable speech-to-text module.

[0052] The system 100 can include a question and answer (Q / A) module (as shown in the example in Figure 7 The Q / A module is preferably configured to receive utterance hypotheses in textual form from the speech-to-text module, but can alternatively receive audio and / or any other suitable input.

[0053] The Q / A module preferably includes one or more Q / A models (e.g., BERT, BERT conditioned on ATC applications, etc.), but can additionally or alternatively include classifiers or other models. The Q / A models are preferably pre-trained language models conditioned on ATC transcripts, but can be untrained or have another format. The Q / A models can be: convolutional neural networks, (pre-trained) large neural language models, Bidirectional Encoder Representations from Transformers (BERT), Generative Pretrained Transformers (GPT), and / or any other suitable language model. However, the Q / A module can include any other suitable neural language model.

[0054] The Q / A module preferably answers a set of flight command queries (e.g., natural language queries). The flight command queries are preferably predetermined (e.g., manually determined, extracted from command templates, etc.), but can be dynamically determined. The flight command queries are preferably semantic queries in human-readable format, but can additionally or alternatively be provided in machine-readable format. The command queries are preferably in natural language (“reading comprehension”), but can alternatively be vectors, tensors, and / or have another format. The set of flight command queries are preferably organized in a hierarchical structure (e.g., with parent-child query relationships), but can alternatively be organized in a serial structure, or otherwise. The flight command queries can be organized in lists, trees, or otherwise. In variants, the flight command queries can be provided as a sequence / series of chained nodes (shown in Figures 11A-11C variants), each node corresponding to a predetermined query, with the nodes including a set of independent nodes and a set of dependent nodes, each dependent node linking to a specific answer / response (e.g., a specific answer value) of a more broad / higher level parent semantic query (e.g., with a limited set of answers or a closed range of answers). Thus, the relevant query can be triggered in response to determining a predetermined answer at a linked node at a higher level. Alternatively, the set of predetermined flight command queries can be provided synchronously or asynchronously in any suitable combination / arrangement of series and / or parallel.

[0055] Command queries can be configured to have binary answers (e.g., "yes," "no"), discrete answers (e.g., letters, integers, etc.), continuous answers (e.g., coordinate values, etc.), and / or any other suitable type of answer value. Different types of commands can have different query structures. For example, a high criticality query (e.g., an aircraft identifier) can be structured as a binary query. In another example, a property with multiple potential answers can be structured as an open-ended question (e.g., "topic?") rather than a binary question (e.g., "does the utterance include heading?" "does the utterance include altitude?"). However, queries can be structured in other ways. Examples of command queries include: whether an aircraft is the intended recipient of an utterance assumption, what or whether a command parameter or topic (e.g., heading, altitude, etc.) is included in an utterance assumption, what or whether a command parameter value (e.g., altitude direction, altitude level, etc.) is included in an utterance assumption, and / or other queries. In a first example, the Q / A module determines that the utterance is intended for the aircraft (e.g., question: "is intended for DAL 456?"; answer: "yes"). In a second example, the Q / A module determines the topic of the utterance (e.g., question: "topic?"; answer: "heading, altitude"). In a third example, the Q / A determines the values associated with the topic of the utterance (e.g., question: "altitude values?"; answer: direction: down, level: 2000"). In examples, the Q / A module can be configured to perform S240.

[0056] Based on the queries, the Q / A module outputs a set of flight commands, which can include guidance commands (e.g., navigation instructions; sequence of waypoints, approach to a landing point, etc.), vehicle state commands (e.g., instructions to modify a vehicle state parameter, increase altitude to 5000 feet, etc.), effector state commands (e.g., effector instructions; deploy landing gear, etc.), flight path commands (e.g., trajectory between waypoints, etc.), and / or any other suitable commands. The commands are preferably output in a prescribed format, such as a standardized human-readable format (e.g., to allow for human verification) and / or a machine-readable format (e.g., to allow for human interpretation / verification of the commands), based on the answers generated by the Q / A module. In a particular example, the commands can be provided as the union of the answers to the command parameter identification query and the answers to at least one command parameter value query (e.g., corresponding to the answers to the command parameter identification query). In a second example, the commands can be directly recognized as the combination of each answer / response as generated by the Q / A module. The output commands are preferably text-based and / or alphanumeric, but can be otherwise suitably provided (e.g., text-to-speech verification, etc.). In some variations, the commands can be post-processed according to any suitable heuristic algorithm, grammatical rule, or formatting protocol, but can otherwise be provided as output from the Q / A module directly to the pilot and / or flight processing system. In a particular example, the Q / A module can convert the utterance hypotheses into commands in a standardized data format (e.g., as can be accepted / interpreted by a certified aircraft processor). In variations, the commands can include a substantially limited set of command parameters corresponding to a predetermined set of topics (e.g., altitude, heading, etc.). Additionally, the command parameters can be within a substantially limited and / or bounded range (e.g., heading limited to compass directions, altitude limited to physical aircraft constraints, command coordination limited to flight envelope, etc.). However, the command parameters can additionally or alternatively be arbitrary, unbounded, and / or substantially unconstrained. However, the Q / A module can generate any other suitable commands.

[0057] However, the system can include any other suitable Q / A module.

[0058] System 100 may optionally include and / or be used in conjunction with a flight processing system for controlling various effectors of the aircraft on command. The flight processing system may include an aircraft flight management system (FMS), a flight control system (FCS), a flight guidance / navigation system, and / or any other suitable processor and / or control system. The flight processing system can control flight effectors / actuators during normal operation, takeoff, landing, and / or continuous flight of the vehicle. Alternatively, the flight processing system may be configured to implement conventional manual flight control in a flight-assisted configuration. The system may include a single flight processing system, multiple (e.g., three) redundant flight processing systems, and / or any other suitable number of flight processing systems. The flight processing system may be located on the aircraft, distributed between the aircraft and remote systems, remote from the aircraft, and / or otherwise appropriately distributed. In a particular example, the flight processing system is configured to perform S250.

[0059] In a variant, the flight processing system can be configured (e.g., certified) to accept only a predetermined set of command inputs and / or inputs with a predetermined format, wherein the output of the Q / A model is provided in a predetermined format and / or a subset of the predetermined set of commands.

[0060] However, the system may include any other suitable components and / or be otherwise appropriately configured to perform the method S200.

[0061] 4. Methods

[0062] method( Figure 2 An example of the method is shown below. It may optionally include a training system component S100; and the use of system S200 to perform inference. The method is used to automatically interpret flight commands from air traffic control (ATC) radio communication streams. The method may additionally or alternatively be used to train and / or update a natural language processing system based on ATC communications.

[0063] 4.1 Training

[0064] Training system component S100 ( Figure 9 The example shown is used to generate an ATC conditioning system capable of interpreting ATC audio signals as flight commands. S100 may include training a speech-to-text model and training a question-and-answer (Q / A) model S150. S100 may optionally include generating an extended ATC transcript S130. However, training the semantic parser S100 may include any other suitable elements. S100 is preferably performed offline and / or by a remote computing system, but may alternatively be performed on the aircraft (e.g., locally, during flight, or asynchronously with the aircraft's flight).

[0065] The training speech-to-text model is used to generate a transcript model specific to ATC communications, taking into account ATC-specific grammar, lexicon, speech patterns, and other characteristics. Training the speech-to-text model can include training an ASR model S110, training an SBD model S120, training a language model S140, and / or any other suitable element. Training can include adjusting network weights, re-determining weights, and / or otherwise training the network. Training (and / or inference) can utilize gradient-based methods (e.g., stochastic gradient descent), belief propagation (e.g., sum-product message passing; max-product message passing, etc.), and / or any other suitable training method.

[0066] The training automatic speech recognition (ASR) module S110 is used to train a neural network to recognize natural language in ATC communications. The ASR model is preferably trained (e.g., using supervised training, semi-supervised training) from a pre-existing ASR model (e.g., Wav2Letter), and can be “tuned” by providing the neural network with a mix (e.g., 50 / 50, 60 / 40, 70 / 30, a predetermined mix, 100 / 0, etc.) of ATC training audio with corresponding ATC transcripts and original training data (e.g., from the pre-existing model). In Figure 4 An example is shown in FIG. 1. ATC training audio with transcripts is preferably determined manually (e.g., by a human, by a domain expert), but can be validated / audited ATC communication audio / transcripts (e.g., generated from an existing ASR model), and / or otherwise determined. The ATC training audio can include a single utterance, multiple utterances, a stream of radio communications over an ATC communication channel, and / or any other suitable training audio. Preferably, utterances (e.g., statements, sentences, etc. from individual speakers) are individually associated with transcripts as part of the training data. However, the ASR model can be trained for ATC speech recognition in other ways.

[0067] The training sentence boundary detection (SBD) module S120 is used to train the speech-to-text module to recognize utterance boundaries (e.g., sentence segment boundaries, sentence boundaries). S120 can optionally train the speech-to-text module to distinguish between unique utterances and / or utterances from different speakers / entities. S120 can train an existing ASR model (e.g., as determined in S110, which generates an integrated ASR / SBD model) or a separate model to generate an SBD module. Preferably, the SBD model can be trained using time length concatenated audio including a series of multi-utterances and silencing periods (e.g., periods of non-speaking) in between, and associated multi-utterance training transcripts. The ATC audio and transcripts used to train the SBD model can be the same as and / or different from the ASR model.

[0068] The multi-utterance training transcript preferably includes boundary annotations (e.g., with unique boundary characters or other identifiers; using “ / ” or “%” characters; etc.) that can delineate unique speakers, unique utterances, breaks between utterances, periods of silence, audio artifacts (e.g., “squelch” when an ATC speaker starts and / or starts broadcasting), and / or any other appropriate boundaries. The boundary annotations are preferably added automatically during transcript stitching, but can be inserted manually, determined from the audio, and / or added in other ways.

[0069] In a particular example, the ASR model is trained by assigning unique “silence speakers” and / or unique “transition speakers” in the audio and / or transcript, which is particularly advantageous in SBD for ATC radio communications, which often exhibit a characteristic radio “squelch” sound before an utterance. By assigning these audio segments to unique “transition speakers” (or “squelch speakers”), the SBD model can more accurately distinguish back-to-back utterances (e.g., with minimal intervening silence), which often occurs in noisy ATC radio channels.

[0070] However, the SBD model can be trained in other ways.

[0071] Training language model S140 is for training a language model to distinguish ATC language patterns. In variants, the language model can determine whether a transcript is correct / logically consistent in context (e.g., based on ATC grammar, etc. is syntactically correct), determine a language / syntax score of a transcript, and / or otherwise determine whether a transcript makes sense. Preferably, S140 conditions a pre-existing language model (e.g., a convolutional neural network, a FairSeq ConvLM, etc.), but can alternatively train an untrained language model. Existing language models can be conditioned on ATC transcripts, which can be single-speech ATC transcripts, multi-speech ATC transcripts, and / or boundary-annotated ATC transcripts (e.g., such as those used to train the SBD model in S120), although the language model can be trained using any suitable ATC transcripts. S140 preferably does not train on ATC audio, but can alternatively train on ATC audio. In variants, the language model can be trained using entity-labeled ATC transcripts that identify ATC-specific entities in the transcripts. The labeled entities can include: carriers, aircraft, waypoints, airports, numbers, directions, and / or any other suitable entities. The entity labels can be assigned manually, automatically (e.g., unsupervised), with a semi-supervised HMM labeler (e.g., using a domain expert evaluation tool, etc.), and / or in any other suitable manner. Individual words or phrases that appear in a transcript can be assigned to multiple entities depending on the context in which they appear (i.e., the entity label dictionary can include multiple phonetically and / or lexically colliding entities that sound and / or spell the same). In an example, “Southwest” can be labeled (and / or passed) as a direction or a carrier, depending on the context in which it appears. Similarly, in a second example, “delta” can be labeled as an aircraft name (e.g., DAL 456 = “delta alpha lima four five six”), a part of a carrier, and / or not labeled (e.g., a change in value or parameter), depending on the context in which it appears. In a third example, “Lima” can be an airport, a waypoint, a part of an aircraft name, and / or otherwise labeled. In a fourth example, waypoints can sound substantially the same (e.g., “ocean”), while corresponding to different waypoint entities depending on the context in which they appear. However, the language model can be trained with any other suitable transcripts and / or information.

[0072] In variations, the portion of training text provided for training the language model is the same as the portion of training text used to initially train the pre-existing language model (e.g., FairSeq ConvLM). Thus, the language model can be “tuned” by providing the neural network with a mix (e.g., 50 / 50, 60 / 40, 70 / 30, a predetermined mix, etc.) of ATC training transcripts and original training data (e.g., from the pre-existing model). However, the language model can be trained in other ways for ATC language patterns.

[0073] S100 can optionally include generating augmented ATC transcripts S130 (e.g., synthetic transcripts) that are used to expand the number / quantity of ATC training transcripts available for training the language model in S140, an example of which is illustrated in Figure 5 In variations, this can be beneficial in order to provide training transcripts specific to areas / regions for which the entity is known (e.g., airport names, waypoints, carriers, etc.) but for which ATC transcripts are not available. Additionally or alternatively, S130 can improve the accuracy of the language model by increasing the size of the training data set (e.g., the number of available speech transcripts). S130 preferably replaces the value of the tagged entity (e.g., the value of the tagged entity within the ATC transcript of the entity tag) with a different entity value from the ATC entity dictionary. The ATC entity dictionary can be generated manually, generated by a domain expert (e.g., a pilot), generated randomly (e.g., numbers replaced), generated using historical flight logs, an aircraft database, an airport database, randomly generated, and / or otherwise generated. In variations, the augmented ATC transcripts can preferentially (e.g., at a higher rate; at a higher frequency; occur more than a threshold number of instances - e.g., 3 or more in the training set) replace phonetically and / or lexically conflicting entity names (e.g., entity names identified by multiple tags in different contexts), such as “southwest” and “delta.” The augmented ATC transcripts can then be used to train the language model in S140 and / or the question and answer model in S150 (e.g., an example of training an ATC-tuned language model is illustrated in Figure 5

[0074] However, the ATC transcripts can be generated in other ways. Alternatively, the system (and / or the neural network models therein) can be trained entirely with real ATC communication transcripts.

[0075] ​S100 can include a training questions and answers (Q / A) module S150 for training a model to answer ATC-specific queries. S150 preferably includes fine-tuning a pre-trained language model, but can include training an untrained model. The language model can be trained using: ATC transcripts, associated resolved meanings (e.g., reference outputs; answers to queries; values of command parameters determined from ATC transcripts, etc.), command query sets, and / or other data. In variants, S150 can also provide language model context information related to a particular utterance— e.g., a specific aircraft’s tail number or carrier, the aircraft’s flight plan, a set of utterance transcripts preceding the particular utterance, and / or any other suitable context information.

[0076] The text transcripts used to train the Q / A model can be the same ATC transcripts used to train the ASR and / or SBD models, the same ATC transcripts (and / or augmented ATC transcripts) used to train the language model, the utterance hypotheses output by the speech-to-text module, and / or other transcripts. However, any suitable ATC transcripts can be used to train the Q / A model.

[0077] The resolved meanings used to train the Q / A model can be: manually determined, manually reviewed by a domain expert, provided by a grammar semantic parser (e.g., SEMPRE, a parser less accurate than SEMPRE, a previous iteration of the system, etc.; an example is shown in Figure 6 FIG. 6) and / or otherwise suitably determined.

[0078] In a particular example, a grammar semantic parser resolves command parameter values from ATC transcripts, where the resolved values (e.g., command hypotheses), source transcripts, optional ATC audio, and / or other data are used in a domain evaluation tool (e.g., a simulator) to determine answers to queries. The answers are used to train the Q / A model. Figure 8The reference output (shown in the middle illustration) is presented to a domain expert. The domain expert can: label the model output (e.g., as “correct,” “incomplete,” “incorrect,” etc.), correct the resolved values, and / or otherwise interact with the resolver output. In variations, reference output labeled as “incorrect” and / or “incomplete” can be examined and used to update or improve the grammar rules of the grammar semantic resolver. In variations, reference output labeled as “incorrect” is not used to train the Q / A model, but can alternatively be used to train the Q / A model (e.g., “incorrect” labels are used to train by counterexample). In variations, reference output labeled as “correct” and / or “incomplete” can be passed into the Q / A model during S150. In variations, incomplete label data can be used to train a subset of queries associated with a particular utterance (e.g., based on the correctly labeled portion of the transcript). For example, where a parameter value can be un-labeled and a topic has been identified, the topic can be used to train a command identification (e.g., “topic?”) query. Likewise, where an aircraft tail number is labeled / identified, the incomplete label data can be used to train a plane-specific speaker identification query. Labels can be used in other ways, however, and model output can be determined appropriately in other ways.

[0079] However, the question and answer model can be trained in other ways as appropriate.

[0080] In variations, the ASR model, SBD model, language model, and / or Q / A model can optionally be retrained and / or updated based on pilot / PIC validation at any suitable update frequency. The models can be updated / retrained independently, synchronously, asynchronously, periodically (e.g., at a common update frequency, different frequencies), never (e.g., where deterministic models are certified, which can be desirable), and / or can be updated or trained as appropriate in other ways. The models can be updated periodically locally, on the aircraft, through remote / cloud (push) updates, and / or can be updated / retrained as appropriate in other ways.

[0081] In variations, the models can be audited based on pilot rejection of the final output parameters in order to locate sources of error within the data pipeline (e.g., as part of a root cause analysis), which can be used as training input to improve the network. For example: an erroneous intermediate parameter (e.g., in an utterance hypothesis or language hypothesis) can result in incorrect output by the Q / A module, even where the Q / A module performed correctly. In variations, the output of each model / module can additionally be audited against formatting templates specified for each step (e.g., to enable certification compliance of the system). The system and / or various subcomponents can be audited in other ways as appropriate, however.

[0082] However, the system components can be suitably trained in other ways.

[0083] 4.2 Runtime / inference

[0084] S200 can include receiving audio utterances at an aircraft from air traffic control S210, converting the audio utterances into a predetermined format S215, determining commands using a question and answer model S240, and regulating the aircraft based on the commands S250. However, the method S200 can additionally or alternatively include any other suitable elements. S200 is used to automatically interpret flight commands from air traffic control (ATC) streams. The flight commands can be used to automatically regulate the aircraft flight; presented to a user (e.g., a pilot, a remote teleoperated robotic arm); relayed to an autopilot system in response to user (e.g., pilot) confirmation; and / or used in other ways.

[0085] All or part of S200 can be performed continuously, periodically, sporadically, in response to a transmission received over the radio, during an aircraft flight, in preparation for a flight, and / or after a flight, at all times, and / or at any other timing. S200 can be performed in real-time or near real-time, or asynchronously with respect to an aircraft flight or audio utterance reception. S200 is preferably performed on the aircraft, but can alternatively be performed partially or entirely remotely.

[0086] Receiving audio utterances from air traffic control S210 is used to receive a communication signal at an aircraft and / or convert the communication signal into an audio input that can be processed by an ASR module. In a particular example, S210 converts an analog radio signal into a digital signal using an A / D converter (and / or other suitable wireless communication chipset), and sends the digital signal to an ASR module (e.g., via a wired connection) as an audio input. S210 preferably monitors a single radio channel (e.g., associated with a particular aircraft), but can alternatively scan multiple channels (e.g., to collect a larger amount of ATC audio data). However, S210 can suitably receive utterances in other ways.

[0087] Converting the audio utterances into a predetermined format S215 is used to generate transcripts from the ATC audio. This can be performed by a speech-to-text module or other system component. Converting the audio utterances into a predetermined (e.g., textual) format can include determining a set of utterance hypotheses for the utterances S220, and selecting an utterance hypothesis from the set of utterance hypotheses S230; however, the ATC audio can be converted in other ways.

[0088] The utterance hypothesis set S220 for an utterance is used to identify audio patterns (e.g., such as letters, phonemes, words, short phrases, etc.) within the utterance. In particular examples, S220 can be performed by a speech-to-text module, an ASR module (and / or ASR models therein), an integrated ASR / SBD module (e.g., with integrated ASR / SBD models therein), a language module, and / or combinations thereof. S220 can optionally include assigning a weight or score to each audio pattern (also referred to as a language hypothesis) using the ASR module and / or other modules. An utterance hypothesis can be: a language hypothesis, a set of language hypotheses, and / or any other suitable hypothesis.

[0089] In a first variant, the ASR and / or integrated SBD / ASR module generates a set of language hypotheses, where the language module receives the language hypotheses and generates a score (e.g., an ASR score; same or different than the language weight / score) for each string or sequence of language hypotheses. One or more sets of language hypotheses can be generated from the same audio clip. The SBD / ASR module can also output a score (ASR score or ASR weight) for each language hypothesis, sequence of hypotheses, and / or set of language hypotheses. However, the utterance hypothesis set can be determined in other ways.

[0090] The utterance hypothesis S230 is selected from the utterance hypothesis set S230 for detecting a language pattern from the set of language hypotheses in the context of the entire utterance. Additionally or alternatively, S230 can be used to select the highest probability string / sequence of language hypotheses as the utterance hypothesis. S230 can be performed by the language module, the Q / A module, and / or another module.

[0091] In a first variant, the language module can select the string or sequence of language hypotheses with the highest combined language weight (or score) and ASR weight (or score) as the utterance hypothesis.

[0092] In a second variant, the outputs of multiple modules are cooperatively used to select the utterance hypothesis. For example, the utterance hypothesis with the highest combined hypothesis score and / or largest hypothesis weight cooperatively determined by the language model and the integrated ASR / SBD model is selected. In a first example, the utterance hypothesis that maximizes the language weight multiplied by the ASR weight of the utterance is selected. In a second example, the hypothesis that maximizes the sum of the language score and the ASR score of the utterance is selected.

[0093] However, the utterance hypothesis can be selected in other ways.

[0094] Determining commands from utterance hypotheses using question and answer model S240 is used to extract flight commands from the utterance hypotheses, which can be interpreted and / or implemented by the flight processing system. S240 is preferably performed by one or more instances of the Q / A module, but can be performed by another component. S240 is preferably performed using a set of flight command queries and the utterance hypotheses, but can be performed in other ways.

[0095] S240 can include providing the Q / A module with a set of command queries in addition to the utterance hypothesis as input, where the Q / A module answers the command queries using the utterance hypothesis as reference text. In a first embodiment, the queries are provided serially, where successive queries are determined based on previous answers. The series of queries can be determined from a command query set structure (e.g., a list, tree, etc.), randomly determined, or otherwise determined. In a particular example, S240 includes querying for the presence of a topic within the utterance hypothesis, and then only querying for the value of the topic confirmed within the utterance. In a second particular example, S240 includes initially determining whether the aircraft (and / or pilot) is an intended recipient of the utterance (associated with the utterance hypothesis), and only further querying whether the utterance is intended for the aircraft / pilot (e.g., utterances not intended for the aircraft / pilot are ignored and / or any commands therein are not passed to the flight processing system; utterances corresponding to transition speaker detection can be ignored; etc.). Alternatively, the Q / A model (or different versions or instances thereof) can be queried with multiple parallel queries, or can be queried in other ways.

[0096] In a second variant, the Q / A module includes pre-embedded queries, where the Q / A module answers a predetermined set of questions based on the utterance hypothesis. For example, the Q / A module can be a multi-class classifier that outputs a value determined from the utterance hypothesis for each of a set of "classes," where each class represents a command parameter. However, S240 can determine command parameter values in other ways as appropriate.

[0097] S200 can optionally include regulating the aircraft based on the commands S250, which is used to modify the aircraft state according to the utterance (e.g., ATC instructions). In a particular example, S250 autonomously regulates effectors and / or propulsion systems of the aircraft according to the commands (e.g., to implement the command values). In a second example, the flight processing system can change waypoints and / or autopilot inputs based on the commands. In a variant, S200 can include providing the commands to the flight processing system (e.g., FCS) in a standardized format (e.g., a standardized machine-readable format).

[0098] However, S250 can appropriately regulate the aircraft based on the commands in other ways. Alternatively, the system can be used entirely in an assistive capacity (e.g., without passing commands to the aircraft processor or regulating the aircraft, such as to enable a hearing-impaired pilot to regulate the aircraft), and / or can be used in other ways.

[0099] However, S200 can include any other suitable elements.

[0100] Alternative embodiments implement the above-described methods and / or processing modules in non-transitory computer-readable media storing computer-readable instructions. The instructions can be executed by a computer-executable component integrated with the computer-readable media and / or processing system. The computer-readable media can include any suitable computer-readable media, such as RAM, ROM, flash memory, EEPROM, optical devices (CD or DVD), hard drives, floppy drives, non-transitory computer-readable media, or any suitable device. The computer-executable component can include a computing system and / or processing system connected to the non-transitory computer-readable media, such as a CPU, GPU, TPU, microprocessor, or ASIC (e.g., including one or more co-located or distributed, remote, or local processors), but the instructions can alternatively or additionally be executed by any suitable dedicated hardware device.

[0101] Embodiments of systems and / or methods can include every combination and permutation of the various system components and the various method processes described herein, wherein one or more instances of the methods and / or processes described herein can be performed asynchronously (e.g., sequentially), simultaneously (e.g., in parallel), or in any other suitable order by and / or using one or more instances of the systems, elements, and / or entities described herein.

[0102] 1) A system for semantic parsing of air traffic control (ATC) utterances for an aircraft, the system comprising:

[0103] a communication system on the aircraft, the communication subsystem configured to receive an ATC radio signal and convert the ATC radio signal into an audio signal;

[0104] a first processing system connected to the communication system, comprising:

[0105] a speech-to-text module configured to determine an utterance hypothesis from the audio signal; and

[0106] a question and answer (Q / A) module configured to determine an aircraft command based on the utterance hypothesis using a plurality of natural language queries; and

[0107] a second processing system connected to the first processing system and configured to regulate the aircraft based on the aircraft commands, wherein the ATC utterance is syntactically non-standardized, wherein the first processing system is configured to automatically provide the commands to the second processing system in a standardized format.

[0108] 2) The system of 1), wherein the first processing system is configured to convert the syntactically non-standardized utterance into the standardized format.

[0109] 3) The system of 1), wherein determining aircraft commands with the Q / A model comprises querying a pre-trained neural network model according to a structured sequence of the natural language queries.

[0110] 4) The system of 3), wherein the pre-trained neural network model is conditioned on ATC audio.

[0111] 5) The system of 3), wherein the structured sequence comprises a tree-based sequence having a plurality of dependencies linking one or more natural language queries to a determination that the aircraft is an intended recipient of an utterance corresponding to the utterance hypothesis.

[0112] 6) The system of 5), wherein the utterance hypothesis comprises speaker recognition, wherein the determination that the aircraft is the intended recipient is based on the speaker recognition.

[0113] 7) The system of 3), wherein each command comprises a command parameter and a set of values corresponding to the command parameter, wherein the command parameter is selected from a predetermined set of command parameters, wherein the set of values and the command parameter are determined via different natural language queries of the structured sequence.

[0114] 8) The system of 7), wherein the structured sequence comprises a tree-based sequence, wherein a value corresponding to the command parameter is queried in response to a determination that the command parameter is a subject of the utterance.

[0115] 9) The system of 1), wherein the plurality of natural language queries are selected from a predetermined set.

[0116] 10) A system for semantic parsing of air traffic control (ATC) utterances for an aircraft, the system comprising:

[0117] a communication system on the aircraft, the communication subsystem configured to receive ATC radio signals and transform the ATC radio signals into audio signals; and

[0118] a first processing system connected to the communication system, comprising:

[0119] a speech-to-text module configured to determine a speech hypothesis from the audio signal; and

[0120] a question and answer (Q / A) module configured to determine an aircraft command based on the speech hypothesis using a plurality of natural language queries.

[0121] 11) The system of 10), wherein each speech hypothesis comprises a boundary hypothesis, wherein the speech-to-text module comprises a sentence boundary detection (SBD) model configured to label entities within the audio signal and generate the boundary hypothesis based on the labeled entities.

[0122] 12) The system of 11), wherein the entities comprise transition talkers, wherein the SBD model is a neural network pre-trained to identify the transition talkers within multi-speech ATC audio based on audio artifact annotations.

[0123] 13) The system of 10), wherein the speech-to-text module comprises an integrated automatic speech recognition (ASR) and sentence boundary detection (SBD) module.

[0124] 14) The system of 13), wherein the integrated ASR / SBD module comprises a pre-trained neural network adjusted for ATC audio.

[0125] 15) The system of 13), wherein the speech-to-text module further comprises an ATC-adjusted language model, wherein determining the speech hypothesis comprises:

[0126] generating a plurality of language hypotheses for each speech using the integrated ASR / SBD module;

[0127] determining a language score for each of the plurality of language hypotheses using the ATC-adjusted language model; and

[0128] selecting a speech hypothesis from the plurality of language hypotheses based on the corresponding language scores.

[0129] 16) The system of 15), wherein the integrated ASR / SBD module is configured to assign a speech score to each of the plurality of language hypotheses, wherein the speech hypothesis is selected based on a combination of the corresponding language scores and speech scores.

[0130] 17) The system of 10), wherein determining the speech hypothesis comprises:

[0131] generating a plurality of utterance hypotheses for the utterance within the audio signal using an ASR module of the speech-to-text module;

[0132] selecting an utterance hypothesis from the plurality of utterance hypotheses using a language model.

[0133] 18) The system of 17), wherein the language model comprises a neural network pre-trained using entity tagged ATC transcripts.

[0134] 19) The system of 18), wherein the entity tagged ATC transcripts comprise labels corresponding to phonetically conflicting entities.

[0135] 20) The system of 10), wherein the utterance hypotheses comprise text transcripts.

[0136] 21) A method comprising:

[0137] receiving an audio utterance;

[0138] converting the audio utterance into a predetermined format comprising:

[0139] determining a set of utterance hypotheses; and

[0140] selecting an utterance hypothesis from the set; and

[0141] determining a command using a question and answer model based on the selected utterance hypothesis.

[0142] 22) The method of 21), wherein the method is performed using the system of any of 1) - 20).

[0143] Modifications and alterations can be made to the preferred embodiments of this application, as will be recognized by one skilled in the art, without departing from the scope of the application as defined in the appended claims.

Claims

1. A system for semantic parsing of ATC discourse for aircraft, the system comprising: The aircraft's communication subsystem is configured to receive ATC radio signals and convert the ATC radio signals into audio signals; A first processing system connected to the communication subsystem includes: A speech-to-text module, configured to determine utterance hypotheses based on the audio signal; and The Question and Answer (Q / A) module is configured to determine aircraft commands based on the utterance hypothesis using multiple natural language queries; and A second processing system, connected to the first processing system and configured to control the aircraft based on the aircraft commands, wherein the ATC utterances are syntactically non-standardized, and wherein the first processing system is configured to automatically provide the commands to the second processing system in a standardized format. The process of determining aircraft commands using the Q / A module includes: querying a pre-trained neural network model based on a structured sequence query of the natural language query, and... The structured sequence includes a tree-based sequence having multiple relevances that link one or more natural language queries to the determination of the aircraft as the intended recipient of the discourse corresponding to the discourse hypothesis.

2. The system according to claim 1, wherein, The first processing system is configured to convert the syntactically non-standardized discourse into the standardized format.

3. The system according to claim 1, wherein, The pre-trained neural network model is adjusted based on ATC audio.

4. The system according to claim 1, wherein, The discourse hypothesis includes speaker identification, wherein the determination that the aircraft is the intended recipient is based on the speaker identification.

5. The system according to claim 1, wherein, Each command includes command parameters and a set of values ​​corresponding to the command parameters, wherein the command parameters are selected from a predetermined set of command parameters, and the set of values ​​and the command parameters are determined via different natural language queries of the structured sequence.

6. The system according to claim 5, wherein, The structured sequence includes a tree-based sequence, wherein the value corresponding to the command parameter is queried in response to the determination that the command parameter is the topic of the utterance.

7. The system according to claim 1, wherein, The multiple natural language queries are selected from a predetermined set.

8. A system for semantic parsing of ATC discourse for aircraft, the system comprising: The aircraft's communication subsystem is configured to receive ATC radio signals and convert the ATC radio signals into audio signals; as well as A first processing system connected to the communication subsystem includes: A speech-to-text module, configured to determine utterance hypotheses based on the audio signal; and The Question and Answer (Q / A) module is configured to determine aircraft commands based on the discourse hypothesis using multiple natural language queries. The process of determining aircraft commands using the Q / A module includes: querying a pre-trained neural network model based on a structured sequence query of the natural language query, and... The structured sequence includes a tree-based sequence having multiple relevances that link one or more natural language queries to the determination of the aircraft as the intended recipient of the discourse corresponding to the discourse hypothesis.

9. The system according to claim 8, wherein, Each utterance hypothesis includes a boundary hypothesis, wherein the speech-to-text module includes a statement boundary detection model configured to label entities within the audio signal and generate the boundary hypothesis based on the labeled entities.

10. The system according to claim 9, wherein, The entity includes a transitional speaker, wherein the statement boundary detection model is a neural network pre-trained to identify the transitional speaker within multi-utterance ATC audio based on audio artifact annotation.

11. The system according to claim 8, wherein, The speech-to-text module includes an integrated automatic speech recognition and sentence boundary detection module.

12. The system according to claim 11, wherein, The integrated automatic speech recognition and sentence boundary detection module includes a pre-trained neural network for ATC audio modulation.

13. The system according to claim 11, wherein, The speech-to-text module also includes an ATC-modulated language model, wherein determining the discourse hypothesis includes: The integrated automatic speech recognition and sentence boundary detection module is used to generate multiple language hypotheses for each utterance; The language model adjusted by the ATC is used to determine a language score for each of the plurality of language hypotheses; and Based on the corresponding language scores, a discourse hypothesis is selected from the plurality of language hypotheses.

14. The system according to claim 13, wherein, The integrated automatic speech recognition and speech boundary detection module is configured to assign speech scores to each of the plurality of language hypotheses, wherein the speech hypothesis is selected based on a combination of the corresponding language scores and speech scores.

15. The system according to claim 8, wherein, Determining the discourse hypothesis includes: The automatic speech recognition module of the speech-to-text module generates multiple speech hypotheses for the speech within the audio signal; A language model is used to select the discourse hypothesis from the plurality of discourse hypotheses.

16. The system according to claim 15, wherein, The language model comprises a neural network pre-trained using entity-labeled ATC manuscripts.

17. The system according to claim 16, wherein, The entity-tagged ATC transcript includes tags corresponding to phonetically conflicting entities.

18. The system according to claim 8, wherein, The discourse assumption includes textual transcripts.

19. A method performed using the system according to any one of claims 1-18, the method comprising: Receive audio messages; Converting the audio speech into a predetermined format includes: Determine the set of discourse hypotheses; and Select discourse hypotheses from the set; and Based on the selected discourse assumptions, the command is determined using the question and answer module.

Images