Air traffic control response system based on flight simulator and response training method

Abstract

The invention relates to the technical field of flight simulation training, in particular to an air traffic control response system based on a flight simulator and a response training method. The method comprises the steps that a voice input module receives input voice audio data and sends the voice audio data to a voice recognition module; the voice recognition module recognizes and converts the voice audio data into voice text data; the instruction understanding module processes the voice text data to generate structured data corresponding to air-ground communication and sends the structured data to the instruction text generation module; the instruction text generation module generates a corresponding air traffic control instruction text according to the structured data; the voice synthesis module converts the air traffic control instruction text into corresponding voice audio data and sends the voice audio data to the voice output module; and the voice output module outputs the generated voice audio data. According to the invention, the recitation condition of the student can be intelligently evaluated, the workload of a teacher is effectively reduced, and the teacher can concentrate on guiding the flight operation of the student.

Description

Technical Field

[0001] The present invention relates to the technical field of flight simulation training, and more specifically to an air traffic control response system and a response training method based on a flight simulator. Background Art

[0002] With the continuous advancement of aviation technology, the key factors to ensure flight safety are gradually concentrated on the crew's communication skills and ability to deal with emergencies. Therefore, it is particularly important to conduct ground-to-air communication training during the crew's simulator training. However, in the current crew training cycle, there is a lack of a dedicated ground-to-air communication training environment. In addition, in daily training, flight instructors have to assume the role of air traffic control, which greatly increases the instructor's workload and affects the instructor's attention to the training crew's control situation.

[0003] At present, the concept of simulated captain is only used in traditional control simulators, that is, the controller needs to play the role of a pilot to complete control training. Accordingly, during simulator training, an independent and intelligent simulated control interaction module needs to be introduced to assist flight trainees in completing simulator training and reduce the workload of instructors. In the flight simulator, the instructor manually selects one or more control instructions to play through the IOS front-end interface.

[0004] Figure 1 The present invention discloses a flow chart of the simulator air-ground communication training in the prior art, such as Figure 1 As shown, the simulator air-ground communication training process includes: the instructor selects the ATC (air traffic control) command text through the front-end interface; the selected command text signal is transmitted to the Host through a wireless network or a data cable; the Host receives the command text signal and forwards it to the sound processing module; in the sound processing module, the command text data is converted into audio data and played through the left and right speakers of the simulator cockpit.

[0005] However, the above-mentioned simulator ground-to-air communication training process in the prior art has the following problems:

[0006] First, all IOS commands must be selected manually by the instructor, which greatly increases the instructor's workload and makes it difficult for the instructor to focus on the students' performance.

[0007] Secondly, the existing IOS instructions cannot adapt to special flight training scenarios (such as single engine failure, bird strike, wind shear, etc.);

[0008] Thirdly, the historical data module of all played commands does not have the functions of storage, calling, processing and analysis, and cannot intelligently evaluate the flight trainees' air-to-ground communication capabilities under normal and abnormal conditions, and will not be able to adapt to the authority's CBTA flight training requirements;

[0009] Finally, the existing air traffic control instructions cannot automatically adapt to the flight phase corresponding to the simulator. In flight training, there are multiple sound sources (such as the power plant sound, aerodynamic sound, landing gear sound, etc. in the simulator cockpit). In such a noisy environment, the voice signal of the flight trainee's command repetition picked up by the recording equipment is often mixed with multiple types of noise, making it difficult for the trainee's repetition voice signal to be clearly recognized. Summary of the invention

[0010] The purpose of the present invention is to provide an air traffic control response system and response training method based on a flight simulator, so as to solve the problem that the simulator air traffic control response system in the prior art is difficult to carry out professional land-to-air communication training.

[0011] In order to achieve the above-mentioned object, the present invention provides an air traffic control response system based on a flight simulator, comprising a speech input module, a speech recognition module, an instruction understanding module, an instruction text generation module, a speech synthesis module, and a speech output module:

[0012] The speech input module receives input speech audio data and sends it to the speech recognition module;

[0013] The speech recognition module converts the speech audio data into speech text data and sends it to the instruction understanding module;

[0014] The instruction understanding module processes the voice text data to generate structured data corresponding to the land-air conversation and sends it to the instruction text generation module;

[0015] The instruction text generation module generates corresponding air traffic control instruction text according to the structured data and sends it to the speech synthesis module;

[0016] The speech synthesis module converts the air traffic control instruction text into corresponding speech audio data and sends it to the speech output module;

[0017] The voice output module outputs the generated voice audio data.

[0018] In some embodiments, the air traffic control response system further includes a special situation processing module connected to the instruction text generation module:

[0019] The special situation processing module activates the predefined special situation scenario according to the trigger command, generates the corresponding special situation instruction and sends it to the instruction text generation module.

[0020] In some embodiments, the air traffic control response system further includes an intelligent evaluation module, which is connected to the speech recognition module and the instruction text generation module respectively:

[0021] The speech recognition module recognizes the flight trainee's repetition speech, converts it into repetition speech text, and sends it to the intelligent evaluation module;

[0022] The intelligent evaluation module receives the air traffic control instruction text generated by the instruction text generation module and generates an expected repetition text, compares and evaluates the expected repetition text with the repetition voice text, and forms an evaluation result.

[0023] In some embodiments, the air traffic control response system further includes a data storage module:

[0024] The data storage module is connected to the voice input module, the voice recognition module, the instruction text generation module and the voice synthesis module respectively, and is used to store the audio data and / or text data generated by the voice input module, the voice recognition module, the instruction text generation module and the voice synthesis module.

[0025] In some embodiments, the speech recognition module performs preprocessing and feature extraction operations on the input speech audio data, recognizes and processes based on the speech recognition model, and outputs speech text data;

[0026] The speech recognition model is constructed through training based on the training land-air call speech data.

[0027] In some embodiments, the speech recognition module performs the following preprocessing on the speech audio data:

[0028] Collect voice and audio data and process them digitally;

[0029] Pre-emphasize the processed speech audio signal;

[0030] Perform frame division and windowing on the pre-emphasized signal to divide the speech signal into frames of fixed length;

[0031] Perform endpoint detection on the frame-processed signal to identify the starting and ending points of valid speech.

[0032] In some embodiments, the instruction understanding module further includes a text segmentation unit, a semantic filling unit, and an intention recognition unit:

[0033] The text segmentation unit segments the identified sentence text according to the call sign and the instruction to obtain key fields corresponding to the call sign and the instruction;

[0034] The semantic filling unit performs semantic filling on the key fields to obtain a corresponding semantic sequence;

[0035] The intention recognition unit performs intention recognition based on the semantic sequence, obtains the final structured data containing the instruction intention and outputs it.

[0036] In some embodiments, the special situation processing module activates a predefined special situation scenario based on a trigger command, connects to the flight simulator management computer, generates corresponding special situation instructions, and compares and judges the aircraft flight status in the simulator system with the control instruction intentions and parameters extracted from the structured data to ensure that the generated air traffic control instructions comply with air traffic control rules.

[0037] In some embodiments, the speech synthesis module performs speech simulation based on the air traffic control instruction text, and realizes the mapping between letters, words and special pronunciations by phoneme permutations and combinations.

[0038] In some embodiments, the intelligent evaluation module uses a keyword evaluation standard to evaluate the flight trainee's recitation of instructions;

[0039] The evaluation indicators of the keyword evaluation standard include call sign accuracy, action instruction accuracy, parameter accuracy and total accuracy.

[0040] In order to achieve the above object, the present invention provides an air traffic control response training method based on a flight simulator, which is implemented by using the above-mentioned air traffic control response system based on a flight simulator, and includes the following steps:

[0041] A voice input step, receiving input voice audio data;

[0042] A speech recognition step, which converts speech audio data into speech text data;

[0043] Instruction text understanding step, the speech text data is processed to generate structured data corresponding to the land-air conversation;

[0044] The instruction text generation step generates the corresponding air traffic control instruction text according to the structured data;

[0045] The speech synthesis step is based on converting the air traffic control instruction text into corresponding speech audio data;

[0046] The speech output step outputs the generated speech audio data.

[0047] The present invention proposes an air traffic control response system based on a flight simulator and a training method thereof, which can autonomously adapt corresponding instructions according to the flight trainee's application permission and flight stage, and introduces a special situation instruction training scenario under the manual trigger mode, so as to accurately evaluate and train the communication and decision-making ability of the flight trainee in dealing with emergencies. By using the historical data of the playback instructions and the evaluation rules of the International Civil Aviation Organization (ICAO), it is possible to intelligently evaluate the student's repetition, effectively reducing the instructor's workload and allowing the instructor to focus more on guiding the student's flight operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The above and other features, properties and advantages of the present invention will become more apparent through the following description in conjunction with the accompanying drawings and embodiments, in which the same reference numerals always represent the same features, wherein:

[0049] Figure 1 The present invention discloses a flow chart for implementing the simulator air-ground communication training in the prior art;

[0050] Figure 2 The present invention discloses a block diagram of the composition principle of an air traffic control response system based on a flight simulator according to an embodiment of the present invention;

[0051] Figure 3 A technical implementation flow chart of an air traffic control response system based on a flight simulator according to an embodiment of the present invention is disclosed;

[0052] Figure 4 A workflow diagram of an air traffic control response system based on a flight simulator according to an embodiment of the present invention is disclosed;

[0053] Figure 5 A working flow chart of a speech recognition processing module according to an embodiment of the present invention is disclosed;

[0054] Figure 6 A preprocessing flow chart of a speech recognition processing module according to an embodiment of the present invention is disclosed;

[0055] Figure 7 A principle block diagram of an instruction understanding module according to an embodiment of the present invention is disclosed;

[0056] Figure 8 A flow chart of text segmentation of an instruction understanding module according to an embodiment of the present invention is disclosed;

[0057] Fig. 9 A flow chart of semantic filling of an instruction understanding module according to an embodiment of the present invention is disclosed;

[0058] Fig.10 A flow chart of instruction generation according to an instruction text generation module of an embodiment of the present invention is disclosed;

[0059] Fig.11 A flow chart of instruction conversion of an instruction text generation module according to an embodiment of the present invention is disclosed;

[0060] Fig.12 A step diagram of an air traffic control response training method based on a flight simulator according to an embodiment of the present invention is disclosed.

[0061] The meanings of the reference numerals in the figures are as follows:

[0062] 100 voice input module;

[0063] 200 speech recognition module;

[0064] 300 instruction understanding module;

[0065] 400 instruction text generation module;

[0066] 500 speech synthesis module;

[0067] 600 Special request processing module;

[0068] 700 intelligent evaluation modules;

[0069] 800 data storage module;

[0070] 900 voice output module. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the invention and are not used to limit the invention.

[0072] The present invention relates to a complete airborne intelligent air traffic control response system based on a flight simulator for flight training. The system is suitable for all flight phases, and can autonomously adapt to the flight frequency, adapt to different flight phases, and integrate a set of command trigger logic functions for different training scenarios. It can be integrated into an existing flight simulator system as a subsystem, showing significant convenience and compatibility.

[0073] Figure 2 The following discloses a block diagram of the composition principle of an air traffic control response system based on a flight simulator according to an embodiment of the present invention. Figure 2 As shown, an air traffic control response system based on a flight simulator proposed by the present invention includes a speech input module 100, a speech recognition module 200, an instruction understanding module 300, an instruction text generation module 400, a speech synthesis module 500, a special situation processing module 600, an intelligent evaluation module 700, a data storage module 800 and a speech output module 900:

[0074] The speech input module 100 is connected to the speech recognition module 200, receives input speech audio data and sends it to the speech recognition module 200;

[0075] The speech recognition module 200 is connected to the instruction understanding module 300, and converts the speech audio data into speech text data and sends it to the instruction understanding module 300;

[0076] The instruction understanding module 300 is connected to the instruction text generation module 400, processes the voice text data to generate structured data corresponding to the land-air conversation and sends it to the instruction text generation module 400;

[0077] The instruction text generation module 400 is connected to the speech synthesis module 500, generates corresponding air traffic control instruction text according to the structured data and sends it to the speech synthesis module 500;

[0078] The speech synthesis module 500 is connected to the speech output module 900, and converts the air traffic control instruction text into corresponding speech audio data and sends it to the speech output module 900;

[0079] The speech output module 900 outputs the generated speech audio data.

[0080] Furthermore, the special situation processing module 600 is connected to the instruction text generating module 400:

[0081] The special situation processing module 600 activates a predefined special situation scenario according to a trigger command, generates a corresponding special situation instruction and sends it to the instruction text generation module 400.

[0082] Furthermore, the intelligent evaluation module 700 is connected to the speech recognition module 200 and the instruction text generation module 400 respectively:

[0083] The speech recognition module 200 recognizes the flight trainee's recitation speech, converts it into recitation speech text and sends it to the intelligent evaluation module 700;

[0084] The intelligent evaluation module 700 generates an expected repetition text based on the air traffic control instruction text generated by the received instruction text generation module 400, compares and evaluates the expected repetition text with the repetition voice text, and forms an evaluation result.

[0085] Furthermore, the data storage module 800 is respectively connected to the voice input module 100, the voice recognition module 200, the instruction text generation module 400, and the voice synthesis module 500, and is used to store the audio data and / or text data generated by the voice input module 100, the voice recognition module 200, the instruction text generation module 400, and the voice synthesis module 500.

[0086] Figure 3 The invention discloses a technical implementation flow chart of an air traffic control response system based on a flight simulator according to an embodiment of the present invention. Figure 4 The following discloses a workflow diagram of an air traffic control response system based on a flight simulator according to an embodiment of the present invention. Figure 3 and Figure 4, the various component modules of the air traffic control response system based on the flight simulator of the present invention are described in detail.

[0087] In this embodiment, the voice input module 100 includes headsets for the driver and the front passenger.

[0088] The voice input module 100 collects the audio data of the flight trainees, normalizes the voice data, converts it into an internal data structure, and sends it to the voice recognition module 200 through the information middleware and stores the audio data in the data storage module 800.

[0089] The speech recognition module 200 uses speech recognition technology to pick up command audio, perform noise processing on the audio, and generate corresponding text to facilitate subsequent EBTA training evaluation.

[0090] The speech recognition module 200 uses advanced speech recognition technology to accurately pick up the command audio, perform noise filtering on the audio, and convert the processed audio into a text format to facilitate subsequent EBTA training and evaluation.

[0091] In this embodiment, the voice recognition module 200 recognizes and converts the collected voice audio data into voice text data, which is the basis of the entire air traffic control response system.

[0092] The speech recognition module 200 converts the voice instructions of the flight trainee's permission application / repeat into text data using speech recognition technology.

[0093] More specifically, the speech recognition module 200 performs preprocessing and feature extraction operations on the input speech audio data, recognizes and processes based on the speech recognition model, and outputs speech text data. The speech recognition module 200 can accurately recognize and process the special pronunciation of letters / instructions for land and air calls, and mixed Chinese and English speech input.

[0094] The goal of the speech recognition module 200 is to find the most likely word sequence W (i.e., speech recognition text) given an observation sequence X (i.e., audio data). The speech recognition module 200 preprocesses the input flight trainee audio signal and converts the preprocessed speech signal recognition into a corresponding text sequence:

[0095] W*=argmax P(W|X)≈argmax P(X|W)P(W);

[0096] Where X is the observation sequence (i.e., audio data), W is the word sequence of the entire audio instruction, P(W|X) is the probability of generating the word sequence W given the observation sequence X, and P(X|W) represents the probability of observing the audio data X under the word sequence W.

[0097] P(W) is the probability of the word sequence W itself, that is, the speech recognition model, which is used to evaluate the rationality of the word sequence W in the language and determines which one of the multiple possible results the speech recognition module 200 selects is more in line with the language rules and context.

[0098] Assume that after m times of speech recognition, the recognition result is a text combination A, A = (a1, a2, ..., am), where am represents the speech recognition text obtained after the mth recognition.

[0099] Figure 5 The following discloses a working flow diagram of a speech recognition processing module according to an embodiment of the present invention. Figure 5 As shown in the figure, the entire speech recognition process starts with the pilot's active ground-to-air audio signal, and then goes through preprocessing and feature extraction operations, and is calculated by the speech recognition model, and finally outputs the most likely word sequence as the recognition result. After multiple recognitions, the system will optimize the multiple text combinations A obtained and select the most appropriate command text as the output.

[0100] Wherein, the speech recognition model is trained and constructed based on training land-air call speech data.

[0101] The speech recognition module 200 recognizes and processes the pilot's audio signal, including a training phase and a recognition phase. In the training phase, a speech recognition model is first constructed by training the land-to-air call speech data. In the recognition phase, speech recognition is achieved by matching the pilot's active land-to-air call speech signal.

[0102] More specifically, see Figure 5 In the training phase, the speech recognition module 200 performs noise elimination, filtering and other operations on the training land-air call speech data through preprocessing, performs feature extraction on the preprocessed speech signal, extracts key feature parameters from the speech signal, and uses the extracted feature parameters to create or update the template library of the speech recognition model, so as to provide a speech recognition model for the subsequent recognition phase;

[0103] During the recognition stage, the speech recognition module 200 subjects the pilot's active ground-to-air call speech signal to the same preprocessing to perform noise elimination and signal enhancement; performs feature extraction on the preprocessed speech signal to obtain speech features for matching; matches the extracted feature parameters with the features in the model library to identify the category or content of the speech; performs necessary post-processing on the results based on the recognition results, and outputs the final recognition results.

[0104] In some embodiments, post-processing includes storing the recognized text results in the data storage module 800 .

[0105] Whether in the training stage or the recognition stage, the speech recognition module 200 will perform preprocessing operations on the speech data. The preprocessing process mainly includes the acquisition, filtering, sampling, normalization, pre-emphasis, frame segmentation and windowing, endpoint detection, etc. of the speech audio signal. The purpose is to convert the original speech audio signal into a digital signal that is convenient for further processing through preprocessing, and determine the effective part of the speech audio signal for feature extraction and subsequent processing.

[0106] Figure 6 A preprocessing flow chart of a speech recognition processing module according to an embodiment of the present invention is disclosed, such as Figure 6 As shown, the speech recognition module 200 performs the following preprocessing on the speech audio data:

[0107] Collect voice and audio data and process them digitally;

[0108] Pre-emphasize the processed speech audio signal to emphasize the high-frequency components and improve the feature extraction effect;

[0109] Perform frame division and windowing on the pre-emphasized signal to divide the speech signal into frames of fixed length;

[0110] Perform endpoint detection on the frame-processed signal to identify the starting and ending points of valid speech.

[0111] Furthermore, the speech recognition module 200 collects the speech audio signals of the flight trainees' active ground-to-air conversations, thereby realizing the collection of ground-to-air conversation audio signals.

[0112] Furthermore, the speech recognition module 200 performs digital processing on the speech audio data, further comprising:

[0113] The collected speech audio signal is filtered to remove noise and interference.

[0114] The filtered signal is sampled to convert the analog signal into a digital signal.

[0115] The sampled speech audio signal is normalized to keep the amplitude of the speech audio signal within a reasonable range to ensure the stability of subsequent processing.

[0116] Furthermore, the speech recognition module 200 performs frame division and windowing operations on the pre-emphasized signal to divide the speech audio signal into frames of fixed length for subsequent analysis.

[0117] Furthermore, the speech recognition module 200 performs endpoint detection after frame processing to identify the starting and ending points of valid speech, exclude non-speech parts (such as silent segments or noise), and determine the exact starting and ending points of the speech signal through endpoint detection to ensure that the recognition process only processes valid speech segments.

[0118] In this embodiment, the instruction understanding module 300 further processes the recognized flight trainee voice text data to generate structured data corresponding to the flight trainee's land-to-air conversation.

[0119] The structured data at least includes necessary parameters such as the control intention of the flight trainee's ground-to-air communication, aircraft call sign, instructions (altitude, speed), etc.

[0120] Figure 7 The principle block diagram of the instruction understanding module according to an embodiment of the present invention is disclosed. Figure 7 As shown, the instruction understanding module 300 further includes a text segmentation unit 301, a semantic filling unit 302 and an intention recognition unit 303:

[0121] The text segmentation unit 301 segments the recognized speech text according to the call sign and the command to obtain key fields corresponding to the call sign and the command;

[0122] The semantic filling unit 302 performs semantic filling on the key field to obtain a corresponding semantic sequence;

[0123] The intention recognition unit 303 performs intention recognition based on the semantic sequence, obtains the final structured data containing the control intention and outputs it.

[0124] Furthermore, the text segmentation unit 301 clearly distinguishes the call sign and the instruction in the voice text by extracting the key words in the call sign and the instruction and cutting the text, so as to facilitate further processing by the system.

[0125] Furthermore, the keyword extraction method is to match and obtain key fields through the call sign library and the instruction library.

[0126] Figure 8 A text segmentation flow chart of an instruction understanding module according to an embodiment of the present invention is disclosed, such as Figure 8 The text segmentation process shown includes:

[0127] In the voice input step, the pilot's voice audio signal is received, such as Figure 8 , the voice audio signal “HO1345descend FL300”, this step is performed by the voice input module 100.

[0128] The speech-to-text step converts the speech audio signal into the corresponding speech text content through speech recognition, such as Figure 8 In the example, the voice signal “HO1345 descend FL300” is recognized as the voice text “Juneyao one three fourfive descend flight level three zero zero”, and this step is performed by the voice recognition module 200.

[0129] The call sign and command extraction step extracts the aircraft's call sign and command parts from the recognized speech text, such as Figure 8 In the video, “Juneyao one three four five” is extracted as the call sign, and “descend flight level threezero zero” is extracted as the instruction part.

[0130] The semantic text segmentation step identifies the segmentation key fields from the extracted call signs and instructions, e.g. Figure 8 In the example, “Juneyao, one three four five” is recognized and cut as the call sign keyword, and “descend, flight level, three zero zero” is recognized and cut as the command keyword.

[0131] Furthermore, the semantic filling unit 302 semantically fills the key fields of the acquired call sign and instruction, so that the call sign and instruction are semantically complete and clear, so as to facilitate the subsequent parsing and execution of the flight instruction.

[0132] The function of the semantic filling unit 302 is to perform semantic filling on the text sequence of the identified key fields and match each key field with its corresponding semantic label.

[0133] The semantic filling unit 302 analyzes the semantic components of the text sequence and fills the key fields mapped by the text segmentation unit 301 into the corresponding instruction statement positions, that is, for a given text sequence W = (w1, w2, ..., w n ), we need to find the corresponding semantic sequence S = (s1, s2, ..., s n ),Right now

[0134]

[0135] in, Represents the semantic label corresponding to the given sequence W. Each w i Corresponding to a semantic label s i ,Semantic tags may represent elements such as call signs, commands, etc.

[0136] In air-ground calls, the structure of "other party's call sign - own call sign - call content" is usually followed. When filling in the meaning, text analysis and semantic filling are carried out according to this standard grammatical structure.

[0137] The grammatical structure specified by the text sequence is: [the other party's call sign, your own call sign (can be omitted according to actual conditions), action, parameter, others].

[0138] The other party's call sign and your own call sign are the identifiers of the calling parties, usually represented by the airline code or flight number. The action represents the action in the instruction, such as "climb" and "descend", and the parameter is the corresponding specific content such as altitude and speed. Others can be supplementary information.

[0139] Through semantic filling, the final semantic sequence S will assign corresponding semantic labels to each word, construct a complete instruction statement, and ensure the accuracy and availability of information.

[0140] Fig. 9 A flow chart of semantic filling of an instruction understanding module according to an embodiment of the present invention is disclosed, such as Fig. 9 As shown, the semantic filling process of the command understanding module matches and parses the extracted call signs and commands through a parameter dictionary to ensure the accuracy of the identified call signs and command information.

[0141] In this embodiment, the parameter dictionary includes a command database and a call sign database.

[0142] Among them, the call sign database and the command database are shown in Table 1 below:

[0143] Table 1

[0144]

[0145]

[0146] More specifically, the semantic filling process of the command understanding module is divided into the call sign part processing process and the command part processing process:

[0147] like Fig. 9 As shown, the call sign part processing flow includes the following steps:

[0148] Enter the call sign text as "Juneyao, one three four five";

[0149] The call sign text is input into the call sign database for parsing and matching. The call sign database includes airline and flight number data. The airline supports inputting the airline's 2-letter code or ICAO 3-letter code (such as HO, MU, CA, CS, etc.). The flight number extracts 3 or 4 digits in the call sign by default.

[0150] The call sign is matched and parsed to obtain the airline and flight number information.

[0151] like Fig. 9 As shown, the instruction part processing flow includes the following steps:

[0152] The input command text is "descend, flight level, three zero zero".

[0153] Input the instruction text into the instruction database for parsing and matching.

[0154] The command database is classified and matched according to different types of parameters, including action, height parameters, speed parameters, angle parameters and other parameters:

[0155] The actions include Climb, Turn, Descend, Maintain, etc.

[0156] The altitude parameters include Flight level, Meter, etc.

[0157] The speed parameters include GS (ground speed), TAS (true airspeed), IAS (indicated airspeed), knot (nautical mile / hour), etc.

[0158] The angle parameters include Climb gradient, Descend gradient, etc.

[0159] The other parameters include PBN (performance-based navigation), RNP (required navigation performance), roger (received instructions), confirm (confirmation), etc.

[0160] The semantic filling unit 302 classifies and matches various parameters in the instruction, thereby achieving accurate understanding and execution of the instruction.

[0161] Furthermore, the intention recognition unit 303 classifies the multiple instruction texts after semantic filling into multiple specific intention categories, and selects the final air traffic control instruction output text according to probability.

[0162] After semantic filling, multiple instruction texts will be classified into m (m ≥ 1) intent categories. Each instruction text represents a possible air traffic control instruction, and the specific intents include "climb", "descend", "turn", "keep heading", etc.

[0163] The multiple instruction texts semantically filled by the semantic filling unit 302 are classified into m (m≥1) specific intent categories, that is,

[0164]

[0165] Among them, P(C|W) represents the probability that the instruction intention is C given the current sequence W, Indicates the final instruction intent classification.

[0166] The intention recognition unit 303 calculates the probability of the intention category of each instruction through a model (such as Bayesian classification, neural network, etc.). Specifically, the model outputs an intention probability distribution P(C|W) based on the training data and the input text W.

[0167] For a given text sequence W, the intention recognition unit 303 evaluates the probability of each possible instruction intention and selects the intention with the highest probability as the final air traffic control instruction output text.

[0168] Assume that after multiple processing, the text combination of m outputs is G(g0, g1, …, gm), where gm represents the instruction text obtained for the mth time, and the intention with the highest probability is selected as the final air traffic control instruction output.

[0169] The instruction text generation module 400 automatically generates air traffic control instruction text corresponding to the structured data related to the pilot and instruction data recognizable by the simulator system according to the civil aviation ground-air communication standard rules.

[0170] The special situation processing module 600 is connected to the instruction text generation module 400. It can activate the predefined special situation scenario by triggering a command under human intervention. It can generate corresponding special situation instructions according to the relevant special situation scenario and send them to the instruction text generation module 400 to support special situation processing training for flight trainees.

[0171] The instruction text generation module 400 is specially designed with instruction repetition rules applicable to a variety of scenarios. It can not only generate regular instructions, but also support adding special scenarios to the instruction generation process.

[0172] The special situation processing module 600 can activate the preset special situation scenario under the intervention of the instructor, and generate corresponding special situation instructions based on the special situation scenario and necessary parameters.

[0173] Fig.10The instruction generation flow chart of the instruction text generation module according to one embodiment of the present invention is disclosed, such as Fig.10 As shown, the instruction understanding module 300 performs instruction keyword recognition and instruction parameter extraction, and the instruction text generation module 400 can generate and send different instructions.

[0174] According to the actual operation scenario, flight training instructions are divided into regular instructions and special situation instructions. The default sending mode is regular instructions. The conversion of regular instructions to special situation instructions requires the instructor to manually activate the special situation scenario of the special situation processing module 600:

[0175] For conventional instructions, the instruction text generation module 400 directly generates the corresponding ATC instructions;

[0176] For special situation scene instructions, the special situation processing module 600 activates the predefined special situation scene according to the trigger command, accesses the flight simulator management computer, reads the flight simulator parameters, and generates corresponding special situation instructions;

[0177] Based on the special situation instructions, the instruction text generation module 400 compares and judges the aircraft flight status in the simulator system with the control instruction intention and parameters extracted from the structured data to ensure that the generated air traffic control instructions comply with air traffic control rules (such as vertical separation, horizontal separation, wake turbulence separation, speed limit, etc.) and meet actual operational requirements.

[0178] In this embodiment, the instruction understanding module 300 outputs the structured data I t :

[0179] I t =AI1P1(x1,x2,...,x n )I2P2(x1,x2,...,x n )…I n P n (x1,x2,...,x n );

[0180] Among them, I1, I2…, I n is the intention extracted by the intention recognition unit 303;

[0181] P n (x1,x2,…,x n ) represents the nth type of intention instruction parameter;

[0182] Among them, the conventional instruction is a request for permission issued by the flight trainee through the intercom system in the simulator. After receiving the permission, the instruction text generation module 400 determines the instruction type. The instruction determination basis is I1, I2…, I n and the text of the instruction;

[0183] For special situation instructions, after the instructor activates the special situation subject on the front-end interface, the special situation processing module 600 compares and judges the flight parameters extracted from the flight simulator management computer with the control instruction intention and parameters, and gives corresponding instructions. The instruction judgment basis is the same as I1, I2…, I n and the instruction text.

[0184] The speech synthesis module 500 converts the text data in the sound computer into speech data, and then transmits it to the flight trainees through the simulator data link, forming a closed loop of training dialogue and negotiation between the flight trainees and the onboard transponder air traffic control system, while supporting multiple rounds of land-to-air conversations.

[0185] In this embodiment, the voice output module 900 is a speaker of the driver and co-pilot in the cockpit, which plays voice audio data to the trainee through the speaker to form a dialogue communication loop.

[0186] The speech synthesis module 500 performs speech simulation based on the air traffic control instruction text, and realizes the mapping between letters, words and special pronunciations by permutation and combination of phonemes.

[0187] Furthermore, the speech synthesis module 500 can accurately synthesize the special pronunciation of letters / commands for land and air communication (for example, reading "three" as "tree" and "nine" as "niner"), and mixed Chinese and English speech (such as "269m: ELEVATIONTOO SIX NIN-er METERS").

[0188] The speech synthesis module 500 performs speech simulation on the text synthesized by the air traffic control instructions, and defines the sound by phonemes, that is, the mapping between letters, words and special pronunciations can be achieved by the arrangement and combination of phonemes.

[0189] In the standard language of air-ground communication, there are three special pronunciations and their pronunciation sequences are as follows:

[0190] a) Processing and synthesis of special pronunciations of letters: using proprietary pronunciations such as Alpha-Zulu to replace the standard AZ pronunciations. For example, AGEL can be recognized as: Alpha Golf Echo Lima;

[0191] b) Special noun pronunciation and synthesis processing: The pronunciation sequence is added to the pronunciation and synthesis dictionary, such as ILS, which is pronounced as I(AY)L(EH L)S(EHS) and so on.

[0192] The intelligent assessment module 700 is designed based on the ICAO / PEPEC assessment rules. It can independently and accurately assess the ability to repeat air-to-ground calls, making it easier for later flight trainees to identify and fill in their weaknesses.

[0193] The intelligent evaluation module 700 performs relationship mapping on the generated air traffic control instruction text, generates the expected repetition text of the flight trainee, and performs intelligent evaluation on the trainee repetition voice text after voice recognition. Optionally, the evaluation is based on the PEPEC evaluation standard.

[0194] Furthermore, the present invention proposes an air traffic control response system based on a flight simulator, comprising a first label processing module and a second label processing module:

[0195] The main function of the first label processing module is to label the pilot flying the aircraft (PF) or the pilot monitoring the aircraft (PM) on the trainee's repeated voice text, because the PF and PM need to be distinguished in the simulator cockpit operation;

[0196] The main function of the second label processing module is to add a time label to the expected repetition text, so as to facilitate the rapid positioning and keyword comparison of the expected repetition text and the student's repetition voice text.

[0197] Fig.11 The instruction conversion flow chart of the instruction text generation module according to one embodiment of the present invention is disclosed. Fig.11 As shown, the detailed instruction conversion process of the instruction text generation module 400 includes the following steps:

[0198] a) ATC instruction text g generated after receiving intent recognition m ;

[0199] b) Send the air traffic control instruction text g m Transmit to the instruction mapping relationship library;

[0200] c) In the command mapping relationship library, the air traffic control command text g m Convert to expected repetition text b m ;

[0201] d) for b m Perform labeling processing to generate the final expected repetition text.

[0202] The instruction text generation module 400 transmits the generated air traffic control instruction text to the instruction mapping relationship library of the intelligent evaluation module 700 to obtain the generation of the expected repetition text. After passing through the second label processing module, the expected repetition text combination B is generated, where B = (b1, b2, ..., b m ), b m Represents the expected repetition text mapped out for the mth time.

[0203] The word order structure of the expected repetition text adopts the N-gram model, which evaluates the possibility of different word orders by calculating the joint probability of word sequences, so as to determine the most likely arrangement order.

[0204] Assume that the appearance of each word in the air traffic control instruction text depends on the m-1 words before it, so as to calculate the conditional probability of the entire word sequence, and select the sequence with the highest probability as the final sequence.

[0205] Specifically, the expression for the final sequence of expected recitation text is as follows:

[0206] p max (word sequence) = p(w1) × p(w2|w1) × … × p(wn|w1w2…wm-1);

[0207] Among them, p(w m ∣w m-1 ) means given the previous word w m-1 Under the condition that word w m The conditional probability of .

[0208] The intelligent evaluation module 700 uses the ROUGE algorithm to calculate the overlapping units between the expected repetition text and the repetition speech text to evaluate the flight trainee's repetition instruction.

[0209] The intelligent evaluation module 700 uses the keyword evaluation criteria to evaluate the flight trainee's recitation of instructions;

[0210] The evaluation indicators of the keyword evaluation standard include call sign accuracy, action instruction accuracy, parameter accuracy and total accuracy.

[0211] The keyword evaluation criteria are mainly based on the PEPEC evaluation criteria, covering pronunciation, structure, vocabulary, fluency, comprehension and interaction.

[0212] Therefore, the intelligent evaluation module 700 can adopt the ROUGE calculation method to measure the evaluation effect of the flight student's repetition of instructions by comparing the overlapping units (such as n-grams, word sequences and word pairs) between the generated summary (i.e., the repetition speech text) and the reference summary (i.e., the expected repetition text).

[0213] Among them, R OUGE-1 and R OUGE-2 The informativeness can be evaluated, R OUGE-L Fluency can be assessed.

[0214] ROUGE-N The calculation formula is,

[0215]

[0216] In the formula, the subscript N is usually 1 or 2; n is the length of n-grams;

[0217] R ef is the set of reference summaries; S is the set of n-grams in the reference summaries;

[0218] In the figure above, the numerator represents the sum of the number of times the n-grams in the generated summary appear in the reference summary, and the denominator represents the total number of n-grams in the reference summary. o,match (g n ) is the maximum number of n-grams that appear simultaneously in the generated summary and the corresponding reference summary; C o (g n ) is the number of n-grams in the reference summary.

[0219] R OUGE-L The calculation formula is:

[0220]

[0221] Where: L CS is the longest common subsequence; l en is the length of the sequence; C is the standard sequence; S is the generated sequence; R LCS is the recall rate; P LCS is the accuracy; F LCS For R OUGE-L value; β is a parameter used to measure the importance between recall rate and precision rate. Generally, β is set to a larger value. In this case, F LCS Will pay more attention to R LCS , β takes 1, indicating that precision and recall are treated equally.

[0222] Due to the particularity of this field, the readback instruction must be completely correct to be recorded as a qualified readback instruction. Airborne air traffic control (ATC) instructions require that the pilot's readback be responded to in accordance with the "Air Traffic Radio Telephone Phrases Specification" (MH / T4014-2003) without losing key information.

[0223] In view of the characteristics of the instruction text and the repetition standard, the present invention introduces a new evaluation standard for this field - a keyword evaluation standard.

[0224] The evaluation indicators of this standard include:

[0225] Call sign accuracy (CSA), action instruction accuracy (AIA), parameter accuracy (PA) and total accuracy (TA).

[0226] The total accuracy is calculated at the end. Only when a repeat instruction meets the three evaluation indicators of call sign accuracy, action instruction accuracy, and parameter accuracy at the same time, can the repeat instruction be regarded as a correct instruction.

[0227] The definition and calculation formula of the specific evaluation indicators of the keyword evaluation standard are as follows:

[0228] 1) The call sign consists of the airline abbreviation and the flight number. The call sign accuracy calculation formula is:

[0229]

[0230] 2) The action command is the action included in the repetition command, such as rise, fall, hold, etc. The action command accuracy calculation formula is:

[0231]

[0232] 3) Parameters refer to the key elements of airborne intelligent air traffic control instructions, including key supplementary information of instruction actions such as speed, altitude, heading and waypoints. The parameter accuracy calculation formula is:

[0233]

[0234] Where: N is the number of samples to be tested;

[0235] g(i), q(i), and h(i) are the characteristic functions of call sign, action command, and parameter, respectively, which can be expressed as:

[0236]

[0237] in, is the condition of the indicator function;

[0238] p redi is the value of the call sign and action parameter in the generated i-th readback instruction;

[0239] t ruthi It is the value of the call sign, action and parameter in the expected repeat command. When the two completely correspond and match, the indicator function value is 1, otherwise it is 0.

[0240] The total accuracy rate is the ratio of effective readback commands to total readback commands. The total accuracy rate of the readback command can meet the requirements only when all call signs, action commands and parameters meet the conditions.

[0241] The calculation formula for the overall accuracy is:

[0242]

[0243] T(i) is the characteristic function of the total accuracy, and its calculation formula is,

[0244]

[0245] Since the speech text after speech recognition of the flight trainee's mth application for instruction is a m , the expected repetition text generated by the mth mapping is g m , then the evaluation effect P of the flight trainee after one training final-result It can be expressed as:

[0246]

[0247] Among them, the obtained P final-result The value range is [0,1]. After processing, the score is displayed in real time on the front-end interface in the form of 0-100.

[0248] The data storage module 800 is responsible for storing all audio data and recognized text data in the process, and the storage is marked with time (year, month, day / hour, minute, and second).

[0249] In this embodiment, the data storage module 800 is responsible for storing all text data after labeling.

[0250] The present invention relates to an airborne intelligent air traffic control answering system based on a flight simulator, wherein the hardware carrier includes a digital signal processor or a general-purpose processor, which is responsible for executing a speech recognition algorithm. In addition, the system is also equipped with a storage medium for storing model data, algorithm library, and sample programs required for speech recognition. The airborne intelligent air traffic control answering system complies with the regulations and requirements of civil aviation ground-to-air communication, and is constructed as a task-oriented dialogue system based on the needs of flight training, which has an input / output interface and a human-computer interaction interface for external communication and operation.

[0251] Using the above-mentioned air traffic control response system based on a flight simulator, the present invention also proposes an air traffic control response training method based on a flight simulator, comprising the following steps:

[0252] S1, voice input step, receiving input voice audio data;

[0253] S2, speech recognition step, converting speech audio data into speech text data;

[0254] S3, a command text understanding step, processing the voice text data to generate structured data corresponding to the land-air conversation;

[0255] S4, instruction text generation step, generating corresponding air traffic control instruction text according to the structured data;

[0256] S5, a speech synthesis step, converting the air traffic control instruction text into corresponding speech audio data;

[0257] S6: Voice output step, outputting the generated voice audio data.

[0258] It should be noted that the specific details of the speech input step have been described in detail in the speech input module, the specific details of the speech recognition step have been described in detail in the speech recognition module, the specific details of the instruction text understanding step have been described in detail in the instruction text understanding module, the specific details of the instruction text generation step have been described in detail in the instruction text generation module, the specific details of the speech synthesis step have been described in detail in the speech synthesis module, and the specific details of the speech output step have been described in detail in the speech output module.

[0259] Please note that the details of speech input have been explained in detail in the speech input module. Similarly, the steps of speech recognition, command text understanding, command text generation, speech synthesis, and speech output have also been explained in detail in the corresponding modules. I will not repeat them here.

[0260] Although the above methods are illustrated and described as a series of actions for simplicity of explanation, it should be understood and appreciated that these methods are not limited by the order of the actions, because according to one or more embodiments, some actions may occur in a different order and / or concurrently with other actions from those illustrated and described herein or not illustrated and described herein but understandable to those skilled in the art.

[0261] The present invention provides an air traffic control response system based on a flight simulator and a training method thereof, which specifically have the following beneficial effects:

[0262] 1) Complete intelligent airborne transponder air traffic control system design: The system integrates speech recognition, command extraction, command repetition, speech synthesis and other technologies. It can intelligently recognize and understand the voice commands of flight trainees and automatically output the repetition commands of simulated controllers, reducing labor costs and simplifying the process of traditional simulation training;

[0263] 2) Detailed readback rules: Through the study and analysis of the voice patterns of real air-to-ground dialogues, a set of detailed readback rules was developed to adapt to the pilot command readback patterns in different scenarios, thus improving the accuracy and applicability of the readback;

[0264] 3) Integration of special situation handling module: Added a repetition mode in special situation scenarios, which can trigger predefined special situation scenarios under manual intervention, and conduct targeted training and assessment on flight trainees' ability to deal with special situations, thus enhancing the practicality and pertinence of training;

[0265] 4) System compatibility and convenience: It can be integrated into the existing controller training simulator system as a subsystem. It has strong convenience and compatibility, and is easy to deploy and use in different training environments.

[0266] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "an" and / or "the" do not refer to the singular and may also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0267] The above embodiments are provided for persons familiar with the art to implement or use the present invention. Personnel familiar with the art can make various modifications or changes to the above embodiments without departing from the inventive concept of the present invention. Therefore, the protection scope of the present invention is not limited to the above embodiments, but should be the maximum scope of the innovative features mentioned in the claims.

Claims

1. An air traffic control response system based on a flight simulator, characterized in that: Including voice input module, voice recognition module, instruction understanding module, instruction text generation module, voice synthesis module, voice output module: The speech input module receives input speech audio data and sends it to the speech recognition module; The speech recognition module converts the speech audio data into speech text data and sends it to the instruction understanding module; The instruction understanding module processes the voice text data to generate structured data corresponding to the land-air conversation and sends it to the instruction text generation module; The instruction text generation module generates corresponding air traffic control instruction text according to the structured data and sends it to the speech synthesis module; The speech synthesis module converts the air traffic control instruction text into corresponding speech audio data and sends it to the speech output module; The voice output module outputs the generated voice audio data.

2. The air traffic control response system based on the flight simulator according to claim 1 is characterized in that: It also includes a special situation processing module, which is connected to the instruction text generation module: The special situation processing module activates the predefined special situation scenario according to the trigger command, generates the corresponding special situation instruction and sends it to the instruction text generation module.

3. The air traffic control response system based on the flight simulator according to claim 1, characterized in that: It also includes an intelligent evaluation module, which is connected to the speech recognition module and the instruction text generation module respectively: The speech recognition module recognizes the flight trainee's repetition speech, converts it into repetition speech text, and sends it to the intelligent evaluation module; The intelligent evaluation module generates an expected repetition text based on the air traffic control instruction text generated by the received instruction text generation module, compares and evaluates the expected repetition text with the repetition voice text, and forms an evaluation result.

4. The air traffic control response system based on the flight simulator according to claim 1, characterized in that: Also includes data storage modules: The data storage module is connected to the voice input module, the voice recognition module, the instruction text generation module and the voice synthesis module respectively, and is used to store the audio data and / or text data generated by the voice input module, the voice recognition module, the instruction text generation module and the voice synthesis module.

5. The air traffic control response system based on the flight simulator according to claim 1, characterized in that: The speech recognition module performs preprocessing and feature extraction operations on the input speech audio data, recognizes and processes based on the speech recognition model, and outputs speech text data; The speech recognition model is constructed through training based on the training land-air call speech data.

6. The air traffic control response system based on the flight simulator according to claim 5, characterized in that: The speech recognition module performs the following preprocessing on the speech audio data: Collect voice and audio data and process them digitally; Pre-emphasize the processed speech audio signal; Perform frame division and windowing on the pre-emphasized signal to divide the speech signal into frames of fixed length; Perform endpoint detection on the frame-processed signal to identify the starting and ending points of valid speech.

7. The air traffic control response system based on the flight simulator according to claim 1, characterized in that: The instruction understanding module further includes a text segmentation unit, a semantic filling unit and an intention recognition unit: The text segmentation unit segments the recognized speech text according to the call sign and the command to obtain key fields corresponding to the call sign and the command; The semantic filling unit performs semantic filling on the key fields to obtain a corresponding semantic sequence; The intention recognition unit performs intention recognition based on the semantic sequence, obtains and outputs the final structured data corresponding to the land-air call.

8. The air traffic control response system based on the flight simulator according to claim 2, characterized in that: The special situation processing module activates the predefined special situation scenario according to the trigger command, connects to the flight simulator management computer, generates corresponding special situation instructions, compares and judges the aircraft flight status in the simulator system with the control instruction intention and parameters extracted from the structured data to ensure that the generated air traffic control instructions comply with air traffic control rules.

9. The air traffic control response system based on the flight simulator according to claim 7, characterized in that: The speech synthesis module performs speech simulation based on the air traffic control instruction text, and realizes the mapping between letters, words and special pronunciations by using phoneme permutations and combinations.

10. The air traffic control response system based on the flight simulator according to claim 3, characterized in that: The intelligent evaluation module uses the ROUGE algorithm to calculate the overlapping units between the expected repetition text and the repetition voice text to evaluate the flight trainee's repetition of instructions.

11. The air traffic control response system based on the flight simulator according to claim 3, characterized in that: The intelligent evaluation module uses a keyword evaluation standard to evaluate the flight trainee's recitation of instructions; The evaluation indicators of the keyword evaluation standard include call sign accuracy, action instruction accuracy, parameter accuracy and total accuracy.

12. An air traffic control response training method based on a flight simulator, implemented by using an air traffic control response system based on a flight simulator as claimed in any one of claims 1 to 11, characterized in that: The following steps are involved: A voice input step, receiving input voice audio data; A speech recognition step, which converts speech audio data into speech text data; Instruction text understanding step, the speech text data is processed to generate structured data corresponding to the land-air conversation; The instruction text generation step generates the corresponding air traffic control instruction text according to the structured data; The speech synthesis step is based on converting the air traffic control instruction text into corresponding speech audio data; The speech output step outputs the generated speech audio data.

Images