Automatic report generation method and system based on cockpit speech recognition

Abstract

A system and method for automatically generating a report based on cockpit voice recognition is provided. The system includes a data acquisition module, a speech engine, and a report generator. The data acquisition module is configured to acquire cockpit voice data and vehicle parameter data. The speech engine is configured to convert the cockpit voice data into formatted text data. The speech engine includes: a pre-processing module configured to convert the cockpit voice data into formatted voice data; an ASR module configured to convert the formatted voice data into text data using automatic speech recognition; and a post-processing module configured to reorganize the text data into formatted text data using text processing. The report generator is configured to generate a vehicle standard operating procedure (SOP) compliance report based on the formatted text data and a report template. The vehicle SOP compliance report indicates how the investigated behavior derived from the cockpit voice data complies with the standard operating procedures defined by the report template.

Description

Technical Field

[0001] The present disclosure relates generally to acoustic speech recognition. More specifically, the present disclosure relates to methods and systems for automatically generating reports based on cockpit speech recognition. Background Art

[0002] There are two main types of technical dialogues between pilots: checklists and standard calls. Checklists are a key factor in ensuring flight safety, and they cover the inspection of key points or items at each stage. Standard calls are standardized terms used in normal flight or abnormal situations to ensure effective communication between crew members. Effective "standard calls" can arouse pilots' situational awareness, help them cross-check, coordinate actions, complete standard operating procedures (SOPs) and correct errors in real time, thereby ensuring flight safety.

[0003] Cockpit speech, including conversations between pilots, is recorded into an onboard voice recorder, which is fed into the cockpit voice recorder (CVR). According to airworthiness regulations, all aircraft are equipped with an onboard voice recorder. Today, onboard voice recorders have been developed so that they can record more than 200 hours of cockpit speech.

[0004] However, the investigation found that although cockpit voice is recorded and stored in the airline's data center, the cockpit voice data has not been fully utilized to bring benefits. Today, most airlines use cockpit voice data to investigate flight safety incidents. In addition, most airlines that have installed airborne voice recorders require experienced pilots to sample cockpit voice, mainly by listening to the cockpit voice data recorded in the airborne voice recorder to analyze whether the standard calls and checklists at critical stages of flight (such as altitudes less than 20,000 feet) meet the specifications, and generate corresponding compliance reports based on the cockpit voice monitoring standards.

[0005] The problem is that the manpower and time cost of the pilot monitoring cockpit voice is very high. Therefore, at present, the cockpit voice monitoring rate of airlines is generally between 1% and 2%. If the cockpit voice monitoring rate is increased according to the current method, the corresponding manpower and time cost will need to be greatly increased.

[0006] Therefore, airlines need an effective method to monitor cockpit voice, which can improve the efficiency of cockpit voice monitoring, reduce manpower and time costs, and more effectively discover problems or hidden dangers of the fleet or even a single pilot through data analysis. Flight efficiency and safety can be further improved by customizing training and formulating specific rules.

[0007] Currently, airlines use pilots or employees of the safety oversight department to manually segment and monitor the cockpit voice data obtained through the onboard voice recorder, and conduct manual evaluation according to the cockpit voice evaluation standards. Airlines will regularly summarize SOP compliance reports and generate reports on a weekly, monthly or quarterly basis. In addition, during the investigation of flight safety incidents, some airlines' safety regulators also need to manually convert the cockpit voice of the flight segment into corresponding text with timestamps.

[0008] In addition, some airlines also combine Quick Access Recorder (QAR) information with cockpit voice to generate corresponding jump points using QAR data, allowing monitoring personnel to jump to specific points during the monitoring process. At the same time, key parameters of QAR can also be played back in the form of simulated cockpit display to assist the cockpit voice monitoring process.

[0009] In general, most airlines are using various means to improve monitoring efficiency, but are still manually checking cockpit voice. The efficiency of cockpit voice monitoring and the cost of manpower and time are not satisfactory. Summary of the invention

[0010] The present invention generally discloses a system and method for report generation and a system for digital twin. According to an exemplary embodiment, the present invention discloses a system for automatically generating reports based on cockpit voice recognition. The system may include a data acquisition module, a speech engine, and a report generator. The data acquisition module may be configured to acquire cockpit voice data and vehicle parameter data. The speech engine may be configured to convert the cockpit voice data into formatted text data. The speech engine may include: a pre-processing module configured to convert the cockpit voice data into formatted voice data; an ASR module configured to convert the formatted voice data into text data using automatic speech recognition; and a post-processing module configured to reorganize the text data into formatted text data using text processing. The report generator may be configured to generate a vehicle standard operating procedure (SOP) compliance report based on formatted text data and a report template. The vehicle SOP compliance report may indicate how the investigated behavior derived from the cockpit voice data complies with the standard operating procedures defined by the report template.

[0011] According to another exemplary embodiment, a method for automatically generating a report based on cockpit voice recognition includes: obtaining cockpit voice data and vehicle parameter data; converting the cockpit voice data into formatted text data; and generating a vehicle standard operating procedure (SOP) compliance report based on the formatted text data and a report template, wherein the vehicle SOP compliance report indicates how the investigated behavior derived from the cockpit voice data complies with the standard operating procedure defined by the report template. Converting the cockpit voice data into formatted text data may include: converting the cockpit voice data into formatted voice data; converting the formatted voice data into text data using automatic speech recognition; and reorganizing the text data into formatted text data using text processing.

[0012] According to another exemplary embodiment, a digital twin system for a vehicle includes a data acquisition module, a speech engine, and a digital twin generator. The data acquisition module can be configured to acquire cockpit voice data, vehicle parameter data, and environmental data. The speech engine can be configured to convert the cockpit voice data into text data using automatic speech recognition. The digital twin generator can be configured to generate at least two simulated dynamic visual models based on text data, vehicle parameter data, and environmental data. The at least two simulated dynamic visual models include: a first dynamic visual model that presents virtual electronic devices in the cockpit of a vehicle; and a second dynamic visual model that presents the posture of the vehicle and its external environment.

[0013] According to various embodiments that include the concepts described in the Summary of the Invention, this Summary of the Invention is provided to describe selected concepts in a simplified form, which will be further described in the Detailed Description of the Invention. The Summary of the Invention is not intended to refer to the claims or otherwise indicate the key or essential features of the disclosed subject matter, nor is it intended to help determine the full scope of the disclosed subject matter, which should be appropriately determined with reference to the various embodiments of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] A better understanding of the present disclosure may be obtained from the following detailed description in conjunction with the following drawings, in which:

[0015] Figure 1 is a schematic block diagram of a cockpit speech recognition and analysis system 100 according to an exemplary embodiment.

[0016] Figure 2 is a schematic block diagram of speech engine 102 according to an exemplary embodiment.

[0017] Figure 3 An example of a vehicle SOP compliance report is illustrated in accordance with an exemplary embodiment.

[0018] Figure 4An example of a fleet report is illustrated in accordance with an exemplary embodiment.

[0019] Figure 5 A flow chart of a method 500 for automatic vehicle report generation is shown in accordance with an exemplary embodiment.

[0020] Figure 6 is a schematic block diagram of a digital twin system 600 according to an exemplary embodiment.

[0021] Figure 7 is a schematic block diagram of a digital twin generator 608 according to an exemplary embodiment.

[0022] Figure 8 The diagram illustrates an example of a first image according to an exemplary embodiment.

[0023] Fig. 9 The diagram illustrates an example of a second image according to an exemplary embodiment. DETAILED DESCRIPTION

[0024] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure described below. However, it will be apparent to those skilled in the art that the embodiments of the present disclosure may be practiced without some of these specific details. In other examples, well-known structures and devices are shown in block diagram form to avoid obscuring the basic principles of the embodiments of the present disclosure.

[0025] It should be noted that the terms "consisting of", "consisting of", "having", "having", "including", "comprising", "including", "containing" or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that constitutes, has, includes, or contains a list of elements includes not only those elements, but may also include other elements that are not explicitly listed or inherent to such process, method, article, or apparatus. Elements beginning with "comprising one", "having one", "including one", or "covering one" do not exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises, has, includes, or covers the element, in the absence of more constraints. The terms "a" and "an" are defined as one or more, unless otherwise expressly stated herein.

[0026] Although the embodiments of the subject matter of the invention implemented in the speech recognition system based on aircraft are described in detail below, it should be understood that these embodiments can be implemented in various other types of speech recognition systems and / or devices. These various types of systems and / or devices include but are not limited to systems implemented in aircraft, spacecraft, motor vehicles, watercraft, other types of vehicles and ships, air traffic control (ATC) systems, electronic recording systems, robotic systems, hardware system control and monitoring systems (e.g., for manufacturing, energy production, mining, construction, etc.), computer system control and monitoring systems, and network control and monitoring systems, etc. Therefore, the reference to the speech recognition system based on aircraft herein is not intended to be limited. Based on the description herein, those skilled in the art will understand that other embodiments can be implemented in various other types of systems.

[0027] The subject matter of the present disclosure may be useful for a variety of applications that may benefit from speech recognition in general, particularly automatic SOP compliance analysis, fleet analysis, and digital twin systems based on speech recognition technology. For example, the subject matter of the present disclosure may be used in the context of incident investigation, pilot training, real-time monitoring, aircraft and pilot management, etc. In the following description, example applications of cockpit speech recognition and analysis are described in more detail.

[0028] Figure 1 is a schematic block diagram of a cockpit speech recognition and analysis system 100 according to an exemplary embodiment. The illustrated embodiment of the system 100 includes, but is not limited to, a speech engine 102, an appropriate number of data stores 104, a user interface 106, a report generator 108, a data acquisition module 110, and a graphics and display system 112. These elements of the system 100 may be coupled together by a suitable interconnection architecture 130 that accommodates data communications, transmission of control or command signals, and / or delivery of operating power within the system 100. It should be understood that Figure 1 is a simplified representation of the cockpit speech recognition and analysis system 100, which will be used for purposes of illustration and ease of description, but Figure 1 It is not intended to limit the application or scope of the subject matter in any way. In practice, as will be appreciated in the art, the cockpit speech recognition and analysis system 100 will include other devices and / or components for providing additional functionality and features. Figure 1 The cockpit speech recognition and analysis system 100 is depicted as a single unit, but the various elements and components of the system 100 may be implemented in a distributed manner using any number of physically distinct hardware or devices.

[0029] The data acquisition module 110 obtains cockpit voice data from an onboard voice recorder or database on the aircraft, obtains flight operation control (FOC) data from an airline, and obtains flight parameter data from a QAR or database on the aircraft. The cockpit voice data records conversations between crew members and ground personnel, conversations between crew members, and various sounds appearing in the cockpit (including system alarms and the working sounds of aircraft engines).

[0030] In addition to the onboard voice recorder, the cockpit voice data can also be recorded by other devices, including later installed devices such as voice recorders. The cockpit voice data can be recorded in mono or multi-channel, such as dual-channel, triple-channel, quad-channel and penta-channel. In the example of four channels, the cockpit voice data may include, for example, a left seat channel, a right seat channel, an observation channel and an ambient channel. In some embodiments, the left seat channel voice data and the ambient channel voice data may be recorded in one stereo audio, while the right seat channel voice data and the observation channel voice data may be recorded in another stereo audio. FOC data records information about the flight, such as flight number, crew, departure time and arrival time. Flight parameter data records various aircraft flight parameters. Voice data and flight parameter data are usually time-related.

[0031] In applications other than aircraft, the flight parameter data, flight SOP compliance report, and virtual flight module described below may be referred to as vehicle parameter data, vehicle SOP compliance report, and virtual vehicle module, respectively.

[0032] The speech engine 102 can be implemented with one or more general purpose processors, content addressable memory, digital signal processor, application specific integrated circuit, field programmable gate array, any suitable programmable logic device, discrete gate or transistor logic, discrete hardware components or any combination designed to perform the functions described herein. The processor device can be implemented as a microprocessor, a controller, a microcontroller or a state machine. In addition, the processor device can be implemented as a combination of computing devices (for example, a combination of a digital signal processor and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a digital signal processor core, or any other such configuration). As described in more detail below, the speech engine 102 can implement a speech recognition algorithm, and when working in this environment, can be considered as a speech recognition system.

[0033] The speech engine 102 can obtain cockpit voice data, FOC data and flight parameter data from the data acquisition module 110 and perform speech recognition using automatic speech recognition (ASR) and natural language processing (NLP). The cockpit voice data accumulated by the onboard voice recorder will be converted into formatted text for subsequent analysis in the report generator 108.

[0034] Figure 2 1 is a schematic block diagram of the speech engine 102 according to an exemplary embodiment. The speech engine 102 includes a pre-processing module 1022, an ASR module 1024, and a post-processing module 1026.

[0035] The pre-processing module 1022 generates formatted cockpit voice data based on the data obtained from the data acquisition module 110. In order to obtain the formatted cockpit voice data, the pre-processing module 1022 can perform voice data extraction to extract the voice data of interest from the voice file. The voice data can be extracted based on a certain time period, as long as the information of interest is included. For example, the pre-processing module 1022 can usually extract the voice data of the entire flight. However, if only a landing report is required, the voice data during landing is sufficient, which can reduce the computational burden. In other words, the time period for extracting voice data can be determined based on the event of interest. The event can include any event or procedure in the flight. The embodiments described herein include a report template for each event of interest for the report generator 108. The report template contains a standard operating procedure associated with the event of interest. Therefore, the time period for extracting voice data can be determined based on the selected report template.

[0036] In addition to voice data extraction, the pre-processing module 1022 also performs multi-channel separation. For example, the pre-processing module 1022 can separate stereo audio to obtain separate channel voice data, such as a left seat channel, a right seat channel, an observation channel, and an ambient channel. The pre-processing module 1022 also transcodes the separated voice data into a format that can be used by the system 100. In some embodiments, the pre-processing module 102 can further perform noise reduction on the cockpit voice data, and can perform filtering and formatting on the flight parameter data and FOC data.

[0037] The ASR module 1024 converts the formatted cockpit voice data from the pre-processing module 1022 into text data for each channel. The ASR module 1024 may utilize existing algorithms, such as methods based on HMM (Hidden Markov Model), neural networks, and the like.

[0038] The post-processing module 1026 uses text processing to process and analyze the text data from the ASR module 1024. As a result, the post-processing module 1026 reorganizes the text data to generate formatted text data for subsequent analysis in the report generator 108 using a report template.

[0039] In one implementation, the post-processing module 1026 performs sentence segmentation and punctuation to separate independent sentences. The text processing described herein includes, but is not limited to, natural language processing (NLP). The post-processing module 1026 then merges the independent sentences from each channel based on the timestamp associated with each channel. As a result, the text data of multiple channels can be merged into a text sequence in chronological order.

[0040] In some embodiments, the post-processing module 1026 also performs text classification, intent recognition, and information extraction based on text processing such as natural language processing (NLP) technology. Each sentence in the text is analyzed to obtain information including communication fields (e.g., pilot and ATC communication, pilot and pilot communication), speaker roles (e.g., pilot flight (PF), pilot monitoring (PM), ATC), intent (e.g., attitude change, heading change), and other information such as flight number and frequency number. The information obtained can be included in the formatted text data together with the content of each sentence.

[0041] An example of text data obtained by post-processing module 1026 is shown below.

[0042]

[0043]

[0044]

[0045] The post-processed exemplary text data includes four independent sentences, "pre-taxi checklist", "pre-taxi checklist, engine power on, on, probe heating on, on, anti-icing not needed, off, off", "some airline eight two three three can taxi out, turn right TANGO two five WHISKEY direction ECHO outside, etc." and "can taxi out, turn right TANGO two five WHISKEY direction ECHO outside, etc.", each sentence is associated with a timestamp. The sentences can originate from different sources, i.e., the pilot operating the aircraft, the pilot monitoring the aircraft, the tower or the flight control system, but are arranged in chronological order.

[0046] Post-processing module 1026 can further perform QAR analysis. QAR analysis determines (multiple) specific time points based on the flight parameters specified in the flight parameter data, and associates the determined (multiple) time points with the parts with the same timestamp in the text data. As a result, the (multiple) time points can be used to locate the corresponding parts in the text data. For example, if the investigators need to check the operation performed at or immediately after the altitude of 1000 feet, the post-processing module 1026 can determine the time point associated with the altitude of 1000 feet in the flight parameter data. The determined time point can then be used to find the corresponding part in the text data. In this way, the accuracy and efficiency of the manual inspection for specific SOP inspection items can be improved.

[0047] Return to reference Figure 1 , the data store 104 may be implemented as a RAM memory, a flash memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. In this regard, the data store 104 may be coupled to the speech engine 102 so that the speech engine 102 can read information from the data store 104 and can write information to the data store 104. In an alternative, the data store 104 may be integrated with the speech engine 102. As an example, the speech engine 102 and the data store 104 may reside in an ASIC. In practice, the functions or logical modules / components of the system 100 may be implemented using program code maintained in the data store 104. For example, the speech engine 102, the user interface 106, the report generator 108, the data acquisition module 110, and the graphics and display system 112 may have associated software program components stored in the data store 104. In addition, the data store 104 may be used to store data (e.g., speech training data) used to support the implementation of speech recognition, report templates, and other required data for the speech engine 102. The data store 104 may further store historical data, such as previously generated SOP compliance reports, fleet and crew information for automated fleet analysis.

[0048] The user interface 106 is suitably configured to receive input from a user (e.g., an investigator or administrator) and, in response to the user input, provide appropriate command signals to the system 100. The user interface 106 may include any one or any combination of various known user interface devices or technologies, including, but not limited to: a cursor control device, such as a mouse, trackball, or joystick; a keyboard; buttons; switches; knobs; push rods; or dials. In addition, the user interface 106 may cooperate with the graphics and display system 112 to provide a graphical user interface. Thus, a user may manipulate the user interface 106 by moving a cursor symbol rendered on a display, and a user may use a keyboard to enter text data, etc. For example, a user may manipulate the user interface 106 to initiate or affect the execution of speech recognition and analysis applications, such as by the speech engine 102 and the report generator 108, etc.

[0049] The graphics and display system 112 includes one or more displays and an associated graphics processor. The speech engine 102, the report generator 108, and the graphics and display system 112 cooperate to display, render, or otherwise communicate one or more graphical representations, composite displays, graphic icons, visual symbols, or images associated with the SOP compliance analysis report and / or the fleet analysis report. Embodiments of the system 100 may utilize existing graphics processing techniques and technologies in conjunction with the graphics and display system 112. For example, the graphics and display system 112 may be appropriately configured to support well-known graphics technologies such as, but not limited to, VGA, SVGA, UVGA, etc.

[0050] The report generator 108 analyzes the formatted text data obtained from the speech engine 102 and generates a flight SOP compliance report based on the formatted text data and the report template mentioned above. The flight SOP compliance report indicates how the investigated behavior during the flight derived from the speech data complies with the standard operating procedures defined by the report template. The report template includes one or more check items. The report generator 108 determines whether each of the one or more check items is met by determining whether a match can be found between the check item and certain content in the formatted text data.

[0051] An example of a report template is shown below.

[0052] Table 1

[0053]

[0054]

[0055] The event of interest in Table 1 is "Before Taxi", which includes three check items: "Shout "Before Taxi Checklist", "Do the Before Taxi Checklist", and "Announce Taxi Route to the Crew". The report generator 108 analyzes the formatted text data obtained from the post-processing module 1026 to determine whether a match can be found for each check item.

[0056] The match between the formatted text data and the check items can be determined by text processing, including traditional keyword searches, pattern matching, and the latest technologies such as NLP (natural language processing) for semantic parsing. It should be understood that a 100% match is not required. A lower limit can be predetermined as needed.

[0057] For example, for the checklist item “call out “pre-taxi checklist””, the report generator 108 may perform a keyword search using the keyword “pre-taxi” or a combination of “pre-taxi” and “checklist”.

[0058] For the check item "do the pre-taxi checklist", this may require the pilot to say a series of specific sentences. In other words, the check item "do the pre-taxi checklist" may correspond to a set of sentences. If the content of the sentence set can be found in the text data with a defined matching rate through text processing, it can be determined to be a match.

[0059] For the inspection item "announce the taxi route to the crew", a match can also be determined by searching for keywords such as "taxi out" or "taxi". Alternatively, semantic analysis or intent recognition can be performed based on the trained module to determine whether a certain sentence(s) indicates the meaning of "announce the taxi route to everyone". In an embodiment, the report generator 108 can understand that the meaning of the text "You can taxi out, turn right TANGO two five WHISKEY direction ECHO outside, etc." is "announce the taxi route to everyone", and therefore can find a match between the text "You can taxi out, turn right TANGO two five WHISKEY direction ECHO outside, etc." and the inspection item "announce the taxi route to everyone".

[0060] Due to the fact that the flight parameter data and the cockpit voice data may use two different time bases, synchronization of the flight parameter data and the cockpit voice data is important when performing playback.

[0061] In this case, the system 100 of the present embodiment can synchronize the flight parameter data and the cockpit voice data based on a system alarm or notification, which records and indicates the selected flight parameter in the form of voice data. Since the system alarm or notification is recorded by the onboard voice recorder, the processor (not shown) of the system 100 can identify the corresponding content of the system alarm or notification in the formatted text data. The processor can then determine the time offset between the flight parameter data and the cockpit voice data, and synchronize based on the determined time offset.

[0062] For example, when the aircraft descends to 1000 feet, 500 feet, and 50 feet, the flight control system of the aircraft may respectively notify "1000 feet", "500 feet", and "50 feet". For example, for an altitude of 1000 feet, the system notifies that the voice of "1000 feet" will be recorded by the onboard voice recorder and then converted into text data by the voice engine 102. As a result, the content "1000 feet" will be included in the formatted text in association with its time point. On the other hand, the radio altitude parameter of 1000 feet can be found in the flight parameter data. The formatted text can thus be synchronized with the flight parameter data. Therefore, the time offset can be determined by comparing the time point of the content "1000 feet" in the formatted text with the time point of the radio altitude of 1000 feet in the flight parameter data. In this way, the formatted text, and then the cockpit voice data, can be synchronized with the flight parameter data based on the determined time offset. In some embodiments, the time offset can be determined for multiple flight parameters, such as altitudes of 1000 feet, 500 feet, and 50 feet. Synchronization can be performed based on the determined multiple time offsets to increase the accuracy of synchronization. Processing of multiple time offsets includes, but is not limited to, averaging.

[0063] Figure 3 The figure shows an example of a generated flight SOP compliance report. The match between the formatted text data and the inspection items can be determined in different ways as described above. The inspection items in the flight SOP compliance report can be associated with the time points described above so that the investigator can easily find a specific part.

[0064] In some embodiments, the report generator 108 may also generate a fleet report based on multiple flight SOP compliance reports for the same fleet. The flight SOP compliance reports may be stored as historical data in the data store 104. Examples of fleet reports are described in Figure 4 The fleet report provides the statistics of the flight SOP compliance report of the fleet within a certain period of time.

[0065] The flight SOP compliance report and the fleet report may be displayed on the graphics and display system 112. However, it should be understood that the form and format of the report template, the flight SOP compliance report and the fleet report are non-limiting and may be designed as desired.

[0066] Figure 5 A flow chart of a method 500 for automatic vehicle report generation according to an exemplary embodiment is shown. In step 502, the pre-processing module 1022 obtains voice data from the data acquisition module 110 and generates formatted voice data through voice data extraction, multi-channel separation and transcoding. The voice data can be extracted based on a certain time period (e.g., takeoff phase). In some embodiments, the "specific time period" can be determined based on an event of interest and then based on a report template associated with the event of interest. The pre-processing module 1022 can further perform noise reduction.

[0067] In step 504, the formatted speech data from the pre-processing module 1022 is converted into text data by the ASR module 1024 using an ASR method.

[0068] In step 506, the converted text data is processed by the post-processing module 1026. The post-processing module 1026 uses natural language processing to perform sentence segmentation and punctuation. The post-processing module 1026 then merges the independent sentences from each channel based on the timestamp associated with each channel. The text data of multiple channels are merged in a text sequence in chronological order as formatted text data.

[0069] In step 508, the report generator 108 generates a flight SOP compliance report based on the formatted text data and the report template. The report template includes one or more inspection items. The report generator 108 analyzes the formatted text data to determine whether a match can be found for each inspection item. Taking into account the differences in pronunciation and synonyms, a match rate exceeding a predetermined threshold is considered a "match (conformity)". In some embodiments, the report generator 108 can also generate a fleet report based on multiple flight SOP compliance reports for the same fleet.

[0070] Compared with the traditional keyword search method, which searches for keywords from voice data and has low accuracy and information utilization, the embodiment described herein converts all extracted voice data into formatted text data and generates (multiple) reports based on the formatted text data. Flight parameter data and report templates help to extract the voice data of interest and locate (multiple) specific parts of the text data. Therefore, a balance between accuracy and computational load can be achieved. In addition, the method of the embodiment can reduce the manual intervention of cockpit voice inspection and automatically perform the whole process of data preparation, cockpit voice ASR recognition, text analysis and report generation. As a result, labor costs can be reduced by reducing the participation of pilots.

[0071] Another embodiment relates to a digital twin system based on acoustic speech recognition. Currently, when airline flight safety oversight departments conduct monitoring activities and incident investigations, they are usually required to convert cockpit voice into text through manual listening and translation to check whether the pilot follows the standard operating procedure (SOP) or is distracted by other conversations, check the recorded flight parameters to determine whether any abnormal operations or failures occur during the flight, and check the historical data on weather and terrain during the flight to see if there are any abnormal conditions affecting the flight. These three expendable tasks are always performed separately and require experienced personnel. However, even without considering resource issues and human factors, some key information may be lost because the pilot (voice), aircraft (flight parameters) and environment (terrain and weather) are not investigated synchronously.

[0072] Currently, when an aircraft encounters some technical problems, sometimes the crew needs to contact the experts at the Airline Operations Center (AOC) for help via VHF or SATCOM (satellite communication) link, only verbally describing the problems they are facing, which limits the quality and efficiency of information transmission. This makes it difficult for ground experts to fully understand the situation in the air, thus preventing them from providing accurate and adequate responses quickly.

[0073] Furthermore, when trying to obtain a type certificate for a new aircraft, it needs to pass all the required flight tests, which can generate a lot of data on different parameters. The data will be recorded and passed down to the ground for analysis by experts in each aircraft system. For the experts, understanding the overall situation of the flight (aircraft condition and pilot experience) through highly synchronized simulations can be of great help in the analysis.

[0074] The embodiments described herein use software-based ultra-highly integrated simulation to create digital twins. The digital twin system provides a series of virtual visual models of highly complex cockpit systems and components. The system uses cockpit voice recordings and automatic and corresponding transcoded text subtitles. In addition, the system synchronizes environmental (terrain and weather) information during flight to generate a three-dimensional virtual aircraft model that displays the environmental process of the flight.

[0075] Embodiments of the digital twin system enable flight monitoring either post-flight analysis (playback) or in real-time when a high-bandwidth air-to-ground data link such as 4G / 5G ATG or SATCOM is available.

[0076] Figure 6 6 is a schematic block diagram of a digital twin system 600 according to an exemplary embodiment. The illustrated embodiment of the system 600 includes, but is not limited to, a speech engine 602, an appropriate amount of data storage 604, a user interface 606, a digital twin generator 608, a data acquisition module 610, and a graphics and display system 612. These elements of the system 600 can be coupled together by a suitable interconnection architecture 630 that accommodates data communications within the system 600, transmission of control or command signals, and / or delivery of operating power.

[0077] The data acquisition module 610 acquires cockpit voice data, flight parameter data, and environmental data. As described above, cockpit voice data records conversations between crew members and ground personnel, conversations between crew members, and various sounds that appear in the cockpit (including the working sounds of aircraft engines) and can be obtained from the onboard voice recorder on the aircraft. Flight parameter data records various aircraft flight parameters and can be obtained from the quick access recorder on the aircraft. Environmental data includes weather data and terrain data that can be obtained from the radar on the aircraft. Cockpit voice data, flight parameter data, and environmental data can be obtained in real time or subsequently (e.g., during simulation). Real-time data can be used for real-time monitoring. Data obtained later can be used for event investigation, pilot training, etc.

[0078] The speech engine 602 converts the cockpit speech data into text data using ASR. It is not necessary to perform speech data extraction, but to convert all speech data, especially for real-time applications. In some embodiments, the speech engine 602 can also identify the role of the speaker for each sentence based on the converted text through machine learning. The identified speaker is associated with the corresponding sentence.

[0079] In some embodiments, the speech engine 602 can be installed on the aircraft and send the formatted text data to the ground via a wireless link such as 4G / 5G ATG or SATCOM. Real-time applications will benefit from such an arrangement because sending text data to the ground can greatly save bandwidth compared to sending voice data.

[0080] The data store 604 , the user interface 606 , and the graphics and display system 612 are substantially similar to the data store 104 , the user interface 106 , and the graphics and display system 112 described above.

[0081] The digital twin generator 608 generates at least two simulated dynamic visual models based on the formatted text data from the speech engine 602, the flight parameter data and the environmental data from the data acquisition module 610. The first dynamic visual model presents virtual electronic equipment in the cockpit, such as display pages (such as the primary flight display (PFD), the missed approach point (MAP), the navigation display (ND), SYNC, etc.), the multi-function control panel (MCP), and the flight control devices (such as the throttle, the control stick, the rudder pedals, etc.). The second dynamic visual model presents an environment close to reality. The environment close to reality reflects the attitude of the aircraft and its external environment, including weather and terrain. The generated dynamic visual model can be displayed by the graphics and display system 612.

[0082] Figure 7 is a schematic block diagram of a digital twin generator 608 according to an exemplary embodiment. Figure 7 The digital twin generator 608 shown includes a virtual cockpit module 6082 and a virtual flight module 6084 .

[0083] The virtual cockpit module 6082 generates a first image using the flight parameter data, the first image simulating a virtual electronic device. Figure 8 An example of a first image according to an exemplary embodiment is shown. In real-time applications, the readings of the virtual electronics are updated based on the flight parameter data from the aircraft. In playback applications, the flight parameter data may be obtained from a database.

[0084] The virtual flight module 6084 generates a second image based on the flight parameter data and the environment data, and the second image provides a close-to-reality environment to reflect the attitude and environment of the aircraft. The flight parameter data is used to determine the attitude of the aircraft, and the environment data is used to simulate the environment. Fig. 9 The figure shows an example of a second image according to an exemplary embodiment (Copyright belongs to Commercial Aircraft Corporation of China). Fig. 9 In the example shown, the text data obtained from the speech engine 602 is incorporated into the second image in the form of text subtitles. In the playback application, speech data can also be played back.

[0085] In playback applications, synchronization of text data, flight parameter data, and environmental data is important. Environmental data inherently contains time information, such as Greenwich Mean Time. For text data, the virtual flight module 6084 identifies the specific content of system alerts or notifications in the formatted text data to synchronize with the flight parameter data, which system alerts or notifications indicate flight parameters.

[0086] Taking an altitude of 1000 feet as an example, when the radio altitude of the aircraft reaches 1000 feet, the flight control system will notify "1000 feet". The voice notified by the system will be recorded by the onboard voice recorder and then converted into text data by the voice engine 602. As a result, the content "1000 feet" will be included in the formatted text in association with its time point. On the other hand, the radio altitude parameter of 1000 feet can be found in the flight parameter data. Therefore, the time offset can be determined by comparing the time point of the content "1000 feet" in the formatted text with the time point of the radio altitude of 1000 feet in the flight parameter data. Thus, the formatted text can be synchronized with the flight parameter data based on the determined time offset. In some embodiments, the time offset can be determined for multiple flight parameters, such as altitudes of 1000 feet, 500 feet, and 50 feet. Synchronization can be performed based on the determined multiple time offsets to improve the accuracy of synchronization. The calculation of multiple time offsets includes but is not limited to averaging.

[0087] Furthermore, since the flight parameter data inherently contains time information, it can be synchronized with the environmental data. In this way, text data, flight parameter data and environmental data can be synchronized.

[0088] The following examples relate to specific technical embodiments and point out specific features, elements, or steps that may be used or otherwise combined in implementing these embodiments.

[0089] In one example, a system for automatically generating a report based on cockpit speech recognition is provided, the system comprising: a data acquisition module configured to acquire cockpit speech data and flight parameter data; a speech engine configured to convert the cockpit speech data into formatted text data, the speech engine comprising: a pre-processing module configured to convert the cockpit speech data into formatted speech data; an ASR module configured to convert the formatted speech data into text data using automatic speech recognition; a post-processing module for reorganizing the text data into formatted text data through text processing; and a report generator configured to generate a flight standard operating procedure (SOP) compliance report based on the formatted text data and a report template, wherein the flight SOP compliance report indicates how the investigated behavior derived from the cockpit speech data complies with the standard operating procedure defined by the report template.

[0090] In one example of the system, the pre-processing module is further configured to extract voice data based on events of interest associated with the report template.

[0091] In one example of the system, the pre-processing module is configured to perform multi-channel separation on the cabin voice data, and wherein the post-processing module is configured to merge the text data of each channel in time sequence.

[0092] In one example of the system, the post-processing module is configured to perform sentence segmentation and punctuation using text processing to separate individual sentences.

[0093] In one example of the system, the report template includes at least one inspection item, and the report generator is further configured to determine whether a match can be found between each of the at least one inspection item and one or more portions of the formatted text data.

[0094] In one example of the system, the post-processing module is also configured to determine one or more specific time points based on (multiple) specified flight parameters in the flight parameter data, and associate the determined (multiple) time points with (multiple) portions in the text data having the same timestamp.

[0095] In one example of the system, the post-processing module is further configured to: analyze each sentence in the text data through text processing to obtain information including one or more of communication domain, speaker role, intent, and other information such as flight number and frequency number; and include the obtained information in the formatted text data.

[0096] In one example of the system, the report generator is configured to identify a match when a match rate reaches a predetermined threshold.

[0097] In one example of the system, the report generator is further configured to generate a fleet report based on a plurality of historical flight SOP compliance reports for the fleet.

[0098] In one example of the system, the system further includes a processor configured to synchronize the cockpit voice data with the flight parameter data based on content of a system alert or notification indicative of the flight parameter in the formatted text data.

[0099] In one example of the system, the processor is further configured to determine a time offset by comparing a time point of content of a system alert or notification in the formatted text data with a time point of an indicated flight parameter in the flight parameter data, and to synchronize the cockpit voice data with the flight parameter data based on the determined time offset.

[0100] In one example of the system, the processor is further configured to determine a plurality of time offsets for the plurality of flight parameters; and synchronize the cockpit voice data with the flight parameter data based on the determined plurality of time offsets.

[0101] In another example, a method for automatically generating a report based on cockpit voice recognition is provided, the method comprising: acquiring cockpit voice data and flight parameter data; converting the cockpit voice data into formatted text data, including: converting the cockpit voice data into formatted voice data; converting the formatted voice data into text data using automatic speech recognition; reorganizing the text data into formatted text data using text processing; and generating a flight standard operating procedure (SOP) compliance report based on the formatted text data and a report template, wherein the flight SOP compliance report indicates how the investigated behavior derived from the cockpit voice data complies with the standard operating procedure defined by the report template.

[0102] In another example, a digital twin system for a vehicle is provided, comprising: a data acquisition module configured to acquire cockpit voice data, flight parameter data, and environmental data; a speech engine configured to convert the cockpit voice data into text data using automatic speech recognition; and a digital twin generator configured to generate at least two simulated dynamic visual models based on the text data, flight parameter data, and environmental data, wherein the at least two simulated dynamic visual models include a first dynamic visual model and a second dynamic visual model, the first dynamic visual model presenting virtual electronic devices in a cockpit of the vehicle, and the second dynamic visual model presenting the posture of the vehicle and its external environment.

[0103] In one example of a digital twin system, a digital twin generator includes: a virtual cockpit module configured to generate a first dynamic visual model based on flight parameter data; a virtual flight module configured to generate a second dynamic visual model based on text data, flight parameter data, and environmental data, wherein the text data is incorporated into the second dynamic visual model in the form of text subtitles, and wherein the flight parameter data is used to determine the attitude of the vehicle.

[0104] In one example of a digital twin system, a speech engine is installed on a vehicle, and text data as well as flight parameter data are acquired in real time from the vehicle via a wireless link.

[0105] In one example of a digital twin system, a speech engine is installed locally, and wherein cockpit speech data and vehicle parameter data are obtained in real time from the vehicle via a wireless link.

[0106] In one example of a digital twin system, the virtual flight module is configured to synchronize the text data with the flight parameter data based on the content of a system alert or notification indicative of the flight parameter in the text data.

[0107] In one example of a digital twin system, the virtual flight module is also configured to determine a time offset by comparing a time point of the content of a system alert or notification in the text data with a time point of an indicated flight parameter in the flight parameter data, and synchronize the text data with the flight parameter data based on the determined time offset.

[0108] In one example of the digital twin system, the virtual flight module is further configured to: determine multiple time offsets for multiple flight parameters; and synchronize the text data with the flight parameter data based on the determined multiple time offsets.

[0109] In one example of a digital twin system, the virtual flight module is also configured to synchronize flight parameter data with environmental data having inherent time information.

[0110] In yet another example, a computer-readable storage medium is provided, storing instructions, which, when executed by a processor, cause the processor to perform the method for automatically generating a report based on cockpit speech recognition as described above.

[0111] The embodiments of the mechanisms disclosed herein may be implemented in hardware, software, firmware, or a combination of such implementation methods. Embodiments of the present invention may be implemented as a computer program or program code executed on a programmable system, the programmable system comprising at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device.

[0112] It is to be understood that the steps of the methods described herein may be implemented by one or more general or special purpose processors (or “processing devices”), such as microprocessors, digital signal processors, custom processors, and field programmable gate arrays (FPGAs), and uniquely stored program instructions (including both software and firmware) that control the one or more processors to implement some, most, or all of the functionality of the methods and / or apparatus described herein, in conjunction with certain non-processor circuits.

[0113] In addition, the systems and methods described herein can be implemented as computer-readable storage media having computer-readable code stored thereon for programming a computer (e.g., including a processor) to perform the methods described and claimed herein. Examples of such computer-readable storage media include, but are not limited to, hard disks, CD-ROMs, optical storage devices, magnetic storage devices, ROMs (read-only memories), PROMs (programmable read-only memories), EPROMs (erasable programmable read-only memories), EEPROMs (electrically erasable programmable read-only memories), and flash memory. In addition, it is expected that those of ordinary skill in the art will be able to easily generate such software instructions and programs and ICs with minimal experimentation, although with potentially significant effort and many design choices motivated by, for example, available time, current technology, and economic considerations, when guided by the concepts and principles disclosed herein.

[0114] Specific embodiments have been described in the foregoing description. However, it will be appreciated by those skilled in the art that various modifications and changes may be made without departing from the scope of the present disclosure as set forth in the claims. Therefore, the specification and the drawings are to be regarded in an illustrative rather than a restrictive sense, and all such modifications are intended to be included within the scope of the present teachings. Additionally, the described embodiments / examples / implementations should not be interpreted as mutually exclusive, but rather as potentially combinable, if such combinations are permitted in any way. In other words, any feature disclosed in any of the foregoing embodiments / examples / implementations may be included in any of the other foregoing embodiments / examples / implementations.

[0115] These benefits, advantages, solutions to problems, and any (multiple) elements that may make any benefit, advantage, or solution occur or become more prominent are not to be construed as critical, required, or essential features or elements of any or all claims. The present disclosure is limited solely by the appended claims, including any amendments made during the pendency of this application and all equivalents of those claims as issued.

[0116] The abstract of the present disclosure is provided to allow the reader to quickly ascertain the nature of the present technical disclosure. This abstract is submitted with the understanding that it will not be used to interpret or limit the scope or meaning of the claims. In addition, in the above-mentioned detailed description, it can be seen that various features are grouped together in various embodiments for the purpose of integrating the present disclosure. This method of disclosure should not be interpreted as reflecting the intention that the claimed embodiments require more features than those explicitly stated in the claims. On the contrary, as reflected in the following claims, the subject matter of the invention lies in less than all the features of a single disclosed embodiment. Therefore, the following claims are thus incorporated into the detailed description, wherein each claim represents itself as a subject matter claimed separately.

Claims

1. A system for automatically generating a report based on cockpit speech recognition, comprising: A data acquisition module, configured to acquire cockpit voice data and vehicle parameter data; A speech engine is configured to convert the cockpit speech data into formatted text data, the speech engine comprising: a preprocessing module configured to convert the cockpit voice data into formatted voice data; An ASR module configured to convert the formatted speech data into text data using automatic speech recognition; and a post-processing module configured to reorganize the text data into formatted text data using text processing, wherein the formatted text data includes one or more of communication domain, speaker role, intent, and other information including flight number and communication frequency; and A report generator is configured to generate a vehicle standard operating procedure (SOP) compliance report based on the formatted text data and a report template, wherein the vehicle SOP compliance report indicates the degree of compliance of the investigated behavior derived from the cockpit voice data with the standard operating procedure defined by the report template.

2. The system according to claim 1, characterized in that The pre-processing module is further configured to extract voice data based on events of interest associated with the report template.

3. The system according to claim 1, characterized in that The pre-processing module is configured to perform multi-channel separation on the cabin voice data, and The post-processing module is configured to merge the text data of each channel in time sequence.

4. The system according to claim 1, characterized in that The post-processing module is configured to perform sentence segmentation and punctuation using text processing to separate individual sentences.

5. The system according to claim 1, wherein: The report template includes at least one inspection item. The report generator is further configured to determine whether a match can be found between each of the at least one inspection item and one or more portions of the formatted text data.

6. The system according to claim 5, characterized in that The post-processing module is further configured to determine one or more specific time points based on one or more specified vehicle parameters in the vehicle parameter data, and associate the determined one or more time points with one or more portions of the text data having the same time stamp.

7. The system according to claim 5, characterized in that The report generator is configured to identify a match when a matching rate reaches a predetermined threshold.

8. The system of claim 1, wherein: The report generator is further configured to generate a fleet report based on a plurality of historical vehicle SOP compliance reports for the fleet.

9. The system of claim 1, further comprising a processor configured to synchronize the cockpit voice data with the vehicle parameter data based on content of a system alert or notification indicative of a vehicle parameter in the formatted text data.

10. The system according to claim 9, characterized in that The processor is further configured to determine a time offset by comparing a time point of content of a system alert or notification in the formatted text data with a time point of an indicated vehicle parameter in the vehicle parameter data, and to synchronize the cabin voice data with the vehicle parameter data based on the determined time offset.

11. The system of claim 10, wherein: The processor is further configured to: determining a plurality of time offsets for a plurality of vehicle parameters; and The cockpit voice data is synchronized with the vehicle parameter data based on the determined plurality of time offsets.

12. A method for automatically generating a report based on cockpit speech recognition, comprising: Acquire cockpit voice data and vehicle parameter data; Converting the cockpit voice data into formatted text data includes: converting the cockpit voice data into formatted voice data; converting the formatted speech data into text data using automatic speech recognition; and reorganizing the text data into the formatted text data using text processing, wherein the formatted text data includes one or more of communication domain, speaker role, intent, and other information including flight number and communication frequency; and A vehicle standard operating procedure (SOP) compliance report is generated based on the formatted text data and a report template, wherein the vehicle SOP compliance report indicates how the investigated behavior derived from the cockpit voice data complies with a standard operating procedure defined by the report template.

13. A digital twin system for a vehicle, comprising: A data acquisition module, configured to acquire cockpit voice data, vehicle parameter data and environment data of the vehicle; a speech engine configured to convert the cockpit speech data into text data using automatic speech recognition; and a digital twin generator configured to generate at least two simulated dynamic visual models based on the text data, the vehicle parameter data and the environment data, The at least two simulated dynamic visual models include: a first dynamic visual model that presents virtual electronic equipment in the cockpit of the vehicle; and a second dynamic visual model that presents the posture of the vehicle and its external environment, wherein the digital twin generator includes: a virtual cockpit module configured to generate the first dynamic visual model based on the vehicle parameter data; and a virtual vehicle module configured to generate the second dynamic visual model based on the text data, the vehicle parameter data and the environment data, wherein the text data is incorporated into the second dynamic visual model in the form of text subtitles, The vehicle parameter data is used to determine the posture of the vehicle.

14. The system of claim 13, wherein: The speech engine is installed on the vehicle, and Wherein, the text data and the vehicle parameter data are obtained in real time from the vehicle via a wireless link.

15. The system of claim 13, wherein: The speech engine is installed locally, and Wherein, the cockpit voice data and the vehicle parameter data are obtained in real time from the vehicle via a wireless link.

16. The system of claim 13, wherein: The virtual vehicle module is configured to synchronize the text data with the vehicle parameter data based on content of a system alert or notification indicative of a vehicle parameter in the text data.

17. The system of claim 16, wherein: The virtual vehicle module is further configured to determine a time offset by comparing a time point of the content of the system alarm or notification in the text data with a time point of the indicated vehicle parameter in the vehicle parameter data, and synchronize the text data with the vehicle parameter data based on the determined time offset.

18. The system of claim 17, wherein: The virtual vehicle module is further configured to: determining a plurality of time offsets for a plurality of vehicle parameters; and The text data is synchronized with the vehicle parameter data based on the determined plurality of time offsets.

19. The system of claim 18, wherein: The virtual vehicle module is further configured to synchronize the vehicle parameter data with the environmental data having inherent time information.

20. A computer-readable storage medium storing instructions, which, when executed by a processor, cause the processor to perform the method of claim 12.

Images