A method, medium and device for monitoring an out-of-limit event based on multi-source real-time data

Abstract

The present application relates to the field of over-limit event monitoring, in particular to an over-limit event monitoring method based on multi-source real-time data, medium and equipment. By fusing QAR data, flight operation information and SOP, an over-limit event database is established, and prompt and positioning node information are configured for each to-be-monitored event. Aircraft QAR information is collected in real time, and prompt start time t1, positioning start time t2 and positioning end time t3 of the event are generated. According to the comparison result of cabin sound information and SOP standard voice call information, it is determined in real time whether the event is over-limit. This method can monitor each over-limit event in real time, improve the calculation efficiency and the timeliness of early warning.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of over-limit event monitoring, in particular to an over-limit event monitoring method based on multi-source real-time data, a medium and equipment. BACKGROUND

[0002] Cockpit voice and flight data are two core data of aircraft, and are also important information sources for civil aviation unsafe event investigation, accident incident investigation and flight quality monitoring. Post-analysis application based on cockpit voice and flight data is relatively mature, but it cannot timely discover potential risks in the flight process. Moreover, in flight safety monitoring, various data resources are not fully integrated, and real-time risk identification is not combined with standard operating procedures (SOP), so it is difficult to timely detect and prevent over-limit events in the flight process. Cockpit voice over-limit event refers to the output result of checking and judging the compliance of pilot standard call according to the standard operating procedure (SOP). Therefore, it is impossible to determine the possible safety risks of the aircraft in the flight process in time, and thus the safety of the aircraft cannot be better guaranteed. SUMMARY

[0003] In view of one of the above technical problems, the technical solution adopted by the present application is:

[0004] According to one aspect of the present application, an over-limit event monitoring method based on multi-source real-time data is provided, the method comprising the following steps:

[0005] Matching the flight monitoring information obtained in the current data collection period with the prompt node information and the positioning node information corresponding to each to-be-monitored event in the over-limit event database to generate the prompt start time t1, the positioning start time t2 and the positioning end time t3 corresponding to each to-be-monitored event; t1

[0006] If there is a new complete and effective speech recognition result in the real-time cockpit voice information collected in the current data collection period, then t1, t2 and t3 corresponding to each to-be-monitored event are traversed.

[0007] If the end time T of the last complete sentence in the speech recognition result of all valid speech channels obtained in the current data collection period satisfies the condition T∈(t1, t2], then the standard speech call information required in the SOP corresponding to the to-be-monitored event is generated.

[0008] If T satisfies the condition T∈(t2, t3] corresponding to any to-be-monitored event, each complete voice recognition information acquired in the time window [t2, T] is compared with the standard voice call information required in the SOP corresponding to the to-be-monitored event;

[0009] If T satisfies the condition T>t3 corresponding to any to-be-monitored event, if all voice recognition information in all voice comparison results of the to-be-monitored event in the time window [t2, t3] is different from the standard voice call information required in the SOP corresponding to the to-be-monitored event, the to-be-monitored event is determined as an overrun event of the flight.

[0010] According to a second aspect of the present application, a non-transitory computer readable storage medium is provided, the non-transitory computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the above-mentioned overrun event monitoring method based on multi-source real-time data.

[0011] According to a third aspect of the present application, an electronic device is provided, which comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to implement the above-mentioned overrun event monitoring method based on multi-source real-time data.

[0012] The present application has at least one of the following beneficial effects:

[0013] In the present application, by fusing multiple data sources, including QAR data, flight operation information and SOP, a database of over-limit events corresponding to the current flight (i.e. aircraft) can be established in advance, which includes each to-be-monitored event (i.e. over-limit event) that needs to be monitored in the entire flight operation process, and the corresponding prompt node information and positioning node information are configured for each to-be-monitored event, so as to generate the prompt start time t1, the positioning start time t2 and the positioning end time t3 corresponding to each to-be-monitored event in real time according to the real-time QAR information of the aircraft obtained in each data acquisition cycle during the actual operation of the flight. And t1, t2 and t3 form the over-limit event monitoring action execution period corresponding to each to-be-monitored event. Then, according to the comparison result between the cockpit voice data obtained in real time in the corresponding determination period and the standard voice call information required in the SOP, the final determination of whether the to-be-monitored event is over-limit is generated. In the present application, by comparing the QAR data segment transmitted by the onboard system in real time with the over-limit event database in real time, different over-limit determination periods corresponding to each to-be-monitored event can be opened in real time, and the corresponding over-limit monitoring task can be started. Therefore, each over-limit event in the flight process can be decoupled and monitored in real time, thereby improving the calculation efficiency and outputting the over-limit result in time.

[0014] In addition, the present application realizes real-time reception and processing of aircraft multi-source data, compared with the traditional post-analysis mode, the over-limit event can be found in time during the flight process, greatly improving the timeliness of flight risk warning, gaining valuable time for taking emergency measures, and effectively reducing the possibility of flight accidents. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0016] Figure 1 A flowchart of a multi-source real-time data-based over-limit event monitoring method provided by an embodiment of the present application.

[0017] Figure 2 A principle diagram of over-limit event identification processing provided by an embodiment of the present application.

[0018] Figure 3 A flowchart of an effective voice channel acquisition method provided by another embodiment of the present application.

[0019] Figure 4The structural schematic diagram of an electronic device provided for another embodiment of the present application is shown. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present application.

[0021] As a possible embodiment of the present application, as shown in Figure 1 A method for monitoring over-limit events based on multi-source real-time data is provided, and the method comprises the following steps:

[0022] S100: Matching flight monitoring information acquired in a current data acquisition period with prompt node information and positioning node information corresponding to each to-be-monitored event in an over-limit event database, to generate prompt start time t1, positioning start time t2 and positioning end time t3 corresponding to each to-be-monitored event. t1 < t2 < t3; the to-be-monitored event is a standard operation task including a shouting task in the SOP corresponding to the current flight; the prompt node information and the positioning node information are time generation strategy information formed by cabin sound information and / or QAR information (in the field of aviation, QAR information is a kind of data for recording flight in real time) generated when the SOP preset operation corresponding to the to-be-monitored event is executed; the flight monitoring information includes one or a combination of any multiple of flight operation information, real-time cabin sound information and real-time QAR information. The QAR information in the embodiment is an engineering value after QAR decoding.

[0023] Each to-be-monitored event in the embodiment is actually each necessary standard shouting event of a pilot in a flight process according to a standard operation procedure (SOP). Some to-be-monitored events are shown in Table 1 as follows:

[0024] Table 1

[0025]

[0026] Each row in Table 1 is a to-be-monitored event, wherein the standard shouting content-PF / L indicates that the standard shouting content needs to be shouted by the captain, and the standard shouting content-PM / R indicates that the standard shouting content needs to be shouted by the co-pilot.

[0027] Specifically, in this step, the corresponding determination information is obtained by receiving the QAR data segment and the cockpit voice data stream downloaded in real time by the receiver system. The specific data collection period can be flexibly set according to the actual scene, such as 1 second, and the monitoring frequency of the overrun event can be flexibly adjusted.

[0028] Before this step is executed, the overrun event database corresponding to the flight of the aircraft needs to be established in priority. Specifically, the overrun event database can be established according to the flight operation information, the text information identified from the cabin voice information, the SOP, and the historical QAR engineering value after decoding. Specifically, the overrun event database is actually to establish the corresponding prompt node information and positioning node information for each to-be-monitored event. The prompt node information is used to locate the pre-warning time corresponding to the to-be-monitored event, that is, t1. The positioning node information is used to locate the call content extraction time corresponding to the to-be-monitored event, that is, t2 and t3.

[0029] The prompt node information and the positioning node information are both time generation logic strategies set according to the internal logic between the related information in the actual flight process. For example, the generation time of a certain QAR information can be determined as t1, t2, or t3. Or the QAR information relied on by t2 and t3 for positioning can not be the same, and even can rely on voice for positioning. Each t1, t2, and t3 can be generated based on the corresponding time offset set by the QAR information.

[0030] Specifically, the QAR retrieval time (t1, t2, and t3) corresponding to each to-be-monitored event in S100 can be obtained according to the following steps:

[0031] S101: The flight monitoring information obtained according to the current data collection period is matched with the time generation strategy information limited by any prompt node information corresponding to the flight in the overrun event database.

[0032] S102: If the currently collected flight monitoring information satisfies the time generation strategy information limited by any prompt node information, t1 of the to-be-monitored event corresponding to the prompt node information is generated.

[0033] S103: The flight monitoring information obtained according to the current data collection period is matched with the time generation strategy information limited by any positioning node information corresponding to the flight in the overrun event database.

[0034] S104: If the currently collected flight monitoring information satisfies the time generation strategy information limited by any positioning node information, t2 and t3 of the to-be-monitored event corresponding to the positioning node information are generated.

[0035] Specifically, the time generation strategy information in the prompt node information and the positioning node information can be obtained according to the following steps:

[0036] S110: Determine the real-time cabin sound information and the QAR information corresponding to the current flight by using the flight identity information (such as the flight number) included in the flight operation information.

[0037] S120: Determine the operation description information recognized from the real-time cabin sound information corresponding to the current flight as the local air-ground dialogue positioning information.

[0038] In actual use scenarios, the over-limit events of multiple flights are monitored at the same time, so the flight number is needed to determine the real-time cabin sound information and the QAR information corresponding to the current flight from multiple cabin sound information and QAR information, so as to facilitate subsequent accurate matching.

[0039] S130: Match the local air-ground dialogue positioning information with the real-time QAR information of the current flight obtained subsequently.

[0040] In actual flight operation processes, the pilot often performs some operations and has an air-ground dialogue with the ground air traffic control center for operation application or confirmation. For example, due to weather conditions, air traffic or other reasons, the flight plan needs to be changed (such as adjusting the flight height or route), and the pilot must communicate with the relevant air traffic control unit and obtain approval. Then, after obtaining the approval, the corresponding operation is performed, and then the corresponding operation QAR information is generated in the subsequent short period.

[0041] Based on the above characteristics, the operation description information recognized from the real-time cabin sound information is used to determine the local air-ground dialogue positioning information in advance, and the timing of matching the real-time QAR information of the current flight obtained subsequently.

[0042] S140: If the operation information in the real-time QAR information is the same as the operation description information in the local air-ground dialogue positioning information, the generation time of the real-time QAR information is taken as the positioning time, and the positioning time is used to generate t1, t2 and t3 corresponding to the monitored event.

[0043] In this embodiment, the flight operation information (such as the flight number) is used in combination with the real-time cabin sound recognition text to position the local air-ground dialogue, and then the positioning time is generated according to the local air-ground dialogue combined with the QAR. These positioning times are the key dependencies for generating t1, t2 or t3 in the time generation strategy. Specifically, the corresponding time generation strategy information can be generated according to the logic between the positioning time and t1, t2 and t3.

[0044] In addition, the time generation strategy information of the prompt node information or the positioning node information corresponding to each to-be-monitored event in the overrun event database can also be constructed according to the flight operation information and the historical QAR engineering value. The generation of t1, t2 or t3 depends on the determination QAR information corresponding to a key node.

[0045] According to the determination QAR information corresponding to the key node, t1, t2 and t3 corresponding to each to-be-monitored event can be generated in time and accurately in subsequent actual use, that is, the QAR retrieval moment in the formula. Figure 2

[0046] In this embodiment, in order to more clearly show the entire execution process of the to-be-monitored event, two key nodes are set. The prompt node (generated by the prompt node information) is usually a node before the to-be-monitored event, which is used to prompt the monitoring personnel in advance that which overrun event the aircraft pilot will perform in the future time period. The prompt node is used to generate t1. The positioning node (generated by the positioning node information) is usually a node at which the standard call operation in the to-be-monitored event should occur, which is used to generate t2 and t3.

[0047] Taking the take-off taxiing 80 knots call corresponding to-be-monitored event as an example, another kind of establishment logic of the overrun event database in this embodiment is explained.

[0048] Key node definition

[0049] The airspeed reaching 60 knots is taken as the prompt node corresponding to the to-be-monitored event to trigger the prompt. The generation time of the QAR engineering value representing the airspeed reaching 60 knots is taken as t1.

[0050] The airspeed reaching 80 knots is taken as the positioning node corresponding to the to-be-monitored event to trigger the determination of the positioning start time and the positioning end time. The generation time of the QAR engineering value representing the airspeed reaching 80 knots is taken as the positioning node information, and t2 and t3 are generated in combination with different time offsets ΔT1 and ΔT2.

[0051] Specifically, in a conforming scenario, the generation steps of t1, t2 and t3 are as follows:

[0052] S101: If the aircraft QAR information acquired in the current data acquisition period is the same as the determination QAR information corresponding to the key node in the prompt node information corresponding to any to-be-monitored event in the overrun event database, the generation time of the aircraft QAR information is determined as t1 corresponding to the to-be-monitored event.

[0053] ​S102: If the aircraft QAR information obtained in the current data collection period is the same as the judgment QAR information corresponding to the key node in the positioning node information corresponding to any to-be-monitored event in the overrun event database, determine t2 and t3 corresponding to the to-be-monitored event according to the generation time E of the aircraft QAR information. t2 and t3 satisfy the following conditions:

[0054] t2=E-△T1, t3=E+△T2. Wherein, △T1 and △T2 are respectively the first time interval and the second time interval. Since the length of the call content exists in the actual operation, and the call starting rhythm of different pilots also exists, in order to more comprehensively obtain the corresponding call information, a certain pre-set period △T1 and a certain post-set period △T2 are set to increase the acquisition time of the voice information.

[0055] For example, the to-be-monitored event corresponding to the 80 knots taxiing call is taken as an example, the positioning start time is 4 seconds before the airspeed reaches 80 knots, that is, △T1=4 seconds, which is the starting time of the overrun event judgment. The positioning end time is 5 seconds after the airspeed exceeds 80 knots, that is, △T2=5 seconds, which is the termination node of the monitoring period.

[0056] In addition, when constructing the overrun event database, the standard operation information to be executed corresponding to each to-be-monitored event in the overrun event database can also be constructed according to the SOP standard call content, such as can include time range, call content, call role, call occurrence number.

[0057] In this step, the QAR data segment downloaded by the airborne system in real time can be compared with the overrun event database in real time to dynamically start different overrun judgment periods corresponding to each to-be-monitored event, and start the corresponding overrun monitoring task. Specifically, in this embodiment, the overrun judgment period includes the judgment prompt period [t1, t2] and the overrun judgment period (t2, t3], and the tasks corresponding to different periods are not the same, which will be described as follows:

[0058] S200: If there is a new complete and valid voice recognition result in the real-time cabin sound information collected in the current data collection period, traverse t1, t2 and t3 corresponding to each to-be-monitored event.

[0059] In this embodiment, only when a new voice recognition result is obtained, t1, t2 and t3 corresponding to each to-be-monitored event are traversed, and the corresponding action execution strategy is started. Therefore, the number of execution times of the judgment strategy can be reduced, and the calculation amount can be reduced.

[0060] S300: If the end time T of the last complete sentence in the speech recognition result of all valid speech channels obtained in the current data collection period meets the condition T e (t1, t2], corresponding to any to-be-monitored event, the required standard voice shouting information in the to-be-monitored event corresponding SOP is generated.

[0061] Generally, because multiple channel sound receiving devices are arranged in the cabin of the aircraft, such as the headset of the main pilot, the headset of the copilot, and the cockpit sound receiving device, the shouting content appears in multiple different speech channels. In order to more comprehensively cover the recognition content of all speech channels, the accuracy of recognition is improved. Therefore, in this example, T is the end time of the last complete sentence in the speech recognition result of all valid speech channels in the current data collection period.

[0062] Shouting prompt and preliminary judgment: If the current time exceeds the preset prompt start time or the positioning start time, the shouting content to be executed will be displayed on the interactive interface.

[0063] S400: If T meets the condition T e (t2, t3], corresponding to any to-be-monitored event, each complete speech recognition information obtained in the time window [t2, T] is compared with the required standard voice shouting information in the to-be-monitored event corresponding SOP.

[0064] The time window from the positioning start time (if not generated, the prompt start time can be used instead) to the current time is used to perform qualification verification on the shouting content, and the content that passes the verification is presented in real time on the interface. Because there may be multiple shouting contents in some to-be-monitored events, not only speech comparison verification is performed, but also the content that passes the current verification is presented in real time on the interface.

[0065] Specifically, in this step, the real-time collected speech information can be first converted into text information, and then the two texts of the standard voice shouting information and the real-time collected speech information are matched and compared using regular expressions. The regular expressions of different shouting content matching rules can be flexibly set according to specific needs, such as "pre-landing check list" and "pre-landing check", which can be written as "pre-landing check (list)?". If you want to exclude the case of "pre-landing check list complete", you can further optimize it to "pre-landing check (list)?(?! complete)".

[0066] Generally, to make a more detailed and comprehensive judgment on the possibility of over-limit events, a warning start time t4 can also be set to trigger an over-limit warning in advance, generally t4 e (t2, t3], which can be configured according to specific needs.

[0067] Overrun early warning trigger: when the overrun event is configured with an early warning start time, and the current time has exceeded the time node, if the pass rate of the group of check items is lower than the pre-set threshold, the overrun event early warning mechanism will be triggered immediately. For example, all the check results of the shouting content in the time window will be displayed completely, so that the operator can quickly master the overall situation of the event.

[0068] In addition, in actual use scenarios, there are usually some similar to-be-monitored events. For example, in the process of aircraft climbing, shouting is required at certain height values, such as 100 meters, 150 meters, 200 meters, etc. However, the interval between these shouts will gradually shorten as the climbing speed of the aircraft increases, and thus the shouting content of a later to-be-monitored event may appear in the [t2, t3] of an earlier to-be-monitored event. It can also be known that once the shouting content of the later to-be-monitored event appears, the shouting content of the earlier to-be-monitored event also exists. Therefore, the step S410 is also provided in this embodiment to update the original t3 of the earlier to-be-monitored event to a more forward time in a timely manner when the above situation occurs. Correspondingly, the voice comparison period of the earlier to-be-monitored event can be shortened in a timely manner, thereby reducing the data processing amount during voice recognition and comparison.

[0069] S410: In the voice information comparison processing of any two adjacent similar to-be-monitored events, if the standard voice shouting information required in the SOP of the later to-be-monitored event is recognized in the time window [t2, t3] of the earlier to-be-monitored event, the t3 of the earlier to-be-monitored event is updated to the time when the standard voice shouting information required in the SOP of the later to-be-monitored event is recognized. The similar to-be-monitored events are a type of to-be-monitored events whose standard voice shouting information required in the SOP has a similarity greater than a pre-set similarity threshold.

[0070] S500: If T and the t3 of any to-be-monitored event satisfy the condition T>t3, and there is any voice recognition information that is the same as the standard voice shouting information required in the SOP of the to-be-monitored event in all voice comparison results of the to-be-monitored event in the time window [t2, t3], it is determined that the to-be-monitored event is qualified, that is, the current to-be-monitored event is not overrun.

[0071] S510: If all voice recognition information is not the same as the standard voice shouting information required in the SOP of the to-be-monitored event in all voice comparison results of the to-be-monitored event in the time window [t2, t3], it is determined that the to-be-monitored event is an overrun event of the flight.

[0072] Positioning end time: as the termination node of the monitoring period, the judgment of the overrun event is stopped at this time point, and the final determination result is output based on the monitoring result within the positioning start time to the positioning end time.

[0073] Taking the event to be monitored corresponding to the take-off roll 80 knots call as an example, the embodiment is described:

[0074] (1) Prompt trigger: once the airspeed reaches 60 knots, the take-off roll 80 knots call content is displayed on the operation interface.

[0075] (2) Dynamic judgment: when the end time of the last complete sentence in the speech recognition result exceeds the 60 knots airspeed moment, the following processing is performed:

[0076] If the time node of "air speed 80 knots 4 seconds before" has been generated, the eligibility judgment is performed from this starting point.

[0077] If the time node has not been generated, the call content can also be checked for eligibility from the 60 knots airspeed moment as the starting point, and the qualified result determined in real time is displayed.

[0078] (3) Final determination: when the end time of the last complete sentence in the speech recognition result of all valid channels exceeds "air speed 80 knots 5 seconds after", a comprehensive eligibility judgment will be performed within the complete 9-second monitoring period (80 knots 4 seconds before to 80 knots 5 seconds after), and the overrun event judgment result will be finally generated.

[0079] The present application realizes real-time receiving and processing of multi-source data of an aircraft (i.e. an aircraft), and the overrun event identification method based on multi-source real-time data makes the identification of the overrun event more accurate and timely. Compared with the traditional post-analysis mode, the overrun event can be discovered in time during flight, greatly improving the timeliness of flight risk warning, gaining valuable time for taking emergency measures, and effectively reducing the possibility of flight accidents.

[0080] In addition, as another possible embodiment of the present application, the following method is also provided:

[0081] S501: If the event to be monitored corresponds to only one set of prompt node information and positioning node information, the state of the event to be monitored is configured as a monitored event.

[0082] S502: If the event to be monitored corresponds to multiple sets of prompt node information and positioning node information, the positioning end time of the current completed monitoring group is updated as the historical monitoring cutoff time.

[0083] S503: If the prompt start time of the newly acquired event to be monitored is greater than the historical monitoring cutoff time, the monitoring task corresponding to the newly acquired prompt start time is started.

[0084] S504: If the newly acquired prompt start time of the to-be-monitored event is less than the historical monitoring end time, the monitoring task corresponding to the newly acquired prompt start time is not started.

[0085] For the to-be-monitored event with only one set of prompt node information and positioning node information, after the determination is completed, the monitoring time period is automatically recorded in the historical file corresponding to the call item, and the state of the to-be-monitored event is modified to a monitored event. In subsequent every second cycle monitoring, the time period of the completed monitoring will be automatically skipped to avoid repeated detection and improve monitoring efficiency.

[0086] For the to-be-monitored event with multiple sets of prompt node information and positioning node information, after the determination of the corresponding detection content is completed each time, the positioning end time of the current completed monitoring group is automatically updated as the historical monitoring end time, and in subsequent every second cycle monitoring, whether the newly acquired prompt start time of the to-be-monitored event is greater than the historical monitoring end time can be used to avoid repeated detection and improve monitoring efficiency.

[0087] As another possible embodiment of the present application, the voice channel is multiple, such as Figure 3 As shown in the figure, the effective voice channel is acquired according to the following steps:

[0088] S600: When the complete voice information of any voice channel is first recognized as the standard voice call information required in the SOP corresponding to the to-be-monitored event, the generation time period of the complete voice information of the voice channel is taken as the determination time period.

[0089] S700: The voice information generated in the determination time period in other voice channels is taken as the determination voice information.

[0090] S800: The similarity of each determination voice information and the complete voice information of the voice channel first recognized is acquired respectively.

[0091] Specifically, the similarity of the determination voice information and the complete voice information of the voice channel first recognized is acquired according to the following steps:

[0092] S801: The determination voice information and the complete voice information of the voice channel first recognized are respectively subjected to frame processing.

[0093] Frame processing: Continuous speech signal is cut into small time segments (usually 20-30 milliseconds) because speech signal is time-varying. That is, speech signal is essentially non-stationary, which means its statistical properties (such as frequency content) vary over time. However, within a short time (e.g. 20-40 milliseconds), it can be assumed that speech signal is "quasi-stationary". By dividing speech signal into short time segments (frames), analysis methods suitable for stationary signals can be applied within each small segment.

[0094] Specifically, before S801, other pre-processing operations on audio can also be set, such as pre-emphasis: a high-pass filter is used to enhance the high-frequency part, to compensate for the loss of high-frequency components in the pronunciation process. After S801, windowing processing can also be set: a Hamming window or other types of window functions are applied to each frame to reduce spectral leakage.

[0095] S802: According to the spectrum information corresponding to each frame, the Mel frequency cepstral coefficients corresponding to the complete speech information of the determined speech information and the first recognized speech channel are obtained respectively.

[0096] S803: Based on the Mel frequency cepstral coefficients, the similarity of the determined speech information and the complete speech information of the first recognized speech channel is obtained using the dynamic time warping method.

[0097] Extract the MFCC coefficients (Mel frequency cepstral coefficients) of speech A and speech B, to get two matrices X ∈ R T1×d and Y ∈ R T2×d , where T1 and T2 are the number of frames, and d is the MFCC dimension.

[0098] Construct a distance matrix D, where , D ij represents the Euclidean distance between the i-th frame of speech A and the j-th frame of speech B, X i ∈X, Y j ∈Y.

[0099] Use dynamic programming (Dynamic Time Warping, DTW) to find the shortest path, and accumulate the sum of distances on the path as the similarity score (the smaller the better).

[0100] The selection of the shortest path needs to meet the following conditions:

[0101] The path must start from the top left corner of the matrix and reach the bottom right corner.

[0102] The path can only move down, right or diagonally (i.e. cannot back up).

[0103] There is a constraint that limits the slope of the path, which ensures the continuity and locality in time.

[0104] S900: When the similarity between the judgment voice information and the complete voice information of the first recognized voice channel is greater than the preset similarity threshold, the voice channel to which the judgment voice information belongs is determined as the valid voice channel.

[0105] Since the multiple voice channels in the cabin may have different transmission distances and different codec device performances when receiving the same voice content, the same voice content may have a certain delay in different voice channels. If the features of the voice channels are directly compared, it may lead to inaccurate similarity calculation. In order to accurately calculate the similarity of two audios in this case, the DTW technology is used for similarity calculation between voice audios in the embodiment. Since DTW allows nonlinear stretching or compression on the time axis to find the best matching path, it can be used to measure the similarity of two time series in the case of nonlinear alignment on the time axis. It can be further applied to the embodiment to more accurately compare the similarity between voice signals of different lengths.

[0106] In addition, although the various steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. In addition or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be divided into multiple steps, etc.

[0107] From the above description of the embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a number of instructions to make a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) execute the methods according to the embodiments of the present disclosure.

[0108] In the example embodiments of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0109] Those skilled in the art can understand that each aspect of the present disclosure can be implemented as a system, a method or a program product. Therefore, each aspect of the present disclosure can be embodied in the form of a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which can be collectively referred to as "circuitry", "module" or "system" herein.

[0110] The electronic device according to this embodiment of the present application. The electronic device is merely an example and should not bring any limitation to the function and use range of the embodiments of the present application.

[0111] The electronic device is in the form of a general computing device. As shown in Figure 4 The components of the electronic device can include, but are not limited to, the at least one processor, the at least one memory, and a bus that connects the different system components, including the memory and the processor.

[0112] The memory stores program codes that can be executed by the processor, so that the processor executes the steps described in the above "Exemplary Method" section according to various exemplary embodiments of the present application.

[0113] The memory can include a readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory, and can further include a read-only memory (ROM).

[0114] The memory can further include programs / utilities with a set of (at least one) program modules, such as an operating system, one or more application programs, other program modules, and program data, each of which or some combination of which can include implementation of a network environment.

[0115] The bus can be one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of a variety of bus architectures.

[0116] The electronic device can also communicate with one or more external devices (such as a keyboard or a pointing device, a Bluetooth device, etc.) through the communication interface, and can also communicate with one or more devices that enable a user to interact with the electronic device, and / or with any devices (such as a router, a modem, etc.) that enable the electronic device to communicate with one or more other computing devices. Such communication can be carried out through an input / output (I / O) interface. In addition, the electronic device can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through a network adapter. The network adapter communicates with other modules of the electronic device through the bus. It should be understood that although not shown in the figure, other hardware and / or software modules can be used in conjunction with the electronic device, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0117] Through the description of the above embodiments, those skilled in the art can easily understand that the example embodiments described herein can be implemented by software, or by software in combination with necessary hardware. Therefore, the technical solutions according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.) or a network, and includes a plurality of instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to perform the method according to the embodiments of the present disclosure.

[0118] In the example embodiments of the present disclosure, a computer readable storage medium is also provided, which stores a program product capable of implementing the method described above. In some possible embodiments, various aspects of the present disclosure can also be implemented in the form of a program product, which includes program codes for causing a terminal device to perform the steps according to various example embodiments of the present disclosure described in the "example method" section of the present specification when the program product is run on the terminal device.

[0119] The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, be but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include an electrical connection having one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0120] The computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which readable program codes are borne. Such a propagated data signal can take on multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable signal medium can also be any readable medium other than the readable storage medium, which can send, propagate or transmit programs for use by or in connection with an instruction execution system, device or apparatus.

[0121] The program codes contained on the readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0122] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0123] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0124] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0125] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for monitoring ultra-limit events based on multi-source real-time data, characterized in that, The method comprises the following steps: The flight monitoring information obtained in the current data acquisition period is matched with the prompt node information and the positioning node information corresponding to each to-be-monitored event in the out-of-limit event database to generate the prompt start time t1, the positioning start time t2 and the positioning end time t3 corresponding to each to-be-monitored event; t1 If there is a new complete and valid voice recognition result in the real-time cabin sound information collected in the current data acquisition period, the t1, t2 and t3 corresponding to each to-be-monitored event are traversed; If the end time T of the last complete sentence in the voice recognition result of all valid voice channels obtained in the current data acquisition period meets the condition T∈(t1, t2], the required standard voice call information in the SOP corresponding to any to-be-monitored event is generated; If T meets the condition T∈(t2, t3] corresponding to any to-be-monitored event, each complete voice recognition information obtained in the time window [t2, T] is compared with the required standard voice call information in the SOP corresponding to the to-be-monitored event; If T meets the condition T>t3 corresponding to any to-be-monitored event, if all voice recognition information in all voice comparison results of the to-be-monitored event in the time window [t2, t3] is different from the required standard voice call information in the SOP corresponding to the to-be-monitored event, it is determined that the to-be-monitored event is an out-of-limit event of the flight.

2. The method of claim 1, wherein, After T meets the condition T>t3 corresponding to any to-be-monitored event, the method further comprises: If there is any voice recognition information in all voice comparison results of the to-be-monitored event in the time window [t2, t3] that is the same as the required standard voice call information in the SOP corresponding to the to-be-monitored event, it is determined that the to-be-monitored event is qualified.

3. The method of claim 2, wherein, After determining whether the to-be-monitored event is out of limit, the method further comprises: If the to-be-monitored event corresponds to only one set of prompt node information and positioning node information, the state of the to-be-monitored event is configured as a monitored event; If the to-be-monitored event corresponds to multiple sets of prompt node information and positioning node information, the positioning end time of the current completed monitoring group is updated as the historical monitoring cutoff time; If the prompt start time of the newly obtained to-be-monitored event is greater than the historical monitoring cutoff time, the monitoring task corresponding to the newly obtained prompt start time is started; If the prompt start time of the newly obtained to-be-monitored event is less than the historical monitoring cutoff time, the monitoring task corresponding to the newly obtained prompt start time is not started.

4. The method of claim 1, wherein, The flight monitoring information obtained in the current data collection period is matched with the prompt node information and the positioning node information corresponding to each to-be-monitored event in the out-of-limit event database to generate a prompt start time t1, a positioning start time t2 and a positioning end time t3 corresponding to each to-be-monitored event, including: The flight monitoring information obtained in the current data collection period is matched with the time generation strategy information limited by any prompt node information corresponding to the flight in the out-of-limit event database; If the flight monitoring information collected currently satisfies the time generation strategy information limited by any prompt node information, the t1 of the to-be-monitored event corresponding to the prompt node information is generated. The flight monitoring information obtained in the current data collection period is matched with the time generation strategy information limited by any positioning node information corresponding to the flight in the out-of-limit event database; If the flight monitoring information collected currently satisfies the time generation strategy information limited by any positioning node information, the t2 and the t3 of the to-be-monitored event corresponding to the positioning node information are generated.

5. The method of claim 4, wherein, The time generation strategy information includes: The real-time cabin sound information and the QAR information corresponding to the current flight are determined by using the flight identity information included in the flight operation information; The operation description information recognized from the real-time cabin sound information corresponding to the current flight is determined as the local air-ground dialogue positioning information; The local air-ground dialogue positioning information is matched with the real-time QAR information of the current flight obtained subsequently; If the operation information in the real-time QAR information is the same as the operation description information in the local air-ground dialogue positioning information, the generation time of the real-time QAR information is taken as the positioning time, and the positioning time is used to generate the t1, the t2 and the t3 of the to-be-monitored event.

6. The method of claim 1, wherein, Further including: In the voice information comparison processing of any two adjacent same type to-be-monitored events, if the standard voice call information required in the SOP corresponding to the latter to-be-monitored event is recognized in the time window [t2, t3] corresponding to the former to-be-monitored event, the t3 corresponding to the former to-be-monitored event is updated to the time when the standard voice call information required in the SOP corresponding to the latter to-be-monitored event is recognized. The same type to-be-monitored event is a type of to-be-monitored event whose standard voice call information required in the SOP has a similarity greater than a preset similarity threshold.

7. The method of claim 1, wherein, The voice channels are multiple, and the effective voice channels are obtained according to the following steps: When the complete voice information of any voice channel is first recognized as the standard voice call information required in the SOP of the to-be-monitored event, the generation time period of the complete voice information of the voice channel is taken as the determination time period; The voice information generated in the determination time period in other voice channels is taken as the determination voice information; The similarity of each determination voice information and the complete voice information of the voice channel first recognized is obtained respectively; When the similarity of any determination voice information and the complete voice information of the voice channel first recognized is greater than a preset similarity threshold, the voice channel to which the determination voice information belongs is determined as an effective voice channel.

8. The method of claim 7, wherein, The similarity between the voice information and the complete voice information of the first recognized voice channel is determined according to the following steps: The voice information and the complete voice information of the first recognized voice channel are respectively subjected to frame processing; According to the corresponding spectrum information of each frame, the mel-frequency cepstral coefficients corresponding to the voice information and the complete voice information of the first recognized voice channel are respectively obtained; Based on the mel-frequency cepstral coefficients, the similarity between the voice information and the complete voice information of the first recognized voice channel is obtained by using a dynamic time warping method. 9.A non-transitory computer-readable storage medium storing a computer program, the computer program comprising instructions causing a processor to perform the method according to any one of claims 1 to 8. The computer program, when executed by a processor, implements the method for monitoring an over-limit event based on multi-source real-time data according to any one of claims 1 to 8.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor, when executing the computer program, implements the method for monitoring an over-limit event based on multi-source real-time data according to any one of claims 1 to 8.

Images