Method, device, medium, program product and equipment for time alignment of cabin sound events

Abstract

The present invention discloses a method, device, medium, program product and equipment for aligning cabin voice event time, the method comprising: obtaining cabin voice data and flight data to be aligned; marking all flight events in the flight data and generating a first event time series; slicing and feature extracting the cabin voice data to obtain a plurality of audio feature sequences; inputting each audio feature sequence into a cabin voice event recognition model to obtain a corresponding recognition result, and integrating all recognition results to obtain a second event time series; based on the time range of the second event time series, intercepting the first event time series to obtain a first event time subsequence; dynamically time-warping the first event time subsequence and the second event time series to obtain the time corresponding to all cabin voice events. The present invention can quickly align the cabin voice data and flight data, and accurately obtain the occurrence time of key events in the cabin voice.

Description

Technical Field

[0001] The present invention relates to the field of aviation technology, and in particular to a method, device, medium, program product and equipment for time alignment of cabin voice events. Background Art

[0002] In the modern aviation industry, flight safety is of vital importance. To ensure flight safety, aircraft are equipped with various monitoring devices, including cockpit voice recorders (CVR) and flight data recorders (FDR). These devices record key information during the flight for investigation and analysis in the event of an accident.

[0003] CVR records the sounds in the cockpit, including pilot conversations, warning sounds and other sounds, while FDR records various parameters during the flight, such as speed, altitude, heading, etc. The data of CVR and FDR are stored separately, and CVR recordings usually do not have time stamps. This makes it a challenging task to align CVR recordings with FDR data to determine the specific time when the recording occurred during an accident investigation.

[0004] Currently, the way to align CVR and FDR data mainly relies on manual analysis. It requires highly professional and experienced investigators to listen to the CVR recordings, identify key events, and then infer the time when these events occurred based on the FDR data. This process is time-consuming and prone to manual errors. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide a method, apparatus, medium, program product and equipment for time alignment of cockpit voice events, which can quickly align cockpit voice data and flight data and accurately obtain the occurrence time of key events in cockpit voice.

[0006] The first aspect of the present invention provides a method for aligning cabin sound events in time, including:

[0007] Acquire the cabin sound data and flight data to be aligned;

[0008] Marking all flight events in the flight data and generating a first event time series;

[0009] Based on a preset time window, the cabin sound data is sliced ​​and features are extracted to obtain a plurality of audio feature sequences; each of the audio feature sequences is input into a cabin sound event recognition model to obtain a corresponding recognition result, and all the recognition results are integrated to obtain a second event time series;

[0010] Based on the time range of the second event time series, intercepting the first event time series to obtain a first event time subsequence;

[0011] Dynamic time warping is performed on the first event time subsequence and the second event time sequence, and the time information of all the flight events in the first event time subsequence is mapped to the second event time sequence to obtain the time corresponding to all cabin sound events.

[0012] Optionally, after acquiring the cabin sound data and the flight data to be aligned, the method further includes:

[0013] Performing a first preprocessing on the flight data to obtain preprocessed flight data; wherein the first preprocessing includes: data cleaning and data conversion;

[0014] The cabin sound data is subjected to a second preprocessing to obtain preprocessed cabin sound data; wherein the second preprocessing includes: noise reduction processing and signal enhancement processing.

[0015] Optionally, marking all flight events in the flight data and generating a first event time series includes:

[0016] The analysis system based on the rapid access recorder analyzes the flight data, and marks all the flight events in the flight data to obtain a corresponding first event time series.

[0017] Optionally, the cabin sound data is sliced ​​and features are extracted based on a preset time window to obtain a plurality of audio feature sequences, including:

[0018] Dividing the cabin sound data according to the preset time window to obtain a plurality of audio segments;

[0019] Feature extraction is performed on each of the audio clips to obtain a corresponding audio feature sequence; wherein each of the audio feature sequences includes: Mel-frequency cepstral coefficients and Mel-frequency spectrum data.

[0020] Optionally, the cabin sound event recognition model is obtained by the following steps:

[0021] Obtain N cabin sound sample data; where N≥1;

[0022] Performing event labeling on all the cabin sound sample data, and dividing all the event-labeled cabin sound sample data according to the preset time window to obtain a plurality of audio sample segments and corresponding event labels;

[0023] Extracting features from each of the audio sample segments to obtain a corresponding sample feature sequence;

[0024] An initial recognition model is created based on a long short-term memory network architecture; all the sample feature sequences and corresponding event labels are used as training data, and the initial recognition model is trained to obtain the cabin sound event recognition model.

[0025] Optionally, the performing dynamic time warping on the first event time subsequence and the second event time sequence, and mapping the time information of all the flight events in the first event time subsequence to the second event time sequence to obtain the time corresponding to all cabin sound events, includes:

[0026] constructing a cumulative distance matrix based on the first event time subsequence and the second event time series;

[0027] Perform path backtracking in the cumulative distance matrix to obtain the shortest regular path;

[0028] Based on the shortest regular path, the time information of all the flight events in the first event time subsequence is mapped to the second event time sequence to obtain the time corresponding to all the cabin sound events.

[0029] A second aspect of the present invention provides a device for aligning cabin sound events in time, for implementing the method for aligning cabin sound events in time as described in any one of the first aspects, the device comprising:

[0030] A data acquisition module, used to acquire cabin sound data and flight data to be aligned;

[0031] A flight event marking module, used for marking all flight events in the flight data and generating a first event time series;

[0032] A cabin sound event recognition module is used to slice and extract features from the cabin sound data based on a preset time window to obtain a plurality of audio feature sequences; each of the audio feature sequences is input into a cabin sound event recognition model to obtain a corresponding recognition result, and all the recognition results are integrated to obtain a second event time series;

[0033] A sequence interception module, configured to intercept the first event time series based on the time range of the second event time series to obtain a first event time subsequence;

[0034] A sequence alignment module is used to dynamically time-align the first event time subsequence and the second event time sequence, and map the time information of all the flight events in the first event time subsequence to the second event time sequence to obtain the time corresponding to all cabin sound events.

[0035] An embodiment of the third aspect of the present invention provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the cabin sound event time alignment method described in any one of the first aspects above.

[0036] An embodiment of a fourth aspect of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the cabin sound event time alignment method described in any one of the first aspects above.

[0037] An embodiment of the fifth aspect of the present invention is an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the cabin sound event time alignment method described in any one of the first aspect above when executing the computer program.

[0038] Compared with the prior art, the embodiments of the present invention provide a method, device, medium, program product and equipment for aligning the time of cabin sound events, the method comprising: obtaining cabin sound data and flight data to be aligned; marking all flight events in the flight data and generating a first event time series; slicing and feature extracting the cabin sound data based on a preset time window to obtain a number of audio feature sequences; inputting each audio feature sequence into a cabin sound event recognition model to obtain a corresponding recognition result, and integrating all recognition results to obtain a second event time series; based on the time range of the second event time series, intercepting the first event time series to obtain a first event time subsequence; dynamically time-warping the first event time subsequence and the second event time series, and mapping the time information of all flight events in the first event time subsequence to the second event time series to obtain the time corresponding to all cabin sound events. The present invention can quickly align the cabin sound data and the flight data, and accurately obtain the occurrence time of key events in the cabin sound. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is a flow chart of an embodiment of the cabin sound event time alignment method provided by the present invention.

[0040] Figure 2 It is a structural schematic diagram of an embodiment of the cabin sound event time alignment device provided by the present invention.

[0041] Figure 3 It is a structural schematic diagram of an embodiment of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this technical field without creative work are within the scope of protection of the present invention.

[0043] See also Figure 1 , is a flow chart of an embodiment of the cabin sound event time alignment method provided by the present invention.

[0044] The first aspect of the present invention provides a method for aligning cabin sound events in time, including steps S1 to S6, which are specifically as follows:

[0045] Step S1: Acquire cabin sound data and flight data to be aligned;

[0046] Step S2: marking all flight events in the flight data and generating a first event time series;

[0047] Step S3: based on a preset time window, the cabin sound data is sliced ​​and features are extracted to obtain a plurality of audio feature sequences; each of the audio feature sequences is input into a cabin sound event recognition model to obtain a corresponding recognition result, and all the recognition results are integrated to obtain a second event time series;

[0048] Step S4: based on the time range of the second event time series, intercept the first event time series to obtain a first event time subsequence;

[0049] Step S5: Dynamically time warp the first event time subsequence and the second event time sequence, and map the time information of all the flight events in the first event time subsequence to the second event time sequence to obtain the time corresponding to all cabin sound events.

[0050] It should be noted that in the above step S4, due to the limited storage capacity of the CVR, it can only record the audio data of the last two hours. If the flight time exceeds two hours, the previous recording data in the CVR will be overwritten by the subsequent recording data. In contrast, the FDR can record the flight data of the last 25 hours. In addition, airlines and aircraft manufacturers will ensure that the CVR and FDR have a certain degree of synchronization in power supply and signal input in order to provide consistent data records in event investigations, that is, the time when the CVR and FDR stop recording is basically the same, because the power supplies of both are usually connected to the same aircraft power system. Based on this, the embodiment of the present invention intercepts a partial sequence of the same time length from the end of the first event time sequence according to the time range of the second event time sequence (the time length of the CVR recording cabin sound data) as the first event time subsequence. Among them, the time granularity of the first event time subsequence and the second event time sequence is consistent.

[0051] In the specific implementation, first, the cockpit sound data (audio data in the cockpit of the aircraft) and flight data (flight parameters during the operation of the aircraft) that need to be aligned are collected. Then, all flight events in the flight data are marked, including but not limited to takeoff, landing, aircraft overspeed, turbulence, rapid climb / descent, aircraft engine failure, etc.; after the marking is completed, a time series containing all flight events is generated as the first event time series, which clearly records the various flight events and their specific time points that occurred during the flight; wherein, the length of the first event time series is consistent with the time length of the flight data record. Similarly, it is necessary to determine the various cockpit sound events that occur in the cockpit sound data to form a second event time series, which includes but is not limited to: cockpit alarm sounds, key events extracted from the driver's dialogue, such as overspeed, climbing too high, descending too low, engine warnings, collision objects, etc.; wherein, the second event time series is consistent with the time length of the cockpit sound data record, and the cockpit sound events and flight events correspond to the same event key-key value space. Next, the first event time subsequence with the same length as the second event time sequence is intercepted from the end of the first event time sequence to ensure that only the relevant time period is focused on during the alignment process, thereby improving the accuracy and efficiency of the alignment. Finally, based on the Dynamic Time Warping (DTW) algorithm, the first event time subsequence and the second event time sequence are optimally aligned, and all flight event time information in the first event time subsequence is mapped to the second event time sequence to obtain the precise time corresponding to all cabin sound events.

[0052] Through the above technical solutions, it can be known that the cockpit voice event time alignment method provided by the embodiment of the present invention can quickly establish an accurate time relationship between the CVR and FDR, so that the pilot's conversation and the aircraft's flight data can be observed on the same timeline. This is very helpful for understanding the pilot's decision-making process and the aircraft's response mode when critical events occur. In addition, the embodiment of the present invention can also analyze whether there are missing critical events in the CVR data, and predict the occurrence time of these missing critical events and related issues, thereby greatly improving the efficiency and accuracy of flight accident investigations and making important contributions to aviation safety.

[0053] In an optional embodiment, after acquiring the cabin sound data and the flight data to be aligned, the method further includes:

[0054] Performing a first preprocessing on the flight data to obtain preprocessed flight data; wherein the first preprocessing includes: data cleaning and data conversion;

[0055] The cabin sound data is subjected to a second preprocessing to obtain preprocessed cabin sound data; wherein the second preprocessing includes: noise reduction processing and signal enhancement processing.

[0056] The embodiment of the present invention can use Python tools to pre-process the flight data and the cabin sound data respectively. Specifically, after obtaining the flight data to be aligned from the FDR, the flight data will be further cleaned (such as removing invalid or erroneous data) and converted (such as converting to the format required for analysis) to improve the quality of the flight data. The cabin sound data to be aligned is obtained from the CVR. The cabin sound data often contains various background noises, such as airflow noise, etc. The background noise components can be effectively reduced through noise reduction processing and signal enhancement processing, thereby improving the clarity and recognition of useful signals in the cabin sound data.

[0057] In an optional embodiment, marking all flight events in the flight data and generating a first event time series includes:

[0058] The analysis system based on the rapid access recorder analyzes the flight data, and marks all the flight events in the flight data to obtain a corresponding first event time series.

[0059] The embodiment of the present invention can automatically identify all flight events in the flight data through the analysis system of the Quick Access Recorder (QAR), such as parsing the flight data collected by the FDR through the Airborne Flight Analysis System Environment (AirFASE), automatically identifying and classifying the key events (flight events) and the time periods when the events occurred during the flight of the aircraft, such as take-off, landing, aircraft overspeed, turbulence, rapid climb / descent, aircraft engine failure, etc., and attaching timestamps to these flight events to form a first event time series; wherein the time length of the first event time series is still consistent with the time length of the flight data.

[0060] AirFASE works based on built-in multiple flight data analysis algorithms and rich ready-to-use functions. It conducts in-depth analysis of flight data (such as speed, altitude, attitude, engine status and other flight parameters) and automatically identifies key events (flight events) during the flight process, thereby enabling accurate and efficient flight monitoring and analysis.

[0061] In an optional embodiment, the cabin sound data is sliced ​​and feature extracted based on a preset time window to obtain a plurality of audio feature sequences, including:

[0062] Dividing the cabin sound data according to the preset time window to obtain a plurality of audio segments;

[0063] Feature extraction is performed on each of the audio clips to obtain a corresponding audio feature sequence; wherein each of the audio feature sequences includes: Mel-frequency cepstral coefficients and Mel-frequency spectrum data.

[0064] In the specific implementation, the size of the preset time window is first determined according to the analysis requirements to ensure that the audio clip corresponding to the preset time window can capture sufficient audio information (for input into the cabin sound event recognition model for prediction), and at the same time ensure that the audio clip covers at most one cabin sound event, so as to facilitate accurate cabin sound event recognition. Then, according to the preset time window, the long-term cabin sound data is divided into several independent audio clips. Of course, the start and end time of each audio clip will also be marked, which is used to determine the specific time period corresponding to the recognized cabin sound event in the second event time sequence when the recognition results are subsequently spliced; wherein, the segmentation of these audio clips can be non-overlapping or overlapping (such as segmentation using a sliding window), and the specific method is selected according to the needs in actual application. Finally, feature extraction (acoustic feature extraction) is performed on each audio clip, including but not limited to: Mel-Frequency Cepstral Coefficients (MFCC) or Mel spectrum; the acoustic features extracted from each audio clip are composed into a corresponding audio feature sequence. Each audio feature sequence will be used as input data for the cockpit sound event recognition model, and the corresponding cockpit sound events will be recognized by the model. These cockpit sound events include but are not limited to cockpit alarm sounds, key events extracted from driver conversations, such as speeding, climbing too high, descending too low, engine warnings, collisions with objects, etc. For key events (cabin sound events), the cockpit sound event recognition model will output specific key values ​​(label numbers), and for non-key events, the model will output None.

[0065] In an optional embodiment, the cabin sound event recognition model is obtained by the following steps:

[0066] Obtain N cabin sound sample data; where N≥1;

[0067] Performing event labeling on all the cabin sound sample data, and dividing all the event-labeled cabin sound sample data according to the preset time window to obtain a plurality of audio sample segments and corresponding event labels;

[0068] Extracting features from each of the audio sample segments to obtain a corresponding sample feature sequence;

[0069] An initial recognition model is created based on a long short-term memory network architecture; all the sample feature sequences and corresponding event labels are used as training data, and the initial recognition model is trained to obtain the cabin sound event recognition model.

[0070] In specific implementation, the process of obtaining training data is as follows: First, a large amount of cockpit sound sample data is collected from the CVR. These cockpit sound sample data usually involve conversations in the cockpit (including the identity of the interlocutor, the content of the conversation, the tone and emotion, etc.) and the cockpit alarm sound. Subsequently, the collected cockpit sound sample data is transcribed into corresponding text samples. The transcription process is usually completed by a professional voice transcriptionist, or automatically transcribed through a speech recognition algorithm. The transcribed text sample should contain detailed text descriptions of all conversations, alarm sounds and other related sounds. Then, each text sample is annotated with detailed events, that is, key events (cabin sound events) and their specific occurrence time are annotated. At the same time, the cockpit sound events that occurred in a certain audio segment are also annotated in the corresponding cockpit sound sample data (with event labels attached), thereby realizing event annotation of all cockpit sound sample data. Obviously, event annotation is a professional, meticulous and time-consuming process, but it is crucial for the construction of efficient and accurate machine learning models, that is, event annotation can greatly improve the learning efficiency and prediction accuracy of the model. It should be noted that the embodiment of the present invention will perform preprocessing operations on each cabin sound sample data, such as noise reduction and signal enhancement, before transcribing the cabin sound sample data into a text sample. After completing the event labeling of all cabin sound samples, the cabin sound sample data after all event labeling is divided according to a preset time window to obtain a number of audio sample segments and corresponding event labels; feature extraction is performed on each of the audio sample segments to obtain a corresponding sample feature sequence. Finally, all sample feature sequences and corresponding event labels are used as training data for the initial recognition model.

[0071] The embodiment of the present invention adopts "supervised learning" to train the model. Specifically, the sample feature sequence is used as the input of the model, and the corresponding event label is used as the output of the model. The following will be explained from the model architecture to the model evaluation:

[0072] Model architecture: The initial recognition model uses the Long Short-Term Memory (LSTM) network as the core architecture. The LSTM architecture includes a deep LSTM network (i.e., multiple LSTM layers) to effectively capture complex dependencies in time series data. In order to prevent overfitting, a dropout layer is introduced for regularization, and the generalization ability of the model is enhanced by randomly disconnecting the connection of the neural network. Of course, the deep LSTM network in the embodiment of the present invention can specifically be a bidirectional LSTM, which can process past and future information at the same time and improve model performance.

[0073] Training strategy: Use the Adam optimizer to accelerate the convergence of the model by taking advantage of its automatic learning rate adjustment feature. Use the cross entropy loss function (suitable for classification problems) to identify and predict key events (cabin sound events) in CVR recordings. Implement an early stopping strategy, that is, stop training when the model performance on the validation set no longer improves to avoid overfitting. In addition, data enhancement (such as adjusting the audio rate, volume, etc.) is used to improve the robustness and generalization ability of the model.

[0074] Evaluation and optimization: Use multi-dimensional indicators such as accuracy, recall, and F1 score to comprehensively evaluate model performance. Combine cross-validation methods to ensure the reliability of evaluation results and the stability of the model. Based on evaluation feedback, continuously adjust the LSTM network structure, training process, and regularization strategy to achieve optimal model performance.

[0075] Validation and testing: The dataset is divided into training set, validation set and test set; the training set is used to train the model, the validation set is used to adjust the model's hyperparameters and prevent overfitting, and the test set is used to evaluate the final performance of the model.

[0076] Through the above method, a cabin voice event recognition model can be obtained that can accurately predict key events (cabin voice events) in cabin voice data and has strong generalization ability. After obtaining the trained cabin voice event recognition model, it is applied to new cabin voice data to automatically identify various key events (cabin voice events) in the new cabin voice data.

[0077] In addition, the cockpit sound event recognition model can be deployed in a specific test scenario to verify its overall execution effect, such as uploading the cockpit sound event recognition model to the server and configuring the necessary operating environment on the server. After loading the model on the server, first verify whether the model can be loaded normally. Then, perform preliminary functional tests, including whether the input and output of the data are normal, whether the model can run normally, and determine whether the model will not crash during operation. At the same time, evaluate the running speed and resource occupancy of the model to determine whether it meets the performance requirements of actual deployment. Obviously, the cockpit sound event recognition model in the present invention can be directly deployed on the server. Before the new cockpit sound data is used to overwrite the old cockpit sound data each time, the old cockpit sound data will be uploaded to the cockpit sound event recognition model in the server, and all cockpit sound events occurring in the old cockpit sound data will be automatically identified and saved in the server as the corresponding second event time series. In this way, when conducting an accident investigation, the cockpit sound data of the last 2 hours will no longer be used alone, but the cockpit sound event information of the same length as the FDR data can be used to perform time alignment of the 25-hour cockpit sound event and the flight event, thereby providing more comprehensive data support for accident analysis and further improving the accuracy of accident investigation.

[0078] In an optional embodiment, the performing dynamic time warping on the first event time subsequence and the second event time sequence, and mapping the time information of all the flight events in the first event time subsequence to the second event time sequence, to obtain the time corresponding to all cabin sound events, includes:

[0079] constructing a cumulative distance matrix based on the first event time subsequence and the second event time series;

[0080] Perform path backtracking in the cumulative distance matrix to obtain the shortest regular path;

[0081] Based on the shortest regular path, the time information of all the flight events in the first event time subsequence is mapped to the second event time sequence to obtain the time corresponding to all the cabin sound events.

[0082] Specifically, after obtaining the first event time subsequence and the second event time series, the dynamic time warping algorithm is used to automatically align the key events in the CVR data (the second event time series) with the key events in the FDR data (the first event time subsequence), so as to accurately determine the time points of each key event (cabin sound event) in the CVR; wherein the data elements contained in the first event time subsequence and the second event time series are the key values ​​(label numbers) of the events.

[0083] The DTW algorithm is an algorithm used to measure and align the similarity of two time series. The algorithm finds a "distortion" method by nonlinearly stretching or compressing the sequence on the time axis so that the similarity between the two distorted sequences is maximized or the difference is minimized. In specific implementation, the DTW algorithm first calculates the cumulative distance matrix between the two event time series, and then finds a shortest regularized path on the cumulative distance matrix as the optimal alignment path. Finally, according to the optimal alignment path, the two event time series are distorted so that the elements in the sequence are aligned on the time axis. That is, according to the shortest regularized path, the best alignment of the first event time subsequence and the second event time series on the time axis is achieved, so that the time information of all flight events in the first event time subsequence is mapped to the second event time series, and the precise time corresponding to all cabin sound events is obtained.

[0084] In addition, after the key events in the CVR data (second event time series) are automatically aligned with the key events in the FDR data (first event time subsequence), it is possible to further determine whether there are missing key events in the CVR data, as well as to infer the time when the missing key events occurred and the key event problems that occurred.

[0085] During the flight, a critical event may occur multiple times, and the number of corresponding events recorded by the CVR is not completely consistent. This is because the CVR records the sound events in the cockpit, and the number of times it is recorded may be affected by whether the pilot verbally confirms and whether the corresponding warning is issued in the cockpit. For example, if the overspeed event occurs twice during the flight, there will be two aircraft overspeed events in the FDR data, while the overspeed event in the CVR is only recorded once. This may be because only one overspeed is accompanied by an obvious sound warning or verbal confirmation by the pilot. In this case, it is impossible to accurately determine which overspeed event (cabin sound event) recorded this time corresponds to the overspeed event in the FDR data based on the CVR data alone. However, by automatically aligning the key events in the CVR data (second event time series) with the key events in the FDR data (first event time subsequence), even if these events are offset in time, after the two event sequences are aligned, it is still possible to accurately identify which event in the FDR the overspeed event recorded in the CVR corresponds to, and then determine whether there is a missing key event in the CVR data, and infer the time when the missing key event occurred.

[0086] See also Figure 2 , is a structural schematic diagram of an embodiment of the cabin sound event time alignment device provided by the present invention.

[0087] A second aspect of the present invention provides a device for aligning cabin sound events in time, for implementing the method for aligning cabin sound events in time as described in any one of the embodiments of the first aspect, the device comprising:

[0088] A data acquisition module 11 is used to acquire cabin sound data and flight data to be aligned;

[0089] A flight event marking module 12, used for marking all flight events in the flight data and generating a first event time series;

[0090] The cabin sound event recognition module 13 is used to slice and extract features of the cabin sound data based on a preset time window to obtain a plurality of audio feature sequences; each of the audio feature sequences is input into the cabin sound event recognition model to obtain a corresponding recognition result, and all the recognition results are integrated to obtain a second event time series;

[0091] A sequence interception module 14, configured to intercept the first event time series based on the time range of the second event time series to obtain a first event time subsequence;

[0092] The sequence alignment module 15 is used to dynamically time-align the first event time subsequence and the second event time sequence, and map the time information of all the flight events in the first event time subsequence to the second event time sequence to obtain the time corresponding to all cabin sound events.

[0093] It should be noted that the cabin sound event time alignment device provided in the embodiment of the second aspect of the present invention can implement all the processes of the cabin sound event time alignment method described in any embodiment of the first aspect above, and the functions of each module and unit in the device and the technical effects achieved are respectively the same as the functions and technical effects achieved by the cabin sound event time alignment method described in the embodiment of the first aspect above, and will not be repeated here.

[0094] An embodiment of the third aspect of the present invention provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the cabin sound event time alignment method described in any embodiment of the first aspect above.

[0095] An embodiment of a fourth aspect of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the cabin sound event time alignment method described in any embodiment of the first aspect above.

[0096] See also Figure 3 , is a structural diagram of an embodiment of an electronic device provided by the present invention.

[0097] An embodiment of the fifth aspect of the present invention provides an electronic device, comprising a processor 21, a memory 22, and a computer program stored in the memory 22 and configured to be executed by the processor 21, wherein the processor 21 implements the cabin sound event time alignment method described in any embodiment of the first aspect when executing the computer program.

[0098] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 22 and executed by the processor 21 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0099] The processor 21 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 21 can also be any conventional processor. The processor 21 is the control center of the electronic device, and various parts of the electronic device are connected using various interfaces and lines.

[0100] The memory 22 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory 22 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory 22 can also be other volatile solid-state storage devices.

[0101] It should be noted that the above electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will understand that Figure 3 The structural diagram shown is only a structural example of the above-mentioned electronic device, and does not constitute a structural limitation of the above-mentioned electronic device. The above-mentioned electronic device may include more or less components than shown in the figure, or a combination of certain components, or different components.

[0102] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for aligning cabin sound events in time, characterized in that: include: Acquire the cabin sound data and flight data to be aligned; Marking all flight events in the flight data and generating a first event time series; Based on a preset time window, the cabin sound data is sliced ​​and features are extracted to obtain a plurality of audio feature sequences; each of the audio feature sequences is input into a cabin sound event recognition model to obtain a corresponding recognition result, and all the recognition results are integrated to obtain a second event time series; Based on the time range of the second event time series, intercepting the first event time series to obtain a first event time subsequence; Dynamic time warping is performed on the first event time subsequence and the second event time sequence, and the time information of all the flight events in the first event time subsequence is mapped to the second event time sequence to obtain the time corresponding to all cabin sound events.

2. The cabin sound event time alignment method according to claim 1, characterized in that: After acquiring the cabin sound data and the flight data to be aligned, the method further includes: Performing a first preprocessing on the flight data to obtain preprocessed flight data; wherein the first preprocessing includes: data cleaning and data conversion; The cabin sound data is subjected to a second preprocessing to obtain preprocessed cabin sound data; wherein the second preprocessing includes: noise reduction processing and signal enhancement processing.

3. The method for aligning cabin sound events in time according to claim 1, characterized in that: The step of marking all flight events in the flight data and generating a first event time series includes: The analysis system based on the rapid access recorder analyzes the flight data, and marks all the flight events in the flight data to obtain a corresponding first event time series.

4. The method for aligning cabin sound events in time according to claim 1, characterized in that: The cabin sound data is sliced ​​and feature extracted based on a preset time window to obtain a plurality of audio feature sequences, including: Dividing the cabin sound data according to the preset time window to obtain a plurality of audio segments; Feature extraction is performed on each of the audio clips to obtain a corresponding audio feature sequence; wherein each of the audio feature sequences includes: Mel-frequency cepstral coefficients and Mel-frequency spectrum data.

5. The cabin sound event time alignment method according to claim 1, characterized in that: The cabin sound event recognition model is obtained by the following steps: Obtain N cabin sound sample data; where N≥1; Performing event labeling on all the cabin sound sample data, and dividing all the event-labeled cabin sound sample data according to the preset time window to obtain a plurality of audio sample segments and corresponding event labels; Extracting features from each of the audio sample segments to obtain a corresponding sample feature sequence; An initial recognition model is created based on a long short-term memory network architecture; all the sample feature sequences and corresponding event labels are used as training data, and the initial recognition model is trained to obtain the cabin sound event recognition model.

6. The cabin sound event time alignment method as claimed in claim 5, characterized in that: The step of dynamically time warping the first event time subsequence and the second event time sequence, and mapping the time information of all the flight events in the first event time subsequence to the second event time sequence to obtain the time corresponding to all the cabin sound events includes: constructing a cumulative distance matrix based on the first event time subsequence and the second event time series; Perform path backtracking in the cumulative distance matrix to obtain the shortest regular path; Based on the shortest regular path, the time information of all the flight events in the first event time subsequence is mapped to the second event time sequence to obtain the time corresponding to all the cabin sound events.

7. A cabin sound event time alignment device, characterized in that: include: A data acquisition module, used to acquire cabin sound data and flight data to be aligned; A flight event marking module, used for marking all flight events in the flight data and generating a first event time series; A cabin sound event recognition module is used to slice and extract features from the cabin sound data based on a preset time window to obtain a plurality of audio feature sequences; each of the audio feature sequences is input into a cabin sound event recognition model to obtain a corresponding recognition result, and all the recognition results are integrated to obtain a second event time series; A sequence interception module, configured to intercept the first event time series based on the time range of the second event time series to obtain a first event time subsequence; A sequence alignment module is used to dynamically time-align the first event time subsequence and the second event time sequence, and map the time information of all the flight events in the first event time subsequence to the second event time sequence to obtain the time corresponding to all cabin sound events.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the cabin sound event time alignment method as described in any one of claims 1 to 6.

9. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the cabin sound event time alignment method as described in any one of claims 1 to 6.

10. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the cabin sound event time alignment method as described in any one of claims 1 to 6 when executing the computer program.

Images