Helicopter voice recognition method and device based on transfer learning and end-to-end model

Abstract

The helicopter voice recognition method and device based on transfer learning and end-to-end model belong to the field of speech recognition technology. In order to solve the problem of low accuracy of helicopter cockpit voice recognition in existing speech recognition technology and the difficulty of recognition model training, the present invention first constructs a voice recognition model with an encoder-decoder structure, and obtains a voice recognition model based on the model transfer learning method by using the model weight parameters corresponding to the trained speech recognition model and the pre-training model; the encoder outputs a high-dimensional feature representation, which then enters the CTC decoder, and the CTC prefix set search algorithm is used to generate the N best results, and the Attention decoder is used to re-score multiple candidate results, and finally the hot words of the voice are enhanced, and the results are sent to the Softmax layer to output; the helicopter cockpit voice recognition model is obtained by merging the acoustic model and the language model with the voice recognition model, which is used to recognize the helicopter cockpit voice to be recognized.

Description

Technical Field

[0001] The invention belongs to the technical field of speech recognition and relates to a helicopter speech recognition method and equipment. Background Art

[0002] As one of the outstanding achievements of aviation technology in the 20th century, helicopters have special vertical take-off and landing and hovering capabilities, which makes them widely used in many different fields, including military, civil, scientific research and construction. However, helicopter crashes have occurred frequently in recent years. Once a crash occurs, the chances of pilots and passengers surviving are very small, and they are very likely to be burned. The Civil Aviation Accident Investigation and Analysis Bureau generally uses black boxes to investigate the causes of air crashes. Black boxes are mainly divided into flight data recorders and cockpit voice recorders. For analyzing the cause of the accident, the voice recorded in the cockpit voice recorder provides a very critical source of information.

[0003] In order to understand the flight process and provide key information for accident investigation, it is necessary to accurately identify the helicopter cockpit voice. Due to the particularity of the helicopter flight environment, the sound signal inside the cockpit voice recorder is relatively complex. Facing strong background noise and radio interference, the content of the conversation is not common words, which seriously affects the accuracy of helicopter cockpit voice recognition. In addition, since there are currently no large number of annotated data sets in the field of helicopter voice, it is difficult to train based on deep learning, and it will affect the accuracy of the trained model detection, which makes helicopter voice recognition challenging.

[0004] The current speech recognition technology is mainly aimed at general fields, focusing on the accurate recognition of standard Mandarin in weak noise environments. At present, if you want to understand the information contained in the voice data, you can only rely on relevant professionals to distinguish it through human ears. When studying the information of the voice data, it takes a huge amount of work to complete the analysis of the voice data. Therefore, using the current sound analysis method and speech recognition technology to accurately identify the voice data contained in the cockpit voice recorder is of great significance for the subsequent analysis and investigation of the cause of the accident. Summary of the invention

[0005] The present invention aims to solve the problem of low accuracy of helicopter cockpit voice recognition by existing speech recognition technology and the problem of great difficulty in training recognition models.

[0006] A helicopter voice recognition method based on transfer learning and an end-to-end model first obtains the helicopter cockpit voice to be recognized, then performs sonic perturbation processing on the helicopter cockpit voice data to be recognized, and then extracts acoustic features; the extracted acoustic features are recognized based on the helicopter cockpit voice recognition model;

[0007] The helicopter cockpit voice recognition model is obtained by the following steps:

[0008] First, an end-to-end speech recognition model is trained. The end-to-end speech recognition model adopts an encoder-decoder structure, and the encoder of the speech recognition model adopts a Conformer encoder. At the same time, the pre-trained model open sourced on the WeNet speech recognition tool is called. Based on the model-based transfer learning method, the weight parameters of the speech recognition model trained on the open source dataset are combined with the pre-trained model structure to obtain a helicopter voice recognition model. The helicopter voice recognition model adopts an encoder-decoder structure, in which the encoder is a Conformer encoder. During the migration process, the weight parameters of the speech recognition model trained on the open source dataset are saved, and the parameters of the pre-trained model are changed according to the parameters of the speech recognition model, and then trained on the helicopter voice dataset. Then, when migrating the pre-trained model, the softmax layer is removed to obtain the helicopter voice recognition model.

[0009] In the test phase of the speech recognition model, the encoder outputs a high-dimensional feature representation, which then enters the CTC decoder. The CTC decoding result is used as the intermediate result, and the CTC prefix set search algorithm is used to generate the N best results. The Attention decoder is then used to re-score multiple candidate results, and finally the speech hot words are enhanced and sent to the Softmax layer to obtain the output result.

[0010] A fusion language model is built on the basis of the speech recognition model as the final helicopter cockpit speech recognition model; the process of building a fusion language model on the basis of the speech recognition model includes the following steps:

[0011] The speech recognition model obtained based on transfer learning is used as the acoustic model. In the test decoding stage, it is decoded by shallow fusion with the language model, and the character sequence is obtained by CTC decoding. Then the language model is added to CTC to calculate the probability of the word sequence. Finally, the character sequence and the word sequence are combined to calculate the probability of each prefix; thus, a fusion language model is obtained that uses the speech recognition model as the acoustic model and is fused with the language model.

[0012] Furthermore, before performing the sonic disturbance processing on the helicopter cockpit voice data, it is necessary to perform noise reduction processing on the helicopter cockpit voice data. The specific process includes the following steps:

[0013] For noise within 1kHz, active noise control is used for preliminary noise reduction, and for noise above 1kHz, secondary noise reduction is performed.

[0014] Furthermore, the preliminary noise reduction adopts an adaptive minimum mean square error algorithm or a recursive least squares method to perform noise reduction processing; the secondary noise reduction adopts a spectral subtraction method or a Wiener filtering method to perform noise reduction processing.

[0015] Furthermore, in the process of recognizing the extracted acoustic features based on the helicopter cockpit voice recognition model, the extracted acoustic features are downsampled and then sent to the linear layer, and then sent to the Conformer encoder for further processing after passing through the linear layer.

[0016] Furthermore, in the process of performing sonic disturbance processing on the helicopter cockpit voice data, the helicopter cockpit voice data is subjected to 0.8 times sonic disturbance processing.

[0017] Furthermore, the end-to-end speech recognition model is trained using an open source Chinese speech dataset.

[0018] Furthermore, the language model is an N-gram language model or a Transformer language model.

[0019] Furthermore, the acoustic feature is an Fbank feature.

[0020] A computer storage medium stores at least one instruction, which is loaded and executed by a processor to implement a helicopter voice recognition method based on transfer learning and an end-to-end model.

[0021] A helicopter voice recognition device based on transfer learning and an end-to-end model, the device comprising a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement a helicopter voice recognition method based on transfer learning and an end-to-end model.

[0022] Beneficial effects:

[0023] 1. The present invention utilizes deep learning methods and techniques to realize automatic recognition of helicopter cockpit voice, overcoming the deficiency of traditional methods that require manual identification of voice data.

[0024] 2. The present invention performs noise reduction processing on the voice data recorded by the cockpit voice recorder, effectively eliminates the noise, thereby obtaining relatively pure voice data and reducing the influence of noise on the voice recognition effect.

[0025] 3. The present invention manually annotates the speech data set, trains the helicopter cockpit speech recognition model based on the transfer learning method, and combines the language model with the end-to-end speech recognition model to improve the recognition accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a schematic diagram of the construction process of a speech recognition model that integrates a language model.

[0027] FIG. 2( a ) is a time-frequency domain waveform of the speaker's voice of the helicopter voice; FIG. 2( b ) is a time-frequency domain waveform of the background noise of the helicopter voice.

[0028] Figure 3 It is the time-frequency domain waveform of the helicopter voice before and after the background noise is denoised.

[0029] Figure 4 This is a comparison chart of the overall denoising effect of the helicopter voice clip.

[0030] Figure 5 This is a schematic diagram of the Conformer encoder structure.

[0031] Figure 6 This is the migration structure diagram of the helicopter voice recognition model. DETAILED DESCRIPTION

[0032] Specific implementation method 1: Combination Figure 1 To explain this embodiment,

[0033] This implementation is a helicopter voice recognition method based on transfer learning and end-to-end model, and the specific steps are as follows:

[0034] Step 1: Organize the voice data to be annotated

[0035] The voice recorded by the helicopter flight parameter system is read, and the noise source and characteristics of the voice data are analyzed. The voice data recorded by the cockpit voice recorder is complex, and there are many types of background noise. By analyzing a large number of voice signals with speaker voices and background noise, it is found that most of the background noise in the voice exists in the low-frequency part. Therefore, this embodiment uses active noise control to perform preliminary noise reduction based on the low-frequency noise in the background sound of the voice. Only using active noise control cannot completely eliminate the noise. Then, for the residual noise in each frequency band after the preliminary noise reduction, other noise reduction algorithms are selected for secondary filtering to obtain a relatively pure voice signal.

[0036] The cockpit voice recorder records complex information, and there are many types of background noise, mainly three categories: the first is the mechanical noise generated when the helicopter rotates, such as the mechanical noise of the engine, rotor, tail rotor, transmission system and other components, which is the main noise of the helicopter; the second is the aerodynamic noise generated by the resistance of the air when the helicopter flies, including wind noise, rotor cutting noise, tail spin noise, etc.; the third is the electronic noise generated by the electronic equipment and circuits on the helicopter, such as the noise of the motor, switch, transformer, amplifier and other components. In addition to these three types of noise, it actually also includes the sound of the pilot operating the components and some signal interference noise such as radio sounds.

[0037] After research, it is found that the spectrum of mechanical noise usually presents a wide passband characteristic, and its main energy is distributed in the low-frequency and medium-frequency range, especially in the frequency band of 20Hz to 500Hz. The spectrum of aerodynamic noise usually presents a narrow passband characteristic, and its main energy is distributed in the high-frequency range, such as the frequency band of 1kHz to 10kHz, and some energy is also distributed in the medium and low frequency range. The spectrum of electronic noise usually presents a relatively flat passband characteristic, and its main energy is distributed in the high-frequency range, such as the frequency band of 1kHz to 10kHz, and some energy is also distributed in the low-frequency range.

[0038] For low-frequency noise within 1kHz, active noise control is used for preliminary noise reduction. The adaptive algorithm in active noise control can be used to reduce noise in the low-frequency band. The adaptive algorithm includes but is not limited to the adaptive minimum mean square error algorithm and the recursive least squares method, such as the FxLMS algorithm.

[0039] Since the signal-to-noise ratio of the voice signal after the initial noise reduction is still not high, it can be observed from the spectrum that there is residual noise in each frequency band, so secondary noise reduction processing is required for high-frequency noise. In the secondary noise reduction, multiple noise reduction algorithms can be selected for comparative analysis, including but not limited to spectral subtraction, improved multi-window spectral subtraction and Wiener filtering. After the overall noise reduction of the voice signal, it can be found that the background noise has been effectively filtered.

[0040] Step 2: Labeling helicopter voice data

[0041] Considering the sensitivity of helicopter voice data, it cannot be uploaded to the Internet for annotation using tools, so the voice data after noise reduction is manually annotated. First, the sound segment without the speaker is removed, then the audio is exported and annotated, and the annotated data is again proofread and reviewed for audio and text; finally, it is saved in the specified path to complete the construction of the voice data set.

[0042] The constructed audio comes from the voice recorded by the helicopter flight data system. During the entire flight of the helicopter, the flight data system collects and records the entire process. These voices include but are not limited to conversations between the crew, conversations between the crew and the tower, and broadcast voices received by the system.

[0043] Step 3: Sound velocity disturbance of voice data

[0044] Depending on the flight status of the helicopter, the speech speed of the pilot and other speakers will also be different. By analyzing the speed of speech in the speech dataset, it is found that the average speed of speech is about 1.2 times that of the open source dataset. In order to make the constructed speech dataset closer to the open source Mandarin dataset and further improve the accuracy of speech recognition, the speed of speech disturbance processing can be performed on the entire speech dataset.

[0045] Since the speaking speed of the pilot and other speakers varies according to different flight states, all the currently available data are divided according to different flight states, where the voices in different flight states include but are not limited to conversations before helicopter takeoff and after landing, conversations during takeoff and climb, conversations during cruising, conversations during descent and approach, and conversations after landing.

[0046] Step 4: Speech Recognition Based on Transfer Learning

[0047] Due to the limited voice data, building a model using only the existing data in the field will not achieve good results. The use of transfer learning methods can make full use of pre-trained models of general domain data sets (open source data sets) and improve the performance of the model in the target field by fine-tuning or reusing some model parameters.

[0048] The language of the helicopter voice data is highly similar to that of the general domain dataset. We use the open source Chinese speech dataset Aishell-1 in the general domain to train an end-to-end speech recognition model (the speech recognition model is a model trained and processed for the open source Chinese speech dataset in the general domain). At the same time, we use the pre-trained model of the WeNet speech recognition tool, which uses a richer Chinese speech dataset in the general domain.

[0049] The speech recognition model used in this embodiment (the speech recognition model is a model trained and processed for helicopter speech data) adopts a Conformer encoder. In order to facilitate the training of the convolutional network layer, a residual structure is introduced in the Conformer block structure. At the same time, the multi-head self-attention layer is adjacent to the convolutional layer, which better integrates the two different feature extraction methods, thereby improving the expression ability and generalization ability of the model.

[0050] Based on the model transfer learning method, the model weight parameters corresponding to the trained end-to-end speech recognition model and the weight parameters of the open source pre-trained model are transferred to the speech recognition model of low resources (labeled helicopter voice data set), thereby effectively improving the model performance of voice under low resources. In fact, WeNet is a speech recognition toolkit. Both the open source Chinese speech data set and the speech data set are trained using the WeNet speech recognition toolkit.

[0051] More specifically,

[0052] In the process of training an end-to-end speech recognition model using an open source Chinese speech dataset, the end-to-end speech recognition model actually adopts an encoder-decoder structure. The encoder of the speech recognition model adopts a Conformer encoder, which first extracts the acoustic features of the speech dataset and then feeds the extracted acoustic features into the Conformer encoder. Figure 5 As shown in the figure, the main core part of the Conformer encoder is the Conformer block. Before the acoustic features of the speech signal are encoded by the Conformer block, they first enter the preprocessing stage, which mainly includes speech enhancement, convolution downsampling and pooling layer; the Conformer block consists of a multi-head attention module, a convolutional network module and a feedforward network module. In order to facilitate training, there are residual connections on the feedforward network module, the convolutional module and the multi-head attention module. At the same time, the convolutional module is set after the multi-head attention module and connected to the multi-head attention module to enhance the effect. Finally, the high-dimensional feature representation is output and then enters the decoder.

[0053] The end-to-end speech recognition model used in the migration structure is the basic model, which is basically consistent with the target task model in model structure. In the basic model training process of the Aishell-1 open source data set, the acoustic features of the Aishell-1 data set are first extracted. The acoustic features are vector representations of the spectrum space extracted based on the vocalization mechanism or the human ear perception mechanism. In some embodiments, speech recognition can use filter banks Fbank (Filter-bank) and Mel-Frequence Cepstral Coefficients MFCC (Mel-Frequence Cepstral Coefficient). Fbank features will be more advantageous in an end-to-end architecture, because MFCC feature extraction undergoes discrete cosine transform, which causes the loss of information in the nonlinear part of the original signal, and Fbank features retain the original signal data more completely. Therefore, it is preferred to use Fbank features.

[0054] The extracted acoustic features are then downsampled and sent to the linear layer, and then to the Conformer encoder. In the Conformer encoder, they pass through the feedforward network layer and multi-head attention module in the Conformer block, and finally output a high-dimensional feature representation, and then enter the CTC decoder. During the decoding process, the decoder generates a text sequence based on the feature sequence output by the encoder, and a corresponding text output is generated for each time step.

[0055] The speech recognition model actually also adopts an encoder-decoder structure. The encoder of the speech recognition model adopts the same Conformer encoder as the Conformer encoder in the end-to-end speech recognition model, and finally outputs a high-dimensional feature representation, which then enters the decoder. During the decoding process, the decoder generates a text sequence based on the feature sequence output by the encoder, and generates a corresponding text output at each time step.

[0056] During the migration process, the end-to-end model trained by the Aishell-1 dataset refers to the "speech recognition model". The parameters of the trained "speech recognition model" are saved, and the parameters of the "Conformer pre-trained model" are changed according to the parameters of the "speech recognition model". Then, the model parameters are trained on the helicopter voice dataset. Then, when migrating the Conformer pre-trained model, the softmax layer is removed to train a "speech recognition model (the model to be trained for helicopter voice)", which reduces the source domain data training process. Migrating the weight parameters of the trained speech recognition model and the weight parameters of the open source pre-trained model to the low-resource speech recognition model can effectively reduce the training time.

[0057] In the training process of the speech data set, the speech data set is firstly processed with sound velocity perturbation. The speech data set processed with sound velocity perturbation can achieve better recognition effect for speech recognition. Then, the acoustic features of the speech data set are extracted, and the extracted acoustic features are downsampled and sent to the linear layer. After the linear layer, they are sent to the Conformer encoder, and finally the high-dimensional feature representation is output. Then, it enters the CTC decoder, and the decoding result of CTC is used as the intermediate result. Then, the CTC prefix set search algorithm is used to generate the N best results. Then, the Attention decoder is used to re-score multiple candidate results. Finally, the hot words of the speech are enhanced to improve the recognition accuracy of the proprietary vocabulary in the speech, and finally the output result is obtained by sending it to the Softmax layer.

[0058] The structure of the Attention decoder is the same as that of the Transformer decoder. Each layer contains a four-level structure, including two levels of attention. The first level input information only comes from the output of the previous layer of decoder, and the second level also covers the output from the encoder. Position embedding is added to ensure the order information of the language, and it is calculated through the self-attention layer and the fully connected layer.

[0059] In fact, the present invention first trains the model, and then uses the hot word enhancement technology for recognition during testing. Since the voice data contains a lot of professional terms, such as "airspace", "tower", etc., and there are also many place names and landmark names, the hot word enhancement technology is used to add voice hot words to improve the recognition accuracy for the professional terms, place names, landmark names and proprietary vocabulary contained in the voice data. Hot word enhancement is equivalent to adding some prior knowledge to the speech recognition system. By calculating the bias score in the hot word, it is interpolated with the basic model using shallow fusion in the beam search process, and the score of each word in the hot word is specified. During the decoding process, when the corresponding prefix is ​​matched, the corresponding score reward will be obtained. In order to prevent the phenomenon that the prefixes are the same and not completely matched, a backoff arc is added to remove the previously rewarded score through this arc. If any hot word that cannot be completely matched is encountered, the backoff arc removes the score reward obtained by the previous word. The specific process of hot word enhancement includes the following steps:

[0060] First, determine the hot word list that needs to be enhanced in the speech data set. Hot words are words that appear less frequently in the data set and have no logical connection between the previous and the next words. When setting the hot word score for the speech, if the recognition path is wrong, a hot word score that is too high will cause the correct recognition results to be trimmed, so it is necessary to set a suitable hot word score. Finally, when performing weighted training on the speech data (weighted training is also performed during the test process), it is necessary to specify the current text file, and each line must contain the context information of the hot word.

[0061] The hot word list includes but is not limited to place names, airport names, professional terms in the aviation field, and some landmark names. When using hot word enhancement technology for enhancement, the probability values ​​of hot words specified in the speech are weighted into the recognition network, thereby increasing the probability of these specialized words appearing in the model and enhancing the weight of speech hot words in the recognition results.

[0062] Step 5: Build a speech recognition model that integrates the language model (i.e., the final helicopter cockpit speech recognition model):

[0063] Considering the particularity of spoken text, the correlation between the previous and next words is quite different from the open source data set. If the speech model is not integrated during decoding, the language information of the spoken text cannot be well learned. Therefore, the language model is integrated into the end-to-end speech recognition model so that the model can learn the language knowledge in the speech and further improve the performance of the model. Therefore, the present invention uses the speech recognition model based on transfer learning as the acoustic model, and decodes it with the language model in a shallow fusion manner during the test decoding stage. The character sequence is obtained by CTC decoding, and then the language model is added to the CTC to calculate the probability of the word sequence. Finally, the character sequence and the word sequence are combined to calculate the probability of each prefix.

[0064] In this embodiment, the N-gram language model and the Transformer language model are trained based on the text corpus of speech, and the trained N-gram language model and the Transformer language model are used as language models respectively. The word error rate after the fusion language model is used as the evaluation index, and the relatively excellent language model is finally selected through analysis (the evaluation standard is the word error rate, and the Transformer language model with a lower word error rate is finally selected). In the test decoding stage, the speech recognition model based on transfer learning is used as the acoustic model, and is decoded with the language model in a shallow fusion manner in the test decoding stage. The character sequence is obtained by decoding, and then the language model is added to calculate the probability of the word sequence. Finally, the two sequences are combined to calculate the probability of each prefix, so that the recognition of speech can achieve a lower word error rate.

[0065] The present invention actually trains multiple language models based on the text corpus of speech. In the process of comparison, the language models include but are not limited to statistical language models and neural network language models. The N-gram language model is the most classic method of statistical language models. N-gram represents the combination probability between N words, and each prediction variable is only related to the context of length N-1. The Transformer language model is a deep neural network model based on the self-attention mechanism. The self-attention mechanism realizes the interactive modeling of different positions in the input sequence, adopts position encoding to maintain the position information of the vocabulary in the sequence, and uses residual connection and layer normalization to speed up the training speed and model convergence.

[0066] Step 6: Based on the speech recognition model of the fusion language model finally obtained, a helicopter cockpit speech recognition platform is built to perform helicopter cockpit speech recognition:

[0067] By calling the trained helicopter cockpit voice recognition model, localized recognition of the conversation voice is achieved, and the recognized text is exported into text format. Specific implementation method 2:

[0069] This embodiment is a computer storage medium, in which at least one instruction is stored. The at least one instruction is loaded and executed by a processor to implement the helicopter voice recognition method based on transfer learning and end-to-end model.

[0070] It should be understood that the instructions include computer program products, software or computerized methods corresponding to any method described in the present invention; the instructions can be used to program a computer system, or other electronic device. Computer storage media may include readable media on which instructions are stored, which may include but are not limited to magnetic storage media, optical storage media; magneto-optical storage media include read-only memory ROM, random access memory RAM, erasable programmable memory (e.g., EPROM and EEPROM) and flash memory layers, or other types of media suitable for storing electronic instructions. Specific implementation method three:

[0072] This embodiment is a helicopter voice recognition device based on transfer learning and end-to-end model, the device includes a processor and a memory, it should be understood that including any device including a processor and a memory described in the present invention, the device may also include other units and modules that perform display, interaction, processing, control, etc. and other functions through signals or instructions;

[0073] At least one instruction is stored in the memory, and the at least one instruction is loaded and executed by the processor to implement the helicopter voice recognition method based on transfer learning and end-to-end model.

[0074] Example

[0075] The helicopter voice recognition method based on transfer learning and end-to-end model described in this embodiment has the following specific steps:

[0076] 1. Organize the voice data to be annotated

[0077] The voice data recorded by the cockpit voice recorder is complex and contains many types of background noise. By analyzing a large number of voice signals with speaker voices and background noise, it is found that most of the background noise in the voice exists in the low-frequency part. Figure 1 FIG2 shows the time-frequency diagram of two segments of speech signals with speaker voices, and FIG3 shows the time-frequency diagram of two segments of background noise. Therefore, a method for reducing low-frequency noise can be used, for example, an adaptive algorithm in active noise control can be used to reduce noise in the low-frequency band, wherein the adaptive algorithm includes but is not limited to an adaptive minimum mean square error algorithm and a recursive least squares method.

[0078] This embodiment first uses an active noise control method to reduce the noise in the low frequency band to achieve preliminary noise suppression. The signal-to-noise ratio of the voice signal after preliminary noise reduction is still not high. It can be observed from the spectrum that there is residual noise in each frequency band, such as Figure 3 As shown in the figure, it can be seen from the background noise spectrum that there is still some noise in the high-frequency part, so secondary noise reduction processing is required for high-frequency noise, using LMS adaptive filtering and multi-window spectrum subtraction processing, as shown in Figure 4 As shown in FIG. 1 , a waveform comparison is made before and after the overall filtering of a certain signal, which proves that the noise is effectively filtered out after the overall noise reduction of the voice signal.

[0079] By performing noise reduction on the speech data, the speech data can be segmented and labeled more accurately, which facilitates the recognition and analysis of the speech data. At the same time, these noise reduction methods also provide the necessary data foundation for subsequent speech recognition based on the end-to-end model, so that different speech data can be recognized more accurately.

[0080] Therefore, the present invention can better construct a voice data set similar to a general field data set by performing secondary noise reduction processing on the voice data, improve the understanding and mastery of voice recognition, and thus more accurately and efficiently identify the voice data of helicopters.

[0081] 2. Helicopter voice data annotation

[0082] After the audio signal of the speech is processed for noise reduction and high-quality speech data is obtained, it is necessary to construct a speech data set that meets the training requirements. Due to the sensitivity of flight data, it is almost impossible to upload helicopter speech data to the Internet, and there is no open source speech recognition corpus for this field. Therefore, it is necessary to manually annotate it. First, the silent segments of the unprocessed long segments of speech data are removed, and then the audio is exported and annotated. In order to ensure the accuracy of the constructed speech data, ensure that the text annotation is correct and the audio text matches, the annotated data is again proofread and reviewed for audio-text, and finally stored in the specified path to complete the construction of the helicopter speech data set.

[0083] 3. Sound velocity disturbance of voice data

[0084] Depending on the flight state of the helicopter, the speech speed of the pilot and other speakers will also be different. During the flight or cruise of the helicopter, the flight commander must provide the pilot with control instructions, intelligence information and warning signals in a timely and effective manner. Therefore, the speed of speaking is faster than in daily conversations, and the speech rate may be as high as twice the normal rate, but the conversation before takeoff and after landing is relatively slow. The speech data set is divided according to different flight states, where the speech in different flight states includes but is not limited to the conversation before takeoff and after landing of the helicopter, the conversation during takeoff and climb, the conversation during cruise, the conversation during descent and approach, and the conversation after landing. After statistics, it is found that the average speed of speech is about 1.2 times that of the open source data set. In order to make the established target domain data set closer to the open source data set, the speech speed of the existing speech data set in different states is analyzed, and the corresponding speed adjustment is made respectively by speed perturbation. Considering that the speech recognition of unknown input data in the later stage cannot determine the state, the speed perturbation processing is uniformly performed on the entire speech data, such as 0.8 times the speed perturbation processing.

[0085] 4. Speech Recognition Based on Transfer Learning

[0086] To address the problem of insufficient available voice data to support model training, the present invention first trains an end-to-end model on an open source dataset, then migrates the model's training parameters to the field of speech recognition, and improves the performance of the model in the target field by fine-tuning or reusing some model parameters.

[0087] The present invention uses a model-based transfer learning method. The language of the speech is Chinese, and has a high similarity with the open source Mandarin data set. With the help of the open source Chinese speech data set in the general field, an end-to-end speech recognition model is first trained. For other non-public and richer Chinese speech data sets in the general field, its open source pre-trained model is called, and the trained model weight parameters and the weight parameters of the open source pre-trained model are respectively migrated to the low-resource speech recognition model, and then the model parameters are trained and fine-tuned on the speech data set, thereby effectively improving the model performance of speech under low resources.

[0088] The model structure of the speech recognition solution based on transfer learning is as follows Figure 6As shown in the figure, the main core of the model includes but is not limited to the Conformer encoder. In the Conformer block structure, a residual structure is introduced to facilitate the training of the convolutional network layer. At the same time, the multi-head self-attention layer is adjacent to the convolutional layer, which better utilizes two different feature extraction methods and improves the expression and generalization capabilities of the model. The decoder part includes but is not limited to the Attention decoder. Each layer contains a four-level structure, including two levels of attention. The first-level input information only comes from the output of the previous layer of decoder, and the second level also covers the output from the encoder. In the decoding part, it includes but is not limited to the CTC prefix beam search method to decode the speech recognition model. First, the CTC decoding result is used as the intermediate result, and the CTC prefix set search algorithm is used to generate the N best results. Then the Attention decoder re-scores multiple candidate results, and finally the speech hot words are weighted to improve the recognition accuracy of the proprietary vocabulary in the speech.

[0089] 5. Hot word enhancement method for voice data

[0090] In the study of speech recognition, the recognition of some common words can achieve a higher accuracy through migration, but the speech contains many professional terms, such as "airspace", "tower", etc., and there are also many place names and landmark names. For these proprietary words, the recognition accuracy is low. The present invention improves the recognition accuracy by using hot word enhancement technology and adding speech hot words.

[0091] By calculating the bias score in the hot word, using shallow fusion to interpolate it with the basic model during the beam search process, specifying the score of each word in the hot word, when the corresponding prefix is ​​matched during the decoding process, the corresponding score reward will be obtained. In order to prevent the phenomenon of identical prefixes but not completely matching, a backoff arc is added to remove the previously rewarded score through this arc. If any hot word that cannot be completely matched is encountered, the backoff arc removes the score reward obtained by the previous word.

[0092] When performing hotword weighted training on speech, you first need to determine the hotword list to be enhanced in the speech data set. By summarizing the existing speech data set, the hotword list includes but is not limited to place names, airport nouns, professional nouns in the aviation field, and some landmark nouns. These words are words that appear less frequently in the data set and have no logical connection between the front and back. When training, you need to specify the current text file, and each line must contain the context information of the hotword. When setting the hotword score for speech, generally speaking, the higher the hotword score, the higher the speech recognition accuracy. However, if the recognition path is wrong, a hotword score that is too high will cause the correct recognition result to be clipped. Therefore, when performing hotword weighted training on speech, you need to set a suitable hotword score.

[0093] 6. Speech recognition with language model integration

[0094] The present invention takes into account the particularity of speech text, and the correlation between the previous and next words is quite different from the open source data. If the speech model is not integrated during decoding, the language information of the speech text cannot be well learned. Several language models are trained and compared based on the speech text corpus, wherein the language model includes but is not limited to a statistical language model and a neural network language model. For example, the statistical N-gram language model and the neural network-based Transformer language model use the word error rate after the fusion language model as the evaluation index, as shown in Table 1, and finally select a relatively excellent Transformer language model so that the speech recognition can achieve a lower word error rate.

[0095] Table 1. Comparison of the effects of N-gram language model and Transformer language model

[0096] Language Model Word Error Rate (WER) 5-gram+Kneser-Ney 13.96％ Transformer 12.58％

[0097] The most commonly used statistical language model is the N-gram language model, which is used to predict the next word given a word sequence. At the same time, the use of smoothing technology can solve the problem of inaccurate estimation caused by data scarcity. Its principle is based on the Markov hypothesis, that is, the appearance of the current word only depends on the previous N-1 words. The N-gram language model estimates the probability distribution of word sequences by counting the word frequency and phrase frequency in the training corpus. Specifically, it divides the text into a sequence of N consecutive words and calculates the probability of each phrase appearing. Based on the probability distribution, the possibility of the next word can be predicted. The N-gram language model is simple and efficient, but it has problems with data sparsity and limited context dependency.

[0098] The Transformer language model is a deep neural network model based on the self-attention mechanism. It uses a multi-layer encoder and decoder structure. Through the self-attention mechanism, it realizes the interactive modeling of different positions in the input sequence and effectively captures the long-distance dependencies in the sequence. First, the input word sequence is represented by word embedding. The position encoding is combined with the word vector after embedding to obtain the position information of the word, which is sent to the encoder. The position encoding is added because the Transformer uses pure self-attention to capture the connection between tokens. It cannot recognize the order of tokens in the sequence. The feature vector processed by the Transformer encoder layer obtains the encoded output. The linear layer is used as the classification layer. The vector corresponding to the vocabulary is mapped with the model's encoding and output as a vector. After the vector is processed by the Softmax function, the value of each dimension represents the probability of the word being predicted at the current moment. Therefore, the final output of the model is a probability distribution, which represents the probability of each word being predicted at the current moment.

[0099] 7. Build a helicopter cockpit voice recognition platform to perform localized voice recognition:

[0100] By calling the trained end-to-end helicopter cockpit speech recognition model, localized recognition of the conversation speech is achieved, and the recognized text is exported into text format.

[0101] The above calculation examples of the present invention are only used to explain the calculation model and calculation process of the present invention in detail, and are not intended to limit the implementation methods of the present invention. For ordinary technicians in the relevant field, other different forms of changes or modifications can be made based on the above description. It is impossible to list all the implementation methods here. All obvious changes or modifications derived from the technical solution of the present invention are still within the scope of protection of the present invention.

Claims

1. A helicopter voice recognition method based on transfer learning and end-to-end model, characterized in that: First, the helicopter cockpit voice to be identified is obtained, and then the helicopter cockpit voice data to be identified is subjected to sonic disturbance processing. During the sonic disturbance processing, the helicopter cockpit voice data is subjected to 0.8 times sonic disturbance processing; then the acoustic features are extracted; The extracted acoustic features are recognized based on the helicopter cockpit voice recognition model; Before performing sonic disturbance processing on the helicopter cockpit voice data, it is necessary to perform noise reduction processing on the helicopter cockpit voice data. The specific process includes the following steps: For noise within 1kHz, active noise control is used for preliminary noise reduction, and the initial noise reduction uses an adaptive minimum mean square error algorithm or a recursive least squares method for noise reduction processing; for noise above 1kHz, secondary noise reduction is performed after preliminary noise reduction, and the secondary noise reduction uses spectral subtraction or Wiener filtering for noise reduction processing; In the process of recognizing the extracted acoustic features based on the helicopter cockpit voice recognition model, the extracted acoustic features are downsampled and then sent to the linear layer, and then sent to the Conformer encoder for further processing; The helicopter cockpit voice recognition model is obtained by the following steps: First, an end-to-end speech recognition model is trained. The end-to-end speech recognition model adopts an encoder-decoder structure, and the encoder of the speech recognition model adopts a Conformer encoder. At the same time, the pre-trained model open sourced on the WeNet speech recognition tool is called. Based on the model-based transfer learning method, the weight parameters of the speech recognition model trained on the open source dataset are combined with the pre-trained model structure to obtain a helicopter voice recognition model. The helicopter voice recognition model adopts an encoder-decoder structure, in which the encoder is a Conformer encoder. During the migration process, the weight parameters of the speech recognition model trained on the open source dataset are saved, and the parameters of the pre-trained model are changed according to the parameters of the speech recognition model, and then trained on the helicopter voice dataset. Then, when migrating the pre-trained model, the softmax layer is removed to obtain the helicopter voice recognition model. In the test phase of the speech recognition model, the encoder outputs a high-dimensional feature representation, which then enters the CTC decoder. The CTC decoding result is used as the intermediate result, and the CTC prefix set search algorithm is used to generate the N best results. The Attention decoder is then used to re-score multiple candidate results, and finally the speech hot words are enhanced and sent to the Softmax layer to obtain the output result. A fusion language model is built on the basis of the speech recognition model as the final helicopter cockpit speech recognition model; the process of building a fusion language model on the basis of the speech recognition model includes the following steps: The speech recognition model obtained based on transfer learning is used as the acoustic model. In the test decoding stage, it is decoded by shallow fusion with the language model, and the character sequence is obtained by CTC decoding. Then the language model is added to CTC to calculate the probability of the word sequence. Finally, the character sequence and the word sequence are combined to calculate the probability of each prefix; thus, a fusion language model is obtained that uses the speech recognition model as the acoustic model and is fused with the language model.

2. The helicopter voice recognition method based on transfer learning and end-to-end model according to claim 1 is characterized in that: The end-to-end speech recognition model is trained using an open source Chinese speech dataset.

3. The helicopter voice recognition method based on transfer learning and end-to-end model according to claim 2 is characterized in that: The language model is an N-gram language model or a Transformer language model.

4. The helicopter voice recognition method based on transfer learning and end-to-end model according to claim 3 is characterized in that: The acoustic feature is the Fbank feature.

5. A computer storage medium, characterized in that: The storage medium stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the helicopter voice recognition method based on transfer learning and end-to-end model as described in any one of claims 1 to 4.

6. A helicopter voice recognition device based on transfer learning and end-to-end model, characterized in that: The device includes a processor and a memory, wherein the memory stores at least one instruction, and the at least one instruction is loaded and executed by the processor to implement the helicopter voice recognition method based on transfer learning and end-to-end model as described in any one of claims 1 to 4.

Images