Streaming audio language identification method and system

Abstract

The invention discloses a streaming audio language recognition method and system, and belongs to the technical field of language recognition. The implementation method of the invention comprises the following steps: 1, preprocessing original audio data by using a voice activity detection method to obtain language recognition training data; 2, performing feature extraction on the language recognition training data; 3, constructing an encoder and decoder model and carrying out language recognition feature training; 4, inputting the language recognition test data into the trained encoder and decoder model to obtain language recognition audio data, and forming an audio data stream from the language recognition audio data in a data accumulation mode; 5, performing activity detection on the audio data stream by using a voice activity detection method; 6, performing window-level language discrimination on the audio data passing the activity detection; specifically, the method comprises the following steps: comparing audio data of a current window with audio data of a previous window to obtain a timestamp and a language result of a current state; compared with the prior art, multilingual mixed speech recognition is realized in a streaming scene.

Description

Technical Field

[0001] The invention relates to a streaming audio language recognition method and system, belongs to the technical field of language recognition, and is applied to mixed speech recognition in streaming scenarios. Background Art

[0002] In recent years, cross-language communication has become more and more frequent. In order to break the communication barriers between countries around the world, speech translation technology has attracted more and more attention. Among them, the integrated approach of speech recognition and machine translation is a mainstream solution for current speech translation technology. In this solution, the accuracy of speech recognition plays a key role in the overall performance. Mixed-language ASR has the ability to recognize multiple languages, but compared with single-language ASR, its performance will be greatly reduced. Therefore, the integrated approach of language recognition and single-language ASR to achieve multi-language ASR has become a more commonly used solution in the industry.

[0003] In speech translation tasks, in addition to the accuracy of the translation results, the real-time nature of the returned results is also an important indicator for measuring system performance. Therefore, the traditional sentence-level language recognition and monolingual ASR ensemble approach are no longer applicable.

[0004] Therefore, how to achieve multi-language mixed speech recognition in streaming scenarios has become an urgent problem to be solved. Summary of the invention

[0005] In order to solve the technical problem of realizing multi-language mixed speech recognition in a streaming scenario, the present invention aims to provide a streaming audio language recognition method and system to integrate language recognition and monolingual ASR.

[0006] The objective of the present invention is achieved through the following technical solutions:

[0007] The present invention discloses a method for identifying language of streaming audio, comprising the following steps:

[0008] Step 1: Use the voice activity detection method to preprocess the original audio data to obtain language recognition training data;

[0009] Step 2: Extract features from the language recognition training data, and further use Fourier transform and Mel filter bank to obtain language recognition features;

[0010] Step 2.1: Use the Fourier transform method to convert the time domain features into frequency domain features for the language recognition training data;

[0011] Step 2.2: Convert the frequency domain features into language recognition features through Mel filter bank filtering;

[0012] Step 3: Build the encoder-decoder model and perform language recognition feature training;

[0013] Step 3.1: Build an encoder-decoder model for language recognition feature training;

[0014] Step 3.2: Use the encoder-decoder model to train language recognition features;

[0015] Step 3.2.1: In the encoder-decoder model, the encoder parameters are initialized with the speech recognition encoder parameters; the decoder parameters are initialized with random variables;

[0016] Step 3.2.2: Train the encoder-decoder model in a loop iterative manner;

[0017] Step 4: Input the language recognition test data into the trained encoder-decoder model to obtain language recognition audio data, and form an audio data stream by accumulating the language recognition audio data;

[0018] Step 5: Perform activity detection on the audio data stream using a voice activity detection method;

[0019] Step 6: Perform window-level language discrimination on the audio data that has passed the activity detection; specifically, compare the audio data of the current window with the audio data of the previous window to obtain the timestamp and language result of the current state;

[0020] Step 6.1: When the current window is the starting window, return the timestamp and language result of the current state;

[0021] Step 6.2: When the previous window is a silent window, the timestamp and language result of the current state are returned;

[0022] Step 6.3: When the previous window is not a silent window and the current window is not a start window, the current language result is compared with the language result at the previous moment to obtain the timestamp and language result of the current state;

[0023] Step 6.3.1: When the current language result is the same as the previous language result, return the timestamp and language result of the current state;

[0024] Step 6.3.2: When the current language result is different from the language result at the previous moment, a frame level detection is performed to obtain the timestamp and language result of the current state;

[0025] Step 6.3.2.1: splicing the audio data of the previous window and the current window head to tail in a time series manner to form spliced ​​audio data;

[0026] Step 6.3.2.2: Use a sliding window to predict the language of the spliced ​​audio data at the frame level, obtain the corresponding language conversion point prediction results by clustering, and correct the language results of the previous window and the current window;

[0027] The present invention discloses a streaming audio language recognition system, which is used to implement the above method. The present invention discloses a streaming audio language recognition system, which includes a VAD module, a language recognition module and a language conversion point detection module.

[0028] The VAD module is used to perform voice activity detection on the voice in the audio data, and then filter out the training data for language recognition, which will be used as the input of the language recognition module;

[0029] The language recognition module is used for language discrimination at the window level, and compares the audio data of the current window with the audio data of the previous window to obtain the timestamp and language result of the current state; which will serve as the input of the language conversion point detection module;

[0030] The language conversion point detection module is used to obtain the timestamp and language result of the current state by correcting the language conversion point when the language of the current window is different from that of the previous window;

[0031] Beneficial effects:

[0032] Compared with the prior art, it has the following beneficial effects:

[0033] 1. The present invention trains a language recognition model to simultaneously realize sentence-level, window-level and frame-level language recognition prediction functions;

[0034] 2. After predicting the frame-level language results, the present invention can accurately locate the language conversion point with the help of the kmeans clustering scheme, thereby achieving the purpose of streaming language recognition prediction.

[0035] 3. Based on traditional language recognition, the present invention creatively proposes a streaming judgment scheme, which expands the application scenarios of language recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of a streaming audio language recognition system provided by an embodiment of the present application;

[0037] Figure 2 It is a schematic diagram of the coding structure of the language recognition conversion point detection module provided in the embodiment of the present application;

[0038] Figure 3 It is a schematic diagram of the decoding structure of the language recognition conversion point detection module provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to better illustrate the purpose and advantages of the present invention, the invention is further described below in conjunction with the accompanying drawings and examples. It should be noted that the implementation of the present invention is not limited to the following embodiments, and any form of modification or change made to the present invention will fall within the protection scope of the present invention.

[0040] Example

[0041] like Figure 1 As shown, a streaming audio language recognition method of this embodiment has the following specific implementation steps:

[0042] Step 1: Use the voice activity detection method to preprocess the original audio data to obtain language recognition training data;

[0043] In the embodiment, the original audio data is subjected to the VAD (Voice Activity Detection) method to remove the silent segments, so as to ensure that each training data human voice is long enough, such as greater than 0.5s;

[0044] Step 2: Extract features from the language recognition training data, and further use Fourier transform and Mel filter bank to obtain language recognition features;

[0045] Step 2.1: Use the Fourier transform method to convert the time domain features into frequency domain features for the language recognition training data;

[0046] Step 2.2: Convert the frequency domain features into language recognition features through Mel filter bank filtering;

[0047] In the embodiment, the language recognition training data is pre-emphasized, framed, and windowed, the frame length is set to 25ms, the frame shift is 10ms, and the window type is selected as the Hamming window; the time domain features are transferred to the frequency domain features using FFT (Fourier transform), and then the energy spectrum is sent to the Mel filter bank to obtain 80-dimensional frequency domain features, and finally the log value is taken to obtain the fbank feature. This method can obtain a lower-dimensional and more compact feature representation, which is more convenient for model training and reasoning.

[0048] Step 3: Build the encoder-decoder model and perform language recognition feature training;

[0049] Step 3.1: Build an encoder-decoder model for language recognition feature training;

[0050] In the embodiment, a language recognition model of an Encoder-Decoder architecture is constructed, wherein the Encoder parameters are migrated from the ASR task, and therefore need to be consistent with the trained ASR model Encoder structure; the Decoder uses a 7-layer Tdnn model to implement language classification and frame-level language information prediction tasks.

[0051] Step 3.2: Use the encoder-decoder model to train language recognition features;

[0052] Step 3.2.1: In the encoder-decoder model, the encoder parameters are initialized with the speech recognition encoder parameters; the decoder parameters are initialized with random variables;

[0053] Step 3.2.2: Train the encoder-decoder model in a loop iterative manner;

[0054] In the embodiment, during training, the encoder parameters are fixed and only the decoder parameters are trained. After the model training converges, all parameters are opened to train the full parameters. Specifically, the encoder-decoder model encoder parameters are initialized with the trained ASR encoder parameters, and the decoder parameters are randomly initialized; the training is divided into two parts. First, the decoder parameters are trained with an initial learning rate of 0.002. After 20 rounds of iteration, the full parameters are trained with an initial learning rate of 0.0001. The total number of rounds is about 30 rounds of training convergence.

[0055] Step 4: Input the language recognition test data into the trained encoder-decoder model to obtain language recognition audio data, and form an audio data stream by accumulating the language recognition audio data;

[0056] In the embodiment, the data receiving module caches the test data stream, and the data is accumulated to a length of 1s before being sent to the downstream for judgment. For scenarios with higher real-time requirements, the window length can be reduced; similarly, for scenarios with higher accuracy requirements, the window length can be increased. In order to balance the real-time performance and accuracy of the system, the window length of the present invention is set to 1s.

[0057] Step 5: Perform activity detection on the audio data stream using a voice activity detection method;

[0058] In the embodiment, VAD is used to judge the human voice, specifically to judge the proportion of human voice in 1s audio. It is considered valid only when the human voice exceeds 0.5s and is sent to the downstream language recognition module for judgment. Otherwise, it is considered to be silent.

[0059] Step 6: Perform window-level language discrimination on the audio data that has passed the activity detection; specifically, compare the audio data of the current window with the audio data of the previous window to obtain the timestamp and language result of the current state;

[0060] Step 6.1: When the current window is the starting window, return the timestamp and language result of the current state;

[0061] Step 6.2: When the previous window is a silent window, the timestamp and language result of the current state are returned;

[0062] In the embodiment, the current window language recognition judgment specifically includes: sufficient audio data of human voice is sent to the same feature extractor as in the training stage to extract 80-dimensional fbank features, and then sent to the language recognition model to obtain the window-level language discrimination result. Specifically, the current state language result is judged, if the current window is the starting window or the previous window is a silent window, the current timestamp and language discrimination result are directly returned;

[0063] Step 6.3: When the previous window is not a silent window and the current window is not a start window, the current language result is compared with the language result at the previous moment to obtain the timestamp and language result of the current state;

[0064] Step 6.3.1: When the current language result is the same as the previous language result, return the timestamp and language result of the current state;

[0065] In the embodiment, when the previous window is not a silent window or the current window is not a start window, the current language result is compared with the language result at the previous moment, and if the results are the same, the current timestamp and language discrimination result are directly returned.

[0066] Step 6.3.2: When the current language result is different from the language result at the previous moment, a frame level detection is performed to obtain the timestamp and language result of the current state;

[0067] Step 6.3.2.1: splicing the audio data of the previous window and the current window head to tail in a time series manner to form spliced ​​audio data;

[0068] Step 6.3.2.2: Use a sliding window to predict the language of the spliced ​​audio data at the frame level, obtain the corresponding language conversion point prediction results by clustering, and correct the language results of the previous window and the current window;

[0069] In the embodiment, Figure 2 and Figure 3As shown in the figure, specifically, first, the audio of the previous window is concatenated with the audio of the current window in the time dimension. Secondly, the features of the input audio are extracted and sent to the language recognition model to calculate the frame-level language discrimination information. The language recognition model adopts the Encoder-Decoder architecture, in which the core of the Encoder module is the Confomer component; the first 5 layers of the Decoder module are Tdnn components, which are used to obtain the frame-level language discrimination information; then, the frame-level features are processed by sliding window averaging in the time dimension, considering that language discrimination needs to contain semantic information, while the frame-level features contain too little semantic information. Therefore, the frame-level features are processed by sliding window averaging in the time dimension, the window length is set to 1s, and the window shift is 0.2s. In this way, 6 compressed frame-level language recognition information can be obtained; finally, the language conversion point prediction result is obtained by kmeans clustering, that is, the 6 language recognition information are clustered by kmeans with the category specified as 2, and the category conversion point position in the clustering result is the corresponding language conversion point prediction result; the language results of the previous window and the current window are corrected, the language from the start point of the previous window to the language conversion point corresponds to the predicted language at the previous moment, and the language conversion point to the end point of the current window corresponds to the predicted language at the current moment.

[0070] The embodiment discloses a streaming audio language recognition system, which is used to implement the above method. The present invention discloses a streaming audio language recognition system, which includes a VAD module, a language recognition module and a language conversion point detection module.

[0071] The VAD module is used to perform voice activity detection on the voice in the audio data, and then filter out the training data for language recognition, which will be used as the input of the language recognition module;

[0072] The language recognition module is used for language discrimination at the window level, and compares the audio data of the current window with the audio data of the previous window to obtain the timestamp and language result of the current state; which will serve as the input of the language conversion point detection module;

[0073] The language conversion point detection module is used to obtain the timestamp and language result of the current state by correcting the language conversion point when the language of the current window is different from that of the previous window;

[0074] The results show that the present invention realizes language recognition in streaming scenarios and can realize multi-language mixed speech recognition when combined with a single-language ASR set.

Claims

1. A method for identifying language of streaming audio, characterized in that: The following steps are included: Step 1: Use the voice activity detection method to preprocess the original audio data to obtain language recognition training data; Step 2: Extract features from the language recognition training data, and further use Fourier transform and Mel filter bank to obtain language recognition features; Step 3: Build the encoder-decoder model and perform language recognition feature training; Step 4: Input the language recognition test data into the trained encoder-decoder model to obtain language recognition audio data, and form an audio data stream by accumulating the language recognition audio data; Step 5: Perform activity detection on the audio data stream using a voice activity detection method; Step 6: Perform window-level language discrimination on the audio data that has passed the activity detection; specifically, compare the audio data of the current window with the audio data of the previous window to obtain the timestamp and language result of the current state.

2. A streaming audio language recognition method as claimed in claim 1, characterized in that: Step 2 is implemented as follows: Step 2.1: Use the Fourier transform method to convert the time domain features into frequency domain features for the language recognition training data; Step 2.2: Convert the frequency domain features into language recognition features through Mel filter bank filtering.

3. The method for identifying language of streaming audio according to claim 1, wherein: Step 3 is implemented as follows: Step 3.1: Build an encoder-decoder model for language recognition feature training; Step 3.2: Use the encoder-decoder model to train language recognition features.

4. A streaming audio language recognition method as claimed in claim 3, characterized in that: Step 3.2 is implemented as follows: Step 3.2.1: In the encoder-decoder model, the encoder parameters are initialized with the speech recognition encoder parameters; the decoder parameters are initialized with random variables; Step 3.2.2: Train the encoder-decoder model in a loop iterative manner.

5. The method for identifying language of streaming audio according to claim 1, wherein: Step 6 is implemented as follows: Step 6.1: When the current window is the starting window, return the timestamp and language result of the current state; Step 6.2: When the previous window is a silent window, the timestamp and language result of the current state are returned; Step 6.3: When the previous window is not a silent window and the current window is not a start window, the current language result is compared with the language result at the previous moment to obtain the timestamp and language result of the current state.

6. A streaming audio language recognition method as claimed in claim 5, characterized in that: Step 6.3 is implemented as follows: Step 6.3.1: When the current language result is the same as the previous language result, return the timestamp and language result of the current state; Step 6.3.2: When the current language result is different from the language result at the previous moment, a frame-level detection is performed to obtain the timestamp and language result of the current state.

7. A streaming audio language recognition method as claimed in claim 6, characterized in that: Step 6.3.2 is implemented as follows: Step 6.3.2.1: splicing the audio data of the previous window and the current window head to tail in a time series manner to form spliced ​​audio data; Step 6.3.2.2: Use a sliding window to perform frame-level language prediction on the spliced ​​audio data, obtain the corresponding language conversion point prediction results through clustering, and correct the language results of the previous window and the current window.

8. A streaming audio language recognition system implementing the method as claimed in claim 1, characterized in that: It includes VAD module, language recognition module and language conversion point detection module; The VAD module is used to perform voice activity detection on the voice in the audio data, and then filter out the training data for language recognition, which will be used as the input of the language recognition module; The language recognition module is used for language discrimination at the window level, and compares the audio data of the current window with the audio data of the previous window to obtain the timestamp and language result of the current state; It will be used as the input of the language transition point detection module; The language conversion point detection module is used to obtain the timestamp and language result of the current state by correcting the language conversion point when the language of the current window is different from that of the previous window.

Images