A method, device, storage medium and terminal device for classifying and identifying mixed languages

Abstract

The present invention discloses a method, apparatus, storage medium and terminal device for classifying and identifying mixed languages. The method for classifying and identifying mixed languages includes the following steps: determining whether each frame of the audio signal to be recognized is a speech frame; extracting Fbank features from all speech frames of each audio; after inputting the Fbank features into a multi-language speech recognition model, extracting the output features of the Conformer layer; using the output features of the Conformer layer as the corresponding BN features for each frame and inputting them into a multi-language classification model to obtain the final language classification prediction result. By extracting Fbank features from the audio signal to be recognized and inputting them into a multi-language speech recognition model to obtain the output features of the Conformer layer, and using the output features of the Conformer layer as BN features and inputting them into a multi-language classification model to predict the language classification, the present invention is not interfered by noise, can quickly predict the language classification, and improves the accuracy of speech evaluation.

Description

Technical Field

[0001] The present invention belongs to the field of language classification and recognition, and particularly relates to a method, device, storage medium and terminal device for classifying and recognizing mixed languages. Background Art

[0002] Currently, the process of a speech evaluation system is as follows: First, preprocessing is performed, then language classification is carried out through a language classifier, and finally evaluation classification is performed through an evaluation classifier.

[0003] Speech evaluation involves various question types, such as paragraph reading, oral translation, and open-ended expression, etc. When the expression ability of non-native speakers is weak, there is often a situation of mixing the mother tongue and the target language. Since the classification method of the language classifier in the evaluation system is usually trained and implemented by single-target language data, this will introduce language noise during the evaluation of the target language. Due to the similarity between phonemes of different languages, it is very difficult for the evaluation model trained by single-language data to identify the language noise, which deteriorates the performance and further leads to inaccurate speech evaluation. Summary of the Invention

[0004] The purpose of the present invention is to overcome the deficiencies of the prior art and provide a method, device, storage medium and terminal device for classifying and recognizing mixed languages, which can quickly and accurately classify and recognize mixed languages for subsequent speech evaluation.

[0005] To achieve the above object, the technical solution adopted by the present invention is: A method for classifying and recognizing mixed languages, comprising the following steps:

[0006] Judge whether each frame of the audio signal to be recognized is a speech frame;

[0007] Extract Fbank features for all speech frames of each audio;

[0008] After inputting the Fbank features into a multi-language speech recognition model, extract the output features of the Conformer layer;

[0009] Use the output features of the Conformer layer as the corresponding BN features for each frame and input them into a multi-language classification model to obtain the final language classification prediction result.

[0010] Further, the multi-language speech recognition model is obtained based on the following training method:

[0011] Obtain training samples, where the training samples are audio and their corresponding reference texts;

[0012] Extract Fbank features for all speech frames of each audio;

[0013] Input the Fbank features into the Conformer structure to obtain BN features;

[0014] Use the fully connected layer and CTC to update and optimize the parameters of the multilingual speech recognition model.

[0015] Furthermore, the multilingual classification model is obtained based on the following training method:

[0016] Obtain training samples, where the training samples are audio and their corresponding language category labels;

[0017] Extract Fbank features for all speech frames of each audio;

[0018] Input the Fbank features into the Conformer structure to obtain BN features;

[0019] Input the BN features of T*N into 3 consecutive TDNN blocks. Each TDNN block outputs features of T*M. Concatenate the features of 3 T*M to get features of T*(3*M), and then pass through the FC layer, RELU layer and BatchNorm layer to output the features F of T*M, where T is the number of frames and N is the input feature dimension;

[0020] Input the features F of T*M into Pooling, calculate the mean and variance of T frames, and concatenate the mean and variance to get features of 1*(2*M);

[0021] Input the features of 1*(2*M) into the FC layer and softmax layer to obtain a language category prediction vector of 1×C, where C is the number of language categories;

[0022] Calculate the loss of CE Loss in combination with the language category labels, and update the parameters of the multilingual classification model.

[0023] Furthermore,

[0024] If it is known that the test audio belongs to only one language:

[0025] Then input the BN features of the entire audio into the language classification model, output a language category prediction vector of 1×C, and take the language category corresponding to the vector with the highest probability as the final prediction result;

[0026] If it is known that the test audio contains multiple languages:

[0027] Take the window length as W frames and the window shift as S frames. Each time, take W-frame BN features and input them into the language classification model, output a language category prediction vector of 1×C, and take the language category corresponding to the vector with the highest probability as the prediction result of the window speech frames;

[0028] Assume that the languages of two adjacent windows are A and B respectively. If A and B are the same, all frames of these two windows belong to language A; if A and B are different, the frames in the overlapping part of the two windows are evenly distributed to the two languages.

[0029] A mixed language classification and recognition device, comprising:

[0030] A judgment module, used to judge whether each frame of the audio signal to be recognized is a speech frame;

[0031] An extraction module, used to extract Fbank features from all speech frames of each audio;

[0032] An output module, used to input the Fbank features into a multi-language speech recognition model and then extract the output features of the Conformer layer;

[0033] A prediction module, used to input the output features of the Conformer layer as the corresponding BN features of each frame into a multi-language classification model to obtain the final language classification prediction result.

[0034] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned mixed language classification and recognition method is implemented.

[0035] A terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the mixed language classification and recognition method is implemented.

[0036] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:

[0037] A mixed language classification and recognition method, device, storage medium and terminal device according to the solution of the present invention can quickly and effectively predict the language classification by inputting the Fbank features extracted from the audio to be recognized into a multi-language speech recognition model and then extracting the output features of the Conformer layer, and inputting the output features of the Conformer layer as BN features into a multi-language classification model. It will not be interfered by other noises, so as to quickly and accurately obtain the language classification, which is convenient for subsequent speech evaluation. Description of the Drawings

[0038] The technical solution of the present invention will be further described below in conjunction with the drawings:

[0039] Figure 1 It is a schematic flowchart of the mixed language classification and recognition method according to an embodiment of the present invention;

[0040] Figure 2Schematic flow diagram of a multi - language speech recognition model training method in an embodiment of the present invention;

[0041] Figure 3 Schematic flow diagram of a multi - language classification model training method in an embodiment of the present invention. Detailed implementation manners

[0042] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] Refer to Figure 1 A method for classifying and recognizing mixed languages according to an embodiment of the present invention includes the following steps:

[0044] Step S10: Determine whether each frame of the audio signal to be recognized is a speech frame;

[0045] Step S20: Extract Fbank features from all speech frames of each audio;

[0046] Step S30: After inputting the Fbank features into the multi - language speech recognition model, extract the output features of the Conformer layer;

[0047] Step S40: Use the output features of the Conformer layer as the corresponding BN features for each frame and input them into the multi - language classification model to obtain the final language classification prediction result.

[0048] Refer to Figure 2 Specifically, the multi - language speech recognition model is used to extract the output features of the Conformer layer and then input them into the multi - language classification model. The multi - language speech recognition model in this embodiment is obtained based on the following training method: Obtain training samples, where the training samples are audio and their corresponding reference texts; The model structure uses an end - to - end framework and adopts the Conformer (Convolution - augmented transformer) structure. Combine the fully - connected layer and CTC to update and optimize the parameters of the multi - language speech recognition model. The input of the model is the Fbank features obtained from the training samples, and the output is a set of sub - word units in multiple languages.

[0049] For each language, train a BPE (Byte Pair Encoding) model using its respective text corpus to obtain the sub - word units of each language; Combine the sub - word units of multiple languages as the set of model output units.

[0050] Refer to Figure 3, the multi - language classification model is obtained based on the following training method: Obtain training samples, where the training samples are audio and their corresponding language category labels, and one audio in the training samples corresponds to only one language category; Extract Fbank features for all speech frames of each audio; After inputting the Fbank features into the multi - language speech recognition model, extract the output features of the Conformer layer.

[0051] Take the output features of the Conformer layer as the BN features corresponding to each frame. The TDNN block is a structure including a TDNN layer, a RELU activation layer, and a BatchNorm layer; Input the T * N BN features into 3 consecutive TDNN blocks. Each TDNN block outputs T * M features. Concatenate the 3 T * M features to obtain T * (3 * M) features, and then pass through an FC layer, a RELU layer, and a BatchNorm layer to output the features F of T * M, where T is the number of frames and N is the input feature dimension.

[0052] Input the features F of T * M into Pooling, calculate the mean and variance of the T frames, and concatenate the mean and variance to obtain features of 1 * (2 * M).

[0053] Input the features of 1 * (2 * M) into an FC layer and a softmax layer to obtain a language category prediction vector of 1 × C, where C is the number of language categories.

[0054] Calculate the loss of CE Loss in combination with the language category labels, and update the parameters of the multi - language classification model.

[0055] In addition, in step 40, take the output features of the Conformer layer as the BN features corresponding to each frame and input them into the multi - language classification model to obtain the final language classification prediction result. The specific method for obtaining the final language classification prediction result is as follows:

[0056] If it is known that the test audio belongs to only one language: Then input the BN features of the entire audio into the language classification model, output a language category prediction vector of 1 × C, and take the language category corresponding to the vector with the highest probability as the final prediction result.

[0057] If it is known that the test audio contains multiple languages: Then take a window length of W frames and a window shift of S frames. Each time, take W - frame BN features and input them into the language classification model, output a language category prediction vector of 1 × C, and take the language category corresponding to the vector with the highest probability as the prediction result of the speech frames of this window; Assume that the languages of two adjacent windows are A and B respectively. If A and B are the same, then all frames of these two windows belong to language A; If A and B are different, then evenly distribute the frames in the overlapping part of the two windows to the two languages.

[0058] In the present invention, by inputting the audio to be recognized into a multi - language speech recognition model to obtain the output features of the Conformer layer, and using the output features of the Conformer layer as the BN features corresponding to each frame and inputting them into the multi - language classification model, the classification of the language can be predicted quickly and effectively without being interfered by other noises, so as to obtain the language classification quickly and accurately, which is convenient for subsequent speech evaluation.

[0059] Based on any of the above - mentioned embodiments, the present invention provides a hybrid language classification and recognition device, including:

[0060] A judgment module, configured to judge whether each frame of the audio signal to be recognized is a speech frame;

[0061] An extraction module, configured to extract Fbank features for all speech frames of each audio;

[0062] An output module, configured to extract the output features of the Conformer layer after inputting the Fbank features into the multi - language speech recognition model;

[0063] A prediction module, configured to use the output features of the Conformer layer as the BN features corresponding to each frame and input them into the multi - language classification model to obtain the final language classification prediction result.

[0064] On the other hand, the present invention also provides a computer program product. The computer program product includes a computer program stored on a non - transitory computer - readable storage medium. The computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the language recognition method provided by each of the above - mentioned methods. The method includes: judging whether each frame of the audio signal to be recognized is a speech frame; extracting Fbank features for all speech frames of each audio; extracting the output features of the Conformer layer after inputting the Fbank features into the multi - language speech recognition model; using the output features of the Conformer layer as the BN features corresponding to each frame and inputting them into the multi - language classification model to obtain the final language classification prediction result.

[0065] On yet another aspect, the present invention also provides a non - transitory computer - readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the language recognition method provided by each of the above - mentioned methods. The method includes: judging whether each frame of the audio signal to be recognized is a speech frame; extracting Fbank features for all speech frames of each audio; extracting the output features of the Conformer layer after inputting the Fbank features into the multi - language speech recognition model; using the output features of the Conformer layer as the BN features corresponding to each frame and inputting them into the multi - language classification model to obtain the final language classification prediction result.

[0066] The above are only specific application examples of the present invention and do not constitute any limitation to the protection scope of the present invention. Any technical solutions formed by equivalent transformation or equivalent substitution fall within the scope of the present invention's protection rights.

Claims

1. A method for classifying and identifying mixed languages, characterized in that, It includes the following steps: Determine whether each frame of the audio signal to be recognized is a speech frame; Extract Fbank features for all speech frames of each audio; After inputting the Fbank features into the multi-language speech recognition model, extract the output features of the Conformer layer; Use the output features of the Conformer layer as the corresponding BN features for each frame and input them into the multi-language classification model to obtain the final language classification prediction result; Among them, the multi-language classification model is obtained based on the following training method: Obtain training samples, where the training samples are audio and their corresponding language category labels; Extract Fbank features for all speech frames of each audio; After inputting the Fbank features into the multi-language speech recognition model, extract the output features of the Conformer layer; Use the output features of the Conformer layer as the corresponding BN features for each frame; Input the BN features of T*N into 3 consecutive TDNN blocks. Each TDNN block outputs features of T*M. Concatenate the features of the 3 T*M to obtain features of T*(3*M), and then pass through the FC layer, RELU layer and BatchNorm layer to output features of T*M, where T is the number of frames and N is the input feature dimension; Input the features F of T*M into the Pooling, calculate the mean and variance of T frames, and concatenate the mean and variance to obtain features of 1*(2*M); Input the features of 1*(2*M) into the FC layer and softmax layer to obtain a language category prediction vector of 1×C, where C is the number of language categories; Calculate the loss of CE Loss in combination with the language category labels and update the parameters of the multi-language classification model.

2. The hybrid language classification and recognition method according to claim 1, characterized in that, The multi-language speech recognition model is obtained based on the following training method: Obtain training samples, where the training samples are audio and their corresponding reference texts; Extract Fbank features for all speech frames of each audio; Input the Fbank features into the Conformer structure to obtain BN features; Use the fully connected layer and CTC to update and optimize the parameters of the multi-language speech recognition model.

3. The method for hybrid language classification and recognition according to claim 1, characterized in that If it is known that the test audio belongs to only one language: Then input the BN features of the whole audio into the language classification model, output a language category prediction vector of 1×C, and use the language category corresponding to the vector with the highest probability as the final prediction result; If it is known that the test audio contains multiple languages: Take a window length of W frames and a window shift of S frames. Each time, take W-frame BN features and input them into the language classification model, output a language category prediction vector of 1×C, and use the language category corresponding to the vector with the highest probability as the prediction result of the speech frames in this window; Suppose the languages of two adjacent windows are A and B respectively. If A and B are the same, then all frames of these two windows belong to language A; if A and B are different, then evenly distribute the frames in the overlapping part of the two windows to the two languages.

4. A hybrid language classification and recognition device, characterized in that It includes: A judgment module for judging whether each frame of the audio signal to be recognized is a speech frame; An extraction module for extracting Fbank features for all speech frames of each audio; An output module, configured to input Fbank features into a multi-lingual speech recognition model, and then extract the output features of the Conformer layer; A prediction module, configured to use the output features of the Conformer layer as the BN features corresponding to each frame and input them into a multi-lingual classification model to obtain the final language classification prediction result.

5. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the mixed language classification and recognition method according to any one of claims 1 to 4.

6. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the mixed language classification and recognition method according to any one of claims 1 to 3.

Images