A speech recognition method and a speech recognition apparatus

Abstract

The embodiment of the present application provides a speech recognition method and a speech recognition device, which can adjust the recognition result output by the acoustic model based on the added language model, and improve the recognition accuracy of the speech recognition model. The speech recognition method comprises the following steps: obtaining historical audio data and corresponding historical audio text data; preprocessing the historical audio text data and extracting vocabulary data in the historical audio text data; determining the phoneme annotation of the vocabulary data based on the pre-stored Chinese dictionary and English dictionary and G2P model, and constructing a vocabulary dictionary; training the acoustic model and the language model based on the vocabulary data of the historical audio data and the historical audio text data; generating a speech recognition model according to the trained acoustic model, the trained language model and the vocabulary dictionary; obtaining to-be-recognized speech data, inputting the to-be-recognized speech data into the speech recognition model, and obtaining text information of the to-be-recognized speech data.

Description

[Technical Field]

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method and a speech recognition device. [Background Technology]

[0002] Currently, with economic development and technological progress, people's lives are becoming increasingly globalized, and the use of mixed languages ​​for communication has become a common phenomenon.

[0003] In existing technologies, speech recognition models for recognizing mixed Chinese and English languages ​​are usually trained by combining general dictionaries with acoustic models. Although speech recognition models trained solely on acoustic models can correctly annotate the input speech, the accuracy of text recognition is low when identifying the corresponding text through phonemes. For example, when a certain phoneme can correspond to multiple words, the recognized text is often not very accurate. [Summary of the Invention]

[0004] This application provides a speech recognition method and a speech recognition device, which can adjust the recognition results output by the acoustic model based on the added language model, thereby improving the recognition accuracy of the speech recognition model.

[0005] In a first aspect, this application provides a speech recognition method, the method comprising:

[0006] Retrieve historical audio data and corresponding historical audio text data;

[0007] The historical audio text data is preprocessed, and vocabulary data is extracted from the historical audio text data;

[0008] Based on pre-stored Chinese and English dictionaries and a G2P model, the phoneme annotations of the vocabulary data are determined, and a vocabulary dictionary is constructed. The G2P model is trained based on the historical audio data.

[0009] Based on the vocabulary data of the historical audio data and the historical audio text data, an acoustic model and a language model are trained, and a speech recognition model is generated based on the trained acoustic model, the trained language model and the vocabulary dictionary.

[0010] Acquire the speech data to be recognized, input the speech data to be recognized into the speech recognition model, and obtain the text information of the speech data to be recognized.

[0011] In this embodiment, historical audio data and corresponding historical audio text data are acquired. The historical audio text data is preprocessed and words are extracted and segmented. Based on pre-stored Chinese and English dictionaries and a G2P model, the segmented words are phoneme-annotated. The historical audio data is input into an acoustic model for training, and the corresponding audio text data is input into a language model for training. The acoustic model can learn the pronunciation features in the historical audio data, and the language model can learn the text features of the audio text data, thereby training a high-performance speech recognition model. The speech recognition model is then used to recognize the speech data to be recognized, obtaining the text information of the speech data to be recognized.

[0012] Optionally, preprocessing the audio text data includes:

[0013] The audio text data is subjected to text regularization processing.

[0014] In this embodiment of the application, it is necessary to perform regularization processing on the audio text data, that is, to change the format of the original audio data so that the text can be divided into individual words in a standardized format, which is convenient for subsequent training and learning.

[0015] Optionally, the vocabulary data extracted from the audio text data includes:

[0016] The historical audio text data is segmented using a pre-built word segmentation tool. Based on the segmented historical audio text data, vocabulary data is extracted from the historical audio text data to form a vocabulary list containing all vocabulary data of the historical audio text data.

[0017] In this embodiment, historical audio text data is segmented based on a pre-built word segmentation tool, and vocabulary data is extracted based on the segmented historical audio text data to obtain a vocabulary list containing all vocabulary data of the historical audio text data. Segmentation based on a pre-built word segmentation tool can improve the accuracy of word segmentation and avoid word segmentation errors.

[0018] Optionally, based on pre-stored Chinese and English dictionaries and a G2P model, the phoneme annotations of the vocabulary data are determined as follows:

[0019] Phoneme annotation is performed on Chinese words in the vocabulary list based on the Chinese dictionary and the Chinese audio data in the historical audio data, and phoneme annotation is performed on English words in the vocabulary list based on the English dictionary and the English audio data in the historical audio data;

[0020] If the phoneme annotation of an English word in the vocabulary list fails, the English word with the failed annotation is input into the G2P model to obtain the pronunciation sequence corresponding to the English word with the failed annotation.

[0021] Based on the pronunciation sequence, the phonemes of the English words that failed to be labeled were re-labeled.

[0022] In this embodiment, the words in the segmented vocabulary list are phoneme-annotated based on pre-stored Chinese and English dictionaries. For English words that the dictionary fails to annotate, the G2P model is used to annotate them to ensure that the words in the audio text data used for training are correctly annotated and brought into the training, so that a high-performance speech recognition model can be trained in the future.

[0023] Optionally, the method for constructing the vocabulary dictionary includes:

[0024] A vocabulary dictionary is generated based on the phoneme annotations of the vocabulary data in the historical audio text data and the pre-stored Chinese and English dictionaries.

[0025] In this embodiment, the vocabulary dictionary consists of vocabulary from historical audio text data and pre-stored dictionaries. That is, the vocabulary dictionary will vary depending on the historical audio text data, making the vocabulary dictionary more in line with the current user's usage characteristics and improving the accuracy of phoneme annotation.

[0026] Optionally, the training method for the G2P model includes:

[0027] Acquire the historical audio data and preprocess the historical audio data;

[0028] Extract the feature vector from the preprocessed historical audio data;

[0029] The extracted feature vectors are input into the acoustic model to obtain the vocabulary corresponding to the historical audio data;

[0030] The G2P model is trained using the vocabulary corresponding to the historical audio data to obtain the trained G2P model.

[0031] In this embodiment, historical audio data is input into an acoustic model to obtain a corresponding vocabulary, and the obtained vocabulary is used to train a G2P model, so that the G2P model better meets the usage characteristics of the current user, thereby obtaining a G2P model with higher phoneme annotation accuracy.

[0032] Optionally, the method for generating the speech recognition model includes:

[0033] An acoustic model is trained based on the historical audio data, a language model is constructed, and the language model is trained based on the lexical data of the historical audio text data.

[0034] A decoder is generated based on the trained language model and the vocabulary dictionary;

[0035] A speech recognition model is generated based on the decoder and the trained speech model.

[0036] In this embodiment, an acoustic model is trained based on historical audio data, and a language model is trained based on historical audio text data. The trained language model and a vocabulary dictionary form a decoder, which is used to annotate the text. The decoder and the trained acoustic model form a speech recognition model, thereby correctly translating the input speech into text and annotating the text phonemes.

[0037] Optionally, after generating the speech recognition model, the method further includes:

[0038] Acquire test audio data and corresponding text data, and extract test vocabulary data from the test audio text data;

[0039] The feature vectors of the test vocabulary data are extracted and input into the decoder of the speech recognition model to obtain the test pronunciation phoneme sequence;

[0040] If the test pronunciation phoneme sequence contains phonemes or words whose pronunciations are inconsistent with those in the test audio data, the speech recognition model is corrected.

[0041] In this embodiment of the application, after generating the speech recognition model, a test will be conducted based on the test audio data and the corresponding text data. When the phoneme sequence annotated for the test text data is inconsistent with the pronunciation of the audio data, the speech model is considered to have low recognition accuracy, and the speech recognition model will be corrected to improve the recognition ability of the speech recognition model.

[0042] Optionally, modifying the speech recognition model includes:

[0043] If there are phonemes in the test pronunciation phoneme sequence that are inconsistent with the word pronunciation in the test audio data, then the correct pronunciation phoneme sequence is obtained and written into the vocabulary dictionary, and the speech recognition model is updated according to the new vocabulary dictionary;

[0044] If there are words in the test pronunciation phoneme sequence whose pronunciation is inconsistent with the words in the test audio data, then the correct pronunciation phoneme sequence and the corresponding words are obtained and written into the vocabulary dictionary; the language model is retrained based on the vocabulary data corresponding to the correct pronunciation phoneme sequence, and the speech recognition model is updated according to the retrained speech model.

[0045] In this embodiment, when there are phonemes with inconsistent pronunciations in the test audio data, the correct phoneme sequence is written into the vocabulary dictionary, and the speech recognition model is updated according to the new vocabulary dictionary; when there are words with inconsistent pronunciations in the test audio data, the correct phoneme sequence and the corresponding words are written into the vocabulary dictionary, and the language model is retrained based on the vocabulary data corresponding to the correct phoneme sequence. A new speech recognition model is constructed based on the retrained language model, thereby obtaining a high-performance speech recognition model.

[0046] Optionally, inputting the speech data to be recognized into the speech recognition model to obtain the text information of the speech data to be recognized includes:

[0047] Extract the feature vector of the speech data to be recognized and input it into the acoustic model to generate the phoneme sequence corresponding to the speech data to be recognized;

[0048] The phoneme sequence is decoded by a decoder consisting of a language model and a vocabulary dictionary to obtain the text information of the speech data to be recognized.

[0049] In this embodiment, after the speech data to be recognized is input into the speech recognition model, the feature vector of the speech data to be recognized is first extracted and input into the acoustic model to generate the phoneme sequence corresponding to the speech data to be recognized. Then, the phoneme sequence is decoded by a decoder composed of a language model and a vocabulary dictionary to obtain the text information of the speech data to be recognized. Since the speech recognition model is trained based on historical audio data and corresponding text data, it can effectively recognize audio data and thus obtain relatively accurate text information.

[0050] Secondly, this application provides a voice recognition device, the device comprising:

[0051] The communicator is used to receive historical audio data and corresponding historical audio text data, as well as the speech data to be recognized.

[0052] The processor, coupled to the communicator, is configured to: preprocess the historical audio-text data and extract vocabulary data from the historical audio-text data; determine the phoneme annotations of the vocabulary data based on pre-stored Chinese and English dictionaries and a G2P model, and construct a vocabulary dictionary; train an acoustic model and a language model based on the historical audio data and the vocabulary data of the historical audio-text data, and generate a speech recognition model based on the trained acoustic model, the trained language model, and the vocabulary dictionary; acquire speech data to be recognized, input the speech data to be recognized into the speech recognition model, and obtain the text information of the speech data to be recognized. [Attached Image Description]

[0053] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This application provides a flowchart illustrating a speech recognition method.

[0055] Figure 2 This application provides a flowchart illustrating a G2P model training method.

[0056] Figure 3 A schematic flowchart illustrating a phoneme annotation method provided in an embodiment of this application;

[0057] Figure 4 A schematic diagram illustrating the training process of a speech recognition model provided in an embodiment of this application;

[0058] Figure 5 This application provides a schematic diagram of the training process for an acoustic model.

[0059] Figure 6 A schematic diagram of a language model training process provided in an embodiment of this application;

[0060] Figure 7 This application provides a schematic diagram of a process for obtaining text information of speech data to be recognized, as an embodiment of the present application.

[0061] Figure 8 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application.

Detailed Implementation Methods

[0062] To better understand the technical solutions in this specification, the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0063] It should be understood that the described embodiments are merely some, not all, of the embodiments in this specification. All other embodiments obtained by those skilled in the art based on the embodiments in this specification without inventive effort are within the scope of protection of this specification.

[0064] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this specification. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0065] With economic development and technological progress, people's lives are becoming increasingly globalized, and the use of mixed languages ​​for communication has become a common phenomenon.

[0066] Research has found that in related technologies, speech recognition models used to recognize mixed Chinese and English languages ​​are usually trained by combining general dictionaries with acoustic models. Although speech recognition models trained solely on acoustic models can correctly annotate the input speech, the accuracy of text recognition is low when identifying the corresponding text through phonemes. For example, when a certain phoneme can correspond to multiple words, the recognized text is often not very accurate.

[0067] In view of this, embodiments of this application provide a speech recognition method. This method involves acquiring historical audio data and corresponding historical audio text data; preprocessing the historical audio text data and extracting vocabulary; and annotating the extracted vocabulary with phonemes based on pre-stored Chinese and English dictionaries and a G2P model. The method trains a speech recognition model by inputting historical audio data into an acoustic model and corresponding audio text data into a language model. The acoustic model learns the pronunciation features of the historical audio data, and the language model learns the text features of the audio text data, thereby training a high-performance speech recognition model. The speech recognition model is then used to recognize the speech data to be recognized, obtaining the text information of the speech data to be recognized.

[0068] The technical solutions provided in the embodiments of this application will be described below with reference to the accompanying drawings. Please refer to... Figure 1 This application provides a speech recognition method, the process of which is described as follows:

[0069] Step 101: Obtain historical audio data and corresponding historical audio text data.

[0070] In this embodiment of the application, a large amount of collected historical audio data and corresponding historical audio text data can be obtained. The historical audio data may include Chinese audio data, English audio data, mixed Chinese and English audio data, and audio data of specific proprietary words.

[0071] For example, historical audio data can be "The weather is very nice today" or "Please wait for me," or it can be mixed Chinese and English audio data in a specific technical field (manufacturing, service industry, etc.). There are no special restrictions here.

[0072] Step 102: Preprocess the historical audio text data and extract the vocabulary data from the historical audio text data to obtain a vocabulary list containing all vocabulary data of the historical audio text.

[0073] In some embodiments, considering that the format of the historical audio text data used for training may be messy or the layout may be scattered, directly extracting words may result in word extraction errors that are difficult to label correctly.

[0074] Therefore, in this embodiment, the historical audio text data can be regularized first, and then words can be extracted to improve the accuracy of word extraction. The specific details of the text regularization process are described below.

[0075] Processing Mechanism 1: In this embodiment of the application, duplicate sentences in historical audio text data can be deduplicated to avoid multiple annotations of the same text data.

[0076] Processing Mechanism 2: In this embodiment of the application, the case of English words contained in the historical audio text data can be uniformly processed, that is, the case of English words contained in the historical audio text data can be uniformly modified to uppercase or lowercase, so as to facilitate the correct identification and extraction of words.

[0077] Processing Mechanism 3: In this embodiment of the application, garbled characters with input errors in historical audio text data can be deleted to facilitate the correct identification and extraction of words.

[0078] Processing Mechanism 4: In this embodiment of the application, Chinese conversion processing can be performed on numbers, special characters, and combinations of numbers and special characters in historical audio text data. For example, number sequences, dates, and units of measurement in historical audio text data can be converted into Chinese according to their pronunciation, which facilitates the correct identification and extraction of words.

[0079] Processing Mechanism 5: In this embodiment of the application, English words with spelling errors in historical audio text data can be corrected to facilitate the correct identification and extraction of words.

[0080] Processing Mechanism Six: In this embodiment of the application, punctuation marks contained in historical audio text data can be converted according to their meanings. For meaningful punctuation marks, they are converted into Chinese characters according to their pronunciation. For meaningless punctuation marks, they are mapped to blanks to facilitate the correct identification and extraction of words.

[0081] The appropriate combination of at least one or more of the above six mechanisms should be selected based on the actual format of the historical audio text data.

[0082] For example, if the historical audio text data is "The Weather Is so nice today", the above-mentioned processing mechanisms two and five can be used to process the audio data into "the weather is so nice today", which facilitates the correct recognition and extraction of words.

[0083] Considering that word segmentation tools may make mistakes when extracting words, such as segmenting a long word into multiple shorter words, resulting in incorrect word extraction, this application embodiment prioritizes storing potentially problematic proprietary words in the word segmentation tool to avoid extraction errors and improve the accuracy of word segmentation and extraction.

[0084] As one possible implementation, the preprocessed historical audio text data can be segmented using a pre-built word segmentation tool, and the vocabulary data of the segmented historical audio text data can be extracted to obtain a vocabulary list containing all vocabulary data of the historical audio text. The pre-built word segmentation tool should at least include custom-defined professional vocabulary for the target domain, which can improve the accuracy of vocabulary extraction.

[0085] For example, the jieba word segmentation tool can be used to segment Chinese words in historical audio text data and extract Chinese vocabulary data, while the NLTK word segmentation tool can be used to segment English words in historical audio text data and extract English vocabulary data. Based on the Chinese and English vocabulary data, a vocabulary list is obtained. The word segmentation tool used in this application embodiment is not particularly limited here.

[0086] Step 103: Based on pre-stored Chinese and English dictionaries and the G2P model, determine the phoneme annotations of the vocabulary data and construct a vocabulary dictionary. The G2P model is trained based on historical audio data.

[0087] In existing technologies, phoneme annotation methods for vocabulary data are based solely on general dictionaries and corresponding audio text data. When there are abbreviations or user-defined words, phoneme annotation may not be performed correctly, which is not conducive to subsequent model training.

[0088] Therefore, in this embodiment, a G2P model can be incorporated into the phoneme annotation method. This G2P model can convert English words not found in Chinese or English dictionaries into their corresponding phonemes. For example, words containing polyphonic characters, abbreviations, or words with Chinese accents are converted into corresponding phonemes by the G2P model to generate their pronunciations, ensuring that all words in the audio data used for training are accurately annotated with phonemes.

[0089] Figure 2This is a flowchart illustrating a G2P model training method provided in an embodiment of this application. Please refer to [link / reference]. Figure 2 In one embodiment, the training method for the G2P model includes:

[0090] Step 201: Obtain historical audio data, which includes Chinese audio data, English audio data, mixed Chinese and English audio data, and audio data of specific proper nouns.

[0091] Step 202: Preprocess and extract features from historical audio data.

[0092] In this embodiment of the application, the preprocessing of historical audio data includes one or more combinations of format conversion, enhancement processing, noise reduction, and de-reverberation.

[0093] In this embodiment of the application, feature extraction is performed on the preprocessed historical audio data to obtain a feature vector, which includes time-domain features, frequency-domain features and acoustic features.

[0094] Step 203: Input the extracted feature vectors into the acoustic model to obtain the vocabulary corresponding to the historical audio data.

[0095] Step 204: Train the G2P model using the vocabulary corresponding to the historical audio data.

[0096] As one possible implementation, for the extracted vocabulary data, the Chinese words in the vocabulary list are first phoneme-annotated based on the pre-stored Chinese dictionary and the Chinese audio data in the historical audio data, and the English words in the vocabulary list are phoneme-annotated based on the pre-stored English dictionary and the English audio data in the historical audio data. When the phoneme annotation of an English word fails, the English word with the failed annotation needs to be input into the G2P model to obtain the correct pronunciation sequence corresponding to the English word with the failed annotation. Based on the pronunciation sequence, the English word with the failed annotation is re-phoneme-annotated.

[0097] For example, the pre-stored Chinese dictionary could be the aishell phonetic dictionary, and the pre-stored English dictionary could be the cmudict dictionary; no particular restrictions are imposed here.

[0098] Figure 3 This is a schematic diagram of the phoneme annotation process provided in an embodiment of this application. In one embodiment, please refer to... Figure 3 The phoneme annotations for the vocabulary data include:

[0099] Step 301: Phoneme annotation of Chinese words in the vocabulary list based on the pre-stored Chinese dictionary and Chinese audio data in historical audio data.

[0100] Step 302: Phoneme-annotate the English words in the vocabulary list based on the pre-stored English dictionary and the English audio data in the historical audio data.

[0101] In this embodiment, a vocabulary dictionary is constructed based on pre-stored Chinese dictionaries, English dictionaries, and phoneme annotations of vocabulary data. The constructed vocabulary dictionary includes a dictionary of pronunciations for all words, which is used to train acoustic and speech models. The dictionary can be added to and modified according to actual pronunciations to improve the English pronunciation recognition of native Chinese speakers. As a supplement to the annotation dictionary, it makes the phoneme annotations most effective.

[0102] Step 303: Obtain the remaining English words in the vocabulary list and input them into the G2P model to obtain the pronunciation sequence corresponding to the remaining English words. Then, perform phoneme annotation on the remaining English words based on the pronunciation sequence.

[0103] In this embodiment, Chinese dictionaries, English dictionaries, and G2P models are used to annotate the phonemes of vocabulary data and to supplement the pronunciation of words outside the existing dictionaries. This effectively reduces the confusion of similar pronunciations when a single-language dictionary is mapped to other languages, preserves reasonable pronunciations, and greatly reduces the workload of manual annotation.

[0104] Step 104: Based on the vocabulary data of historical audio data and historical audio text data, train the acoustic model and language model. Then, generate a speech recognition model based on the trained acoustic model, language model, and vocabulary dictionary. The acoustic model is used to identify the corresponding phonemes based on the audio data; the language model is used to identify the corresponding text data based on the phonemes.

[0105] In this embodiment, the speech recognition model is composed of an acoustic model and a language model. The acoustic model is mainly used to annotate the input audio data with phonemes and output the corresponding text data based on the annotation results. The language model continuously adjusts the text data output by the acoustic model to be closer to the actual text of the audio data. Therefore, the speech recognition model trained based on the acoustic model and the language model has high recognition accuracy.

[0106] The following section details how to obtain a speech recognition model based on historical audio data, lexical data from historical audio text data, and a vocabulary dictionary. Please refer to [link / reference needed]. Figure 4 The training process for the speech recognition model is described below:

[0107] Step 401: Train the acoustic model based on historical audio data.

[0108] Please see Figure 5 In some embodiments, the training process of the acoustic model is described as follows:

[0109] Step 501: Obtain historical audio data, which includes Chinese audio data, English audio data, mixed Chinese and English audio data, and audio data of specific proper nouns.

[0110] Step 502: Enhance the historical audio data.

[0111] In this embodiment of the application, historical audio data is enhanced to increase the amount of historical audio data used for training.

[0112] Step 503: Extract features from the enhanced historical audio data to obtain feature vectors, including time-domain features, frequency-domain features, and acoustic features.

[0113] Step 504: Use the extracted feature vectors to train the acoustic model and obtain the trained acoustic model.

[0114] In this embodiment, an acoustic model is constructed based on a Convolutional Neural Network (CNN) and a Time-Temporal Neural Network for Factorization (TDNN-F). CNNs excel at extracting local features, performing spatial abstraction and generalization, and are capable of extracting more expressive high-dimensional features in the spatial dimension. In the Time-Temporal Neural Network for Factorization (TDNN-F), each layer has a different temporal resolution, and the output of the previous layer is concatenated and used as the input of the current layer.

[0115] The specific process of training an acoustic model using feature vectors from audio data is obvious to those skilled in the art and will not be elaborated here.

[0116] Step 402: Train a language model based on lexical data from historical audio text data.

[0117] Please see Figure 6 In some embodiments, the training process for training a language model based on lexical data from historical audio text data is described as follows:

[0118] Step 601: Construct an N-gram language model.

[0119] In this embodiment of the application, the vocabulary data of historical audio text data is weighted to obtain an N-gram language model.

[0120] Step 602: Train the language model using the vocabulary data of the historical audio text data to obtain the trained language model.

[0121] The specific process of training a language model using vocabulary data from audio text data is obvious to those skilled in the art and will not be elaborated here.

[0122] Step 403: Generate a decoder based on the trained language model and vocabulary dictionary.

[0123] In this embodiment, the trained language model and vocabulary dictionary are combined to obtain the HCLG decoder.

[0124] Step 404: Generate a speech recognition model based on the decoder and the trained acoustic model.

[0125] In this embodiment, the decoder and the trained speech model are compiled to obtain a speech recognition model.

[0126] Considering that even after the speech recognition model is determined through the above training method, inaccuracies may still occur, this embodiment of the application selects test audio data to test the trained speech recognition model, and updates and adjusts the speech recognition model accordingly based on the test results.

[0127] As one possible implementation, the method further includes:

[0128] Acquire test audio data and corresponding text data, and extract test vocabulary data from the test audio and text data; extract feature vectors from the test vocabulary data and input them into the decoder of the speech recognition model to obtain a test pronunciation phoneme sequence; if there are phonemes or words in the test pronunciation phoneme sequence that are inconsistent with the pronunciation of the words in the test audio data, correct the speech recognition model.

[0129] In some embodiments, if there are phonemes in the test pronunciation phoneme sequence that are inconsistent with the pronunciation of words in the test audio data, the correct pronunciation phoneme sequence is obtained and written into the vocabulary dictionary, and the speech recognition model is updated according to the new vocabulary dictionary.

[0130] In some embodiments, if there are words in the test pronunciation phoneme sequence that are inconsistent with the pronunciation of words in the test audio data, the correct pronunciation phoneme sequence and the corresponding words are obtained and written into the vocabulary dictionary; the language model is retrained based on the vocabulary data corresponding to the correct pronunciation phoneme sequence, and the speech recognition model is updated according to the retrained speech model.

[0131] By using the methods described above, the language model can be updated to obtain a speech recognition model that includes the new phonetic symbols, which can be used to identify audio files.

[0132] Step 105: Obtain the speech data to be recognized.

[0133] In this embodiment of the application, the speech data to be recognized may be mixed Chinese and English speech data.

[0134] Step 106: Input the speech data to be recognized into the speech recognition model to obtain the text information of the speech data to be recognized.

[0135] In this embodiment, the speech data to be recognized is input into the speech recognition model. This model has been trained based on mixed Chinese and English audio data and corresponding audio text data, and can recognize the audio data well, so that the final obtained mixed Chinese and English text is relatively accurate.

[0136] Please see Figure 7 In some instances, the process of inputting the speech data to be recognized into a speech recognition model to obtain the text information of the speech data is described as follows:

[0137] Step 701: Extract the features of the speech data to be recognized and input them into the acoustic model to generate a phoneme sequence.

[0138] Step 702: Decode the phoneme sequence using a decoder consisting of a language model and a vocabulary dictionary to obtain the text information of the speech data to be recognized.

[0139] In this embodiment, the above-described speech recognition method acquires historical audio data and corresponding historical audio text data. The historical audio text data is preprocessed and words are extracted and segmented to obtain vocabulary data. Phoneme annotation is performed on the segmented words based on pre-stored Chinese and English dictionaries and a G2P model, and a vocabulary dictionary is constructed. The historical audio data is input into an acoustic model for training, and the vocabulary data of the historical audio text data is input into a language model for training. The decoder, composed of the trained language model and the vocabulary dictionary, generates a speech recognition model based on the decoder and the trained acoustic model. The speech recognition model is then used to process the speech data to be recognized, obtaining the text information of the speech data, thereby improving recognition accuracy and reducing manual workload.

[0140] Please see Figure 8 Based on the same inventive concept, this application provides a voice recognition device, which includes a communicator 801 and a processor 802.

[0141] Communicator 801 is used to receive historical audio data, historical audio text data, and voice data to be recognized.

[0142] The processor 802, coupled with a communicator, is used for: preprocessing historical audio-text data and extracting vocabulary data from the historical audio-text data; determining phoneme annotations for the vocabulary data based on pre-stored Chinese and English dictionaries and a G2P model, and constructing a vocabulary dictionary; training an acoustic model and a language model based on the vocabulary data from the historical audio data and historical audio-text data, and generating a speech recognition model based on the trained acoustic model, trained language model, and vocabulary dictionary; acquiring the speech data to be recognized, inputting the speech data to be recognized into the speech recognition model, and obtaining the text information of the speech data to be recognized.

[0143] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

Claims

1. A voice recognition method, characterized by, The method comprises: acquiring historical audio data and corresponding historical audio text data; preprocessing the historical audio text data and extracting lexical data in the historical audio text data; determining phoneme annotation of the lexical data based on a pre-stored Chinese dictionary and an English dictionary and a G2P model, and constructing a lexical dictionary, wherein the G2P model is obtained by training based on the historical audio data; training an acoustic model and a language model based on the historical audio data and lexical data of the historical audio text data, generating a speech recognition model based on the trained acoustic model, the trained language model and the lexical dictionary; acquiring to-be-recognized speech data, inputting the to-be-recognized speech data into the speech recognition model to obtain text information of the to-be-recognized speech data; the training method of the G2P model comprises: acquiring the historical audio data, preprocessing the historical audio data, wherein the historical audio data comprises Chinese audio data, English audio data, mixed Chinese-English audio data and audio data of recorded specific special vocabulary; extracting feature vectors of the preprocessed historical audio data; inputting the extracted feature vectors into an acoustic model to obtain a vocabulary table corresponding to the historical audio data; training a G2P model by using the vocabulary table corresponding to the historical audio data to obtain a trained G2P model.

2. The voice recognition method of claim 1, wherein, The preprocessing of the audio text data comprises: performing text normalization processing on the audio text data.

3. The voice recognition method of claim 1, wherein, The extraction of lexical data in the historical audio text data comprises: performing word segmentation processing on the historical audio text data based on a pre-constructed word segmentation tool, extracting lexical data in the historical audio text data based on the word segmented historical audio text data, and forming a lexical list containing all lexical data of the historical audio text data.

4. The voice recognition method of claim 3, wherein, The determination of phoneme annotation of the lexical data based on the pre-stored Chinese dictionary and English dictionary and the G2P model comprises: performing phoneme annotation on Chinese lexical data in the lexical list based on the Chinese dictionary and Chinese audio data in the historical audio data, and performing phoneme annotation on English lexical data in the lexical list based on the English dictionary and English audio data in the historical audio data; if phoneme annotation of English lexical data in the lexical list fails, inputting the failed English lexical data into a G2P model to obtain a pronunciation sequence corresponding to the failed English lexical data; re-performing phoneme annotation on the failed English lexical data based on the pronunciation sequence.

5. The voice recognition method of claim 1, wherein, The construction method of the lexical dictionary comprises: generating a lexical dictionary based on phoneme annotation of lexical data of the historical audio text data, and pre-stored Chinese dictionary and English dictionary.

6. The voice recognition method of claim 1, wherein, The generation method of the speech recognition model comprises: training an acoustic model based on the historical audio data, constructing a language model and training the language model based on lexical data of historical audio text data; generating a decoder based on the trained language model and the lexical dictionary; generating a speech recognition model based on the decoder and the trained acoustic model.

7. The voice recognition method of claim 1, wherein, After the speech recognition model is generated, the method further includes: obtaining test audio data and corresponding text data, and extracting test vocabulary data in the test audio text data; extracting a feature vector of the test vocabulary data and inputting the feature vector into a decoder of the speech recognition model to obtain a test pronunciation phoneme sequence; if there is a phoneme or a vocabulary in the test pronunciation phoneme sequence that is inconsistent with pronunciation of the vocabulary in the test audio data, correcting the speech recognition model.

8. The voice recognition method of claim 7, wherein, The correction of the speech recognition model includes: if there is a phoneme in the test pronunciation phoneme sequence that is inconsistent with pronunciation of the vocabulary in the test audio data, obtaining a correct pronunciation phoneme sequence and writing the correct pronunciation phoneme sequence into a vocabulary dictionary, and updating the speech recognition model according to the new vocabulary dictionary; if there is a vocabulary in the test pronunciation phoneme sequence that is inconsistent with pronunciation of the vocabulary in the test audio data, obtaining a correct pronunciation phoneme sequence and corresponding vocabulary and writing the correct pronunciation phoneme sequence and the corresponding vocabulary into a vocabulary dictionary, and retraining a language model based on the vocabulary data corresponding to the correct pronunciation phoneme sequence, and updating the speech recognition model according to the retrained speech model.

9. The voice recognition method of claim 1, wherein, The inputting of the to-be-recognized speech data into the speech recognition model to obtain text information of the to-be-recognized speech data includes: extracting a feature vector of the to-be-recognized speech data and inputting the feature vector into an acoustic model to generate a phoneme sequence corresponding to the to-be-recognized speech data; decoding the phoneme sequence by a decoder composed of a language model and a vocabulary dictionary to obtain text information of the to-be-recognized speech data.

10. A speech recognition apparatus characterized by comprising: The apparatus includes: a communicator configured to receive historical audio data and corresponding historical audio text data and to-be-recognized speech data; a processor coupled to the communicator and configured to: preprocess the historical audio text data, extract vocabulary data in the historical audio text data, determine phoneme labeling of the vocabulary data based on a pre-stored Chinese dictionary and an English dictionary and a G2P model, and construct a vocabulary dictionary; train an acoustic model and a language model based on the historical audio data and the vocabulary data of the historical audio text data, generate a speech recognition model based on the trained acoustic model, the trained language model, and the vocabulary dictionary, obtain to-be-recognized speech data, and input the to-be-recognized speech data into the speech recognition model to obtain text information of the to-be-recognized speech data; the processor is further configured to: obtain the historical audio data, preprocess the historical audio data, and the historical audio data includes Chinese audio data, English audio data, mixed Chinese-English audio data, and audio data of a recorded specific proprietary vocabulary; extract a feature vector of the preprocessed historical audio data; input the extracted feature vector into an acoustic model to obtain a vocabulary table corresponding to the historical audio data; train a G2P model using the vocabulary table corresponding to the historical audio data to obtain a trained G2P model.

Images