End-to-end speech recognition model processing method, speech recognition method and related device

Abstract

The end-to-end speech recognition model processing method, speech recognition method and related device provided by the present invention include: obtaining a preset word list and a text corpus; wherein the preset word list is used to maintain hot words in any live broadcast field, the pronunciation path and modeling unit corresponding to the hot words; according to the text corpus, the model parameters of the initial language model are trained to obtain the target language model corresponding to the live broadcast field; according to the pronunciation path and modeling unit of the hot words, the target language model is updated, and the weight of the hot words in the updated target language model is determined; according to the updated target language model, an end-to-end speech recognition model is generated. Since the weights of the hot words in the live broadcast field are included in the generated target speech recognition model, the model can automatically select the hot words corresponding to the best weights during the speech recognition process, thereby improving the recognition accuracy of the hot words.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to an end-to-end speech recognition model processing method, a speech recognition method and related devices. Background Art

[0002] Speech recognition is a common processing content in the field of artificial intelligence. For example, in scenarios such as human-computer interaction and speech-to-text, speech recognition is an indispensable processing content.

[0003] For speech recognition software actually used in products, for the same pronunciation, in a specific field, such as the gaming field, the model can recognize the specific field's proper nouns. However, the commonly used end-to-end speech recognition model currently has a poor recognition effect on hot words in a specific field, that is, professional nouns and professional terms.

[0004] How to improve the recognition accuracy of hot words in specific fields in the end-to-end speech recognition model is a technical problem that needs to be solved. Summary of the invention

[0005] One of the purposes of the present invention is to provide an end-to-end speech recognition model processing method, a speech recognition method and related devices, which can improve the recognition accuracy of hot words in a specific field.

[0006] The embodiments of the present invention can be implemented as follows:

[0007] In a first aspect, the present invention provides a method for processing an end-to-end speech recognition model, comprising: obtaining a preset vocabulary and a text corpus; wherein the preset vocabulary is used to maintain hot words in any live broadcast field, pronunciation paths and modeling units corresponding to the hot words; according to the text corpus, training model parameters of an initial language model to obtain a target language model corresponding to the live broadcast field; according to the pronunciation paths and modeling units of the hot words, updating the target language model, and determining the weights of the hot words in the updated target language model; and generating an end-to-end speech recognition model according to the updated target language model.

[0008] In a second aspect, the present invention provides a speech recognition method, the method comprising: obtaining a speech to be recognized; determining a live broadcast field corresponding to the speech to be recognized, and inputting the speech to be recognized into an end-to-end speech recognition model corresponding to the live broadcast field to obtain a text corresponding to the speech to be recognized; wherein the end-to-end speech recognition model corresponding to the live broadcast field includes a language model; the language model is obtained based on a text training set, weights of hot words in a preset vocabulary and a pronunciation path, and the preset vocabulary is used to maintain the hot words in the live broadcast field and the pronunciation paths corresponding to the hot words.

[0009] In a third aspect, the present invention provides an end-to-end speech recognition model processing device, comprising: an acquisition module, used to acquire a preset vocabulary and a text corpus; wherein the preset vocabulary is used to maintain hot words in any live broadcast field, the pronunciation paths and modeling units corresponding to the hot words; a training module, used to train the model parameters of the initial language model according to the text corpus, and obtain the target language model corresponding to the live broadcast field; an update module, used to update the target language model according to the pronunciation paths and modeling units of the hot words, and determine the weights of the hot words in the updated target language model; a generation module, used to generate an end-to-end speech recognition model according to the updated target language model.

[0010] In a fourth aspect, the present invention provides a speech recognition device, comprising: an acquisition module, used to acquire a speech to be recognized; a recognition module, used to determine a live broadcast field corresponding to the speech to be recognized, input the speech to be recognized into an end-to-end speech recognition model corresponding to the live broadcast field, and obtain a text corresponding to the speech to be recognized; wherein the end-to-end speech recognition model corresponding to the live broadcast field includes a language model; the language model is obtained based on a text training set, weights of hot words in a preset vocabulary, a pronunciation path and a modeling unit, and the preset vocabulary is used to maintain the hot words in the live broadcast field and the pronunciation paths corresponding to the hot words.

[0011] In a fifth aspect, the present invention provides an electronic device, comprising a processor and a memory, wherein the memory stores a computer program that can be executed by the processor, and the processor can execute the computer program to implement the method described in any one of the aforementioned embodiments and / or the method described in the aforementioned embodiments.

[0012] In a sixth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method described in any one of the aforementioned embodiments and / or the method described in the aforementioned embodiments is implemented.

[0013] An end-to-end speech recognition model processing method, a speech recognition method and related devices provided by an embodiment of the present invention include: firstly, obtaining a target language model corresponding to a live broadcast field through text corpus training; secondly, updating the generated target language model based on the weights and pronunciation paths of hot words in the live broadcast field, so that the target language model maintains the hot word information in the live broadcast field; finally, generating an end-to-end speech recognition model based on the updated target language model; since the generated target speech recognition model includes the weights of the hot words in the live broadcast field, the model can automatically select the hot words corresponding to the best weights during the speech recognition process, thereby improving the recognition accuracy of the hot words. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 is an example of WFST;

[0016] Figure 2 It is an application environment for speech recognition;

[0017] Figure 3 A schematic flow chart of a method for processing an end-to-end speech recognition model provided by an embodiment of the present invention;

[0018] Figure 4 It is a ternary NGRAM language model arpa file;

[0019] Figure 5 A schematic flow chart of step S303 provided in an embodiment of the present invention;

[0020] Figure 6 A schematic flowchart of step S303-2 provided in an embodiment of the present invention;

[0021] Figure 7 A schematic flow chart of a speech recognition method provided by an embodiment of the present invention;

[0022] Figure 8 A functional module diagram of an end-to-end speech recognition model processing device provided by an embodiment of the present invention;

[0023] Fig. 9 A functional module diagram of a speech recognition device provided by an embodiment of the present invention;

[0024] Fig.10 A block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0026] Accordingly, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the scope of protection of the present invention.

[0027] It should be noted that like reference numerals and letters denote like items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0028] In the description of the present invention, it should be noted that if terms such as "upper", "lower", "inner", "outer", etc. are used to indicate the orientation or positional relationship, it is based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the invention product is usually placed during use. This is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and thus should not be construed as a limitation of the present invention.

[0029] In addition, terms such as "first", "second", etc. are only used for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0030] It should be noted that the features in the embodiments of the present invention can be combined with each other without conflict.

[0031] Before introducing the embodiments of the present invention, the relevant terms in the embodiments of the present invention are first introduced.

[0032] Hot words: Professional nouns, technical terms within a specific field, and words or phrases that are generated in real time and have a usage frequency higher than a preset threshold.

[0033] Modeling unit: The modeling unit for training the end-to-end speech recognition model. Different modeling units are used for Chinese and English. The Chinese modeling unit is Chinese characters. For example, for "你好" (Hello), the corresponding modeling units are "你" (you) and "好" (good). For English, BPE modeling can be used.

[0034] Pronunciation path: Homophones, phrases, confusing words, etc. that may exist for the same pronunciation. For example, for "rogue", the pronunciation paths created according to its actual pronunciation can be "ro g ue", "r哥" (r brother), "肉哥" (meat brother), etc.

[0035] Weighted Finite State Transducers (WFST): is a weighted finite state transducer for large-scale speech recognition, where each state transition is labeled with input and output symbols. Therefore, the constructed network (WFST) is used to generate a mapping from an input symbol sequence or string to an output string. WFST also weights state transitions in addition to input and output symbols. The weight value can be an encoded probability, duration, or any other quantity accumulated along the path to calculate the overall weight of mapping the input string to the output string. WFST is used for speech recognition and usually represents various possible path choices and their corresponding probabilities for outputting recognition results after inputting a speech signal in speech processing.

[0036] For example, see Figure 1 , Figure 1 This is an example of WFST. The numbers in the circle represent states. The input on each edge is a word, and the output is also a word. The numerical value indicates the probability of a word appearing in the current path (i.e., ngram probability). For example, on the first edge, the input is a single "using", and the output word is also "using" with a probability of 1. Next, on the edge from state 1 to state 2, the output word is "data", and the output word is "data" with a probability of 0.66. This continues until the last state is reached, and a string and the corresponding probability of the string can be obtained.

[0037] Speech recognition, also known as Automatic Speech Recognition (ASR), aims to convert the vocabulary content in human speech into computer-readable input, such as keystrokes, binary codes or character sequences. With the development of science and technology, speech recognition technology has been applied to people's lives. For example, in scenarios such as human-computer interaction and speech-to-text, speech recognition is an indispensable processing content.

[0038] See also Figure 2 , Figure 2 The present invention is an application environment of speech recognition, which includes a terminal 110, a server 120 and a server 130, wherein the terminal 110, the server 120 and the server 130 may be connected via a network.

[0039] The terminal 110 can send a training instruction for the speech recognition model to the server 120. The server 120 responds to the model training instruction and performs model training for speech recognition according to the text corpus and speech corpus of different live broadcast scenes. Then, the server 120 deploys the trained speech recognition model to the server 130. The terminal 110 collects the speech to be recognized and sends the speech to be recognized to the server 130, so that the server 130 performs speech recognition on the speech to be recognized through the deployed speech recognition model, obtains the text corresponding to the speech to be recognized, and the server 130 can perform corresponding operations according to the detection result. Alternatively, the server 130 can also send the detection result to the terminal 110 to show the recognition result, and / or perform corresponding operations according to the recognition result.

[0040] In some possible embodiments, the above-mentioned speech recognition can be applied in a live broadcast scenario. For example, server 120 and server 130 can be merged into the same live broadcast server. The live broadcast server trains and deploys the speech recognition model. The speech signal can be collected by the anchor end and transmitted to the live broadcast server. The live broadcast server executes speech recognition technology, which can convert the host's speech content during the live broadcast into text and generate subtitles in real time to improve the user experience.

[0041] The terminal 110 is a smart phone, a tablet computer, a laptop computer, a desktop computer, etc. In addition, the server 120 and the server 130 can be independent physical devices or a server cluster composed of multiple physical servers. For example, the server 120 and the server 130 can be, but are not limited to, a live broadcast server, a game server, a streaming media server, etc.

[0042] Among them, the server 120 has a model training function, and the server 130 has a voice recognition function. This is just an example and not a limitation of their functions. They can also be the same server that integrates both model training and liveness detection functions.

[0043] Please continue to see Figure 2 At present, the commonly used speech recognition model is the end-to-end speech recognition model. Compared with the traditional speech recognition model, the end-to-end speech recognition model significantly reduces the speech recognition error rate. However, with the increase in user demand, for the speech recognition software actually used in products, for the same pronunciation, in specific fields, such as the game field, users expect the model to be able to recognize the proper nouns in the specific field.

[0044] However, the commonly used end-to-end speech recognition models are all integrated into a neural network model. Text information is only used during decoding, and the decoder only retains a small number of candidate words at each step during the search process, resulting in poor decoding, especially for hot words in certain fields, that is, professional nouns and professional terms.

[0045] Therefore, in order to improve the recognition accuracy of the end-to-end speech recognition model for hot words in a specific field, an embodiment of the present invention first provides a processing method for an end-to-end speech recognition model, which is mainly through training a speech model that matches the specific field, and then improving the decoding network of the existing end-to-end speech recognition model according to the speech model and preset pronunciation words, so that the improved end-to-end speech recognition model can accurately recognize the hot words in the specific field, and achieve the effect that for the same pronunciation, the model can give priority to outputting nouns related to the specific field.

[0046] See also Figure 3 , Figure 3 A schematic flow chart of a processing method of an end-to-end speech recognition model provided in an embodiment of the present invention, wherein the processing method of the end-to-end speech recognition model can be applied to Figure 2 The service device 120 or the service device 130 in the embodiment includes:

[0047] S301, obtaining a preset word list and a text corpus; wherein the preset word list is used to maintain hot words in any live broadcast field and pronunciation paths corresponding to the hot words;

[0048] S302, training the model parameters of the initial language model according to the text corpus to obtain a target language model corresponding to the live broadcast field;

[0049] S303: Update the target language model according to the pronunciation path and modeling unit of the hot word, and determine the weight of the hot word in the updated target language model.

[0050] S304: Generate an end-to-end speech recognition model based on the updated target language model.

[0051] According to the end-to-end speech recognition model provided by the embodiment of the present invention, a target language model corresponding to the live broadcast field is first obtained by training with a text corpus set. Secondly, the generated target language model is updated based on the weights and pronunciation paths of the hot words in the live broadcast field, so that the target language model maintains the hot word information in the live broadcast field. Finally, an end-to-end speech recognition model is generated based on the updated target language model, which can effectively improve the recognition accuracy of hot words in actual use.

[0052] It should be noted that the above-mentioned end-to-end speech recognition model can be a pre-processed model, which can be deployed in a device with speech recognition function to achieve speech recognition effect, or it can be an end-to-end speech recognition model generated in real time before speech recognition, which is not limited here.

[0053] The following is an introduction to each exemplary step S301 to step S304 in the embodiment of the present application.

[0054] In step S301, a preset vocabulary and a text corpus are first obtained.

[0055] In the example of this application, the live broadcast field can be, but is not limited to, various types of live broadcast scenes, game event scenes, etc. The hot words can be various proper nouns in the live broadcast field, and can be maintained in the form of a vocabulary, which can be, but is not limited to, maintaining hot words, pronunciation paths of hot words, modeling units, and other information, and can also maintain classification labels corresponding to each hot word, which can be used to characterize the importance of the hot word and to indicate whether to determine weight information for the hot word.

[0056] For example, in a game event scenario, hot words can be nouns representing various game heroes, equipment, skills, etc., or they can be the names of contestants in the event. In one case, classification label a can be used to identify hot words such as names, classification label b can be used to identify basic hot words such as hero names and equipment names, and d can be used to identify other hot words, where a and b are also used to indicate that weight information needs to be determined for the hot word, and d indicates that weight information does not need to be determined for the hot word.

[0057] In an embodiment of the present application, the text corpus may be some pre-collected texts. In order to adapt the trained text to the live broadcast field, the text corpus may include general annotated texts, annotated texts in the live broadcast field, and other texts in the live broadcast field, wherein the annotated text refers to the text from the anchor or host, etc., and the other text in the live broadcast field refers to the text corresponding to any person in the live broadcast field; the general annotated text refers to the text of the anchor or host, etc. in any scene.

[0058] In step S302, it is necessary to train the model parameters of the initial language model based on the text corpus to obtain the target language model corresponding to the live broadcast field.

[0059] It can be understood that a language model is a rational model for establishing text logic and counting the correlation between words. Commonly used language models include NGRAM models. In order to make the decoding accuracy of the end-to-end speech recognition model more accurate, this application can use the ngram model to model the word segmentation of the text corpus, and then convert it into the form of WFST to form the decoding network of the end-to-end speech recognition model.

[0060] In a possible implementation, a text corpus can be used to train an ngram model. Since the higher the number of NGRAM language models, the better, but limited by efficiency considerations, generally ternary or quaternary models are sufficient. Therefore, the present application embodiment adopts a 3gram model as the initial language recognition model, and then trains 3grams with a text corpus. Therefore, the present invention embodiment also provides a possible implementation of step S202:

[0061] a1, determine the first category text corpus, the second category text corpus and the third category text corpus from the text corpus.

[0062] Among them, the text corpora in the first category and the second category have anchor labels; the text corpora in the first category are general texts; the text corpora in the second category and the third category are text corpora in any live broadcast field.

[0063] a2, respectively use the first type of text corpus, the second type of text corpus and the third type of text corpus to train the initial language model to obtain the first language model, the second language model and the third language model.

[0064] a3, interpolate and fuse the first language model, the second language model and the third language model according to a preset ratio to obtain the target language model.

[0065] In the embodiment of the present application, since the data volume of the third type of text corpus is large and the third type of data is particularly large, the trained third language model is very large. In addition, it is text data itself and is quite different from the spoken corpus. Therefore, the third language model can be pruned before fusion.

[0066] Through the above steps a1 to a3, a 3-gram language model can be generated, and then an arpa file of the 3-gram language model can be generated. The arpa file is a form of existence of the language model. The arpa file can easily represent any n-gram language model, but in practice n is usually equal to 3, 4 or 5.

[0067] To understand the above arpa file, please refer to Figure 4 , Figure 4It is a ternary NGRAM language model arpa file. Each line of the arpa file represents a grammar item, which usually contains three parts: probability word(s)[backoff probability], where probability is a value in each line, indicating the probability of the word or phrase occurring. The second value word(s) in each line indicates a specific word or phrase. The third value backoffprobablitiy in each line is optional, indicating the backoff probability. The backoff probability exists when the lower-level grammar has a grammar item with the current word or phrase as a prefix. For example, in the above arpa file, the bigram grammar item "how is the weather" is the prefix of the ternary grammar item "how is the weather", so it has a backoff probability (0 in this simple arpa, usually not 0). With the above arpa file, we can calculate the language model score for a given sentence.

[0068] Optionally, before executing step S303, in order to improve the recognition effect of the model, the embodiment of the present invention further provides an implementation method for preprocessing the text corpus, namely:

[0069] b1. Perform data preprocessing on the text corpus, wherein the preprocessing includes filtering at least one of punctuation marks, special characters, and English characters.

[0070] In this embodiment, data preprocessing is performed on the text corpus, which may include data cleaning and regularization. Regularization includes operations such as converting numbers to Chinese numbers and other operations for converting written language into spoken language.

[0071] b2 performs word segmentation processing on each of the multiple text corpora in the text corpus set, and removes the words obtained after the word segmentation processing that do not exist in the preset pronunciation dictionary.

[0072] In the embodiment of the present application, the word segmentation processing method of Jieba can be used for Chinese but is not limited to it, and the words are kept as they are for English. The obtained word segmentation can also be filtered in combination with a preset pronunciation dictionary, and the words not in the dictionary are filtered by using <unk>Instead.

[0073] b3, Perform regular processing on the text corpus according to the preset vocabulary list.

[0074] In the embodiments of the present application, hot words can be replaced and unified. For example, for words with different forms in the text, such as "huanfeng" and "焕峰", they can be unified into a regular form, which can be unified into pinyin form or Chinese form. In some live event scenarios, the recognition of player names may occur. Since player names generally do not have Chinese characters and are all English words or pinyin, if there are player names in Chinese text, regular processing needs to be performed to uniformly convert them into pinyin form.

[0075] After obtaining the target language model, in order to enable the model to support the hot word boosting function, so that the model can automatically select the best weight and improve the hot word recognition accuracy, the target language model can be updated and obtained using step S303.

[0076] In step S303, update the target language model according to the pronunciation path and modeling unit of the hot word, and determine the weight of the hot word in the updated target language model.

[0077] That is to say, after generating the target language model in the embodiments of the present application, in order to support the hot word boosting function, the weight of the hot word can be determined in the target language model. In this way, during the process of decoding using the language model, the recognition rate of the hot word can be improved based on the weight of the hot word.

[0078] The following introduces two implementation methods for determining the weight of hot words in the embodiments of the present application.

[0079] The first implementation method is to determine the weight information of the hot words in the preset vocabulary list in response to the user's weight configuration information.

[0080] That is to say, first, according to the hot word list, add the 1gram of the pronunciation path of the previously determined hot word into the arpa file of the target language model and initialize the weight to 0. The pronunciation path of the hot word is included in the target language model, and then the weight of the hot word is manually increased, and the weight range can be between 0 and 9.

[0081] The second implementation method, that is, step S303 may include the following steps, please refer to Figure 5 , Figure 5 Schematic flowchart of step S303 provided by the embodiments of the present invention:

[0082] S303-1, Generate a test set according to the preset vocabulary list; the test set includes multiple test texts.

[0083] In the embodiment of the present application, some hot words in the preset vocabulary may exist in the test set, while some may not. For the hot words in these two situations, the embodiment of the present application subsequently provides different implementation methods for determining weights.

[0084] S303-2, based on the target language model and the preset acoustic model, multiple rounds of decoding are performed on multiple test texts to determine the decoding error rate after each round of decoding and the scores of hot words in the preset vocabulary and the weights corresponding to the scores.

[0085] In the embodiment of the present application, it can be seen from the embodiment of the above step S301 that each hot word in the preset vocabulary has a different classification label, some of which indicate that the hot word does not need to determine the weight. Therefore, in step S303-2, for the hot word whose classification label indicates that it is not necessary to determine the weight of the hot word, it is not necessary to determine the weight.

[0086] The following is an implementation method for determining the weight of the hot words in two situations that may occur in step S303-1. Therefore, step S303-2 may include the following steps: Figure 6 , Figure 6 A schematic flow chart of step S303-2 provided in an embodiment of the present invention:

[0087] S303-2-1, after each round of decoding, determine whether each hot word corresponds to at least one test text.

[0088] That is to say, if a hot word does not exist in any test text, it can be considered that the hot word has no corresponding test text and is not in the test set; conversely, if the hot word appears in at least one test text, it can be considered that the hot word has a corresponding test text and exists in the test set.

[0089] S303-2-2, if so, determine the score based on the recall rate, precision rate and classification label corresponding to each hot word. If the score is higher than the score of the previous round, increase the weight corresponding to the score of the previous round according to the preset rules to obtain the weight corresponding to the score. Alternatively, if the score is lower than the score of the previous round, reduce the weight corresponding to the score of the previous round according to the preset rules to obtain the weight corresponding to the score; wherein the classification label is used to characterize the importance of the hot word.

[0090] S303-2-3, if not, then when the number of recognition errors for each hot word is greater than the preset number, the previous weight is reduced according to the preset rule, and the obtained weight is used as the score; or, when it is less than or equal to the preset number, the previous weight is increased according to the preset rule, and the obtained weight is used as the score.

[0091] That is to say, the test set is decoded, and then the weights of all hot words are initialized to 0. After each round of decoding, the word error rate (wer) of the test set and the corresponding decoding results can be obtained.

[0092] For hot words included in the test set, the corresponding recall and precision can be calculated according to the decoding results, and different F-beta scores can be calculated according to the classification labels corresponding to the hot words. Then, the current score is compared with the previous score obtained in the previous round of decoding. If the current score is higher than the previous score, the weight is increased, otherwise the weight is reduced. For hot words not included in the test set, a threshold for the number of incorrect recognitions of the hot word is set. If the number of incorrect recognitions of the hot word is less than two, the weight is increased, and if it is higher, the weight is reduced, and the score is set to be consistent with the weight.

[0093] In the above embodiment, it involves increasing weights and decreasing weights. In a possible implementation, a preset rule for increasing or decreasing can be set. The preset rule can be but is not limited to 1 unit. For example, if the previous weight is 6, it is increased by 1 unit or decreased by 1 unit to achieve the effect of dynamically adjusting the weight according to the decoding result, so that the obtained weight can finally form an adaptive effect with the target language model.

[0094] After determining the decoding error rate corresponding to each round of decoding, the score and weight corresponding to the hot word, step S303 - 3 can be executed to determine the best weight of the hot word, and the decoding process ends.

[0095] S303-3, if there are consecutive preset rounds, and the decoding error rate after each round of decoding is less than the preset threshold, the highest score is determined from the scores corresponding to each hot word, and the weight corresponding to the highest score is determined as the weight of the hot word.

[0096] That is to say, after obtaining the weights after the first iteration, the weights are added to the arpa file of the target language model to update the target language model, and then the iteration continues. The initial learning rate is 1, and each round of iteration is reduced to the previous 0.9. When the decoding error rate of consecutive preset rounds (assuming three rounds) is lower than the preset threshold, the weight corresponding to the best score of the hot word with the determined weight can be used to generate the final weight, and the target language model is updated based on the final weight.

[0097] Optionally, in order to ensure the adaptability of the target language model, after each round of decoding, the target language model can be updated based on the weights corresponding to the hot words determined after each round of decoding; the updated target language model can regenerate a decoding network, and the regenerated decoding network can then perform the next round of decoding.

[0098] After the target language model is obtained, a decoding network can be constructed according to the target language model in combination with a preset pronunciation dictionary, and an end-to-end speech recognition model can be generated based on the constructed decoding network. Therefore, step S304 can be continued.

[0099] In step S304, an end-to-end speech recognition model is generated according to the updated target language model.

[0100] In an embodiment of the present application, after obtaining the target language model, the final decoding network can be obtained in combination with the pronunciation dictionary, and the decoder of the existing end-to-end speech recognition model can be improved based on the generated decoding network, so as to obtain an improved end-to-end speech recognition model.

[0101] In the embodiment of the present application, the decoder in the improved end-to-end speech recognition model can be but is not limited to a CTC model. The pronunciation dictionary and the language model provided by the embodiment of the present invention are integrated into the CTC decoding, and N-best is generated by CTC WFSTsearch, which can improve the recognition accuracy of hot words.

[0102] Based on the above improved end-to-end speech recognition model, the embodiment of the present invention also provides a speech recognition method, which can be applied to Figure 2 For server 120 or server 130 in Figure 7 , Figure 7 A schematic flow chart of a speech recognition method provided by an embodiment of the present invention:

[0103] S401, obtaining speech to be recognized.

[0104] S402, determining the live broadcast domain corresponding to the speech to be recognized, and inputting the speech to be recognized into an end-to-end speech recognition model corresponding to the live broadcast domain to obtain text corresponding to the speech to be recognized.

[0105] Among them, the end-to-end speech recognition model corresponding to the live broadcast field includes a language model; the language model is obtained based on the text training set, the weights and pronunciation paths of hot words in the preset vocabulary, and the preset vocabulary is used to maintain the hot words in the live broadcast field, the pronunciation paths corresponding to the hot words, and the modeling units.

[0106] According to an embodiment of the present invention, a speech recognition method is provided. First, a speech to be recognized is obtained, and then a live broadcast field corresponding to the recognized speech is determined, and recognition is performed based on an end-to-end speech recognition model corresponding to the live broadcast field. Since the language model of the end-to-end speech recognition model is obtained based on a text training set, weights and pronunciation paths of hot words in a preset vocabulary, and a modeling unit, the end-to-end speech recognition model can automatically select hot words with the best weights, thereby effectively improving the recognition accuracy of hot words.

[0107] It should be noted that the above end-to-end speech recognition model may be a pre-processed model, which may be deployed in a device with a speech recognition function to achieve a speech recognition effect, or may be a pre-processed model that is pre-processed before step S401 is performed. Figure 3 The end-to-end speech recognition model generated in real time by each step in is not limited here.

[0108] In order to realize the above-mentioned embodiment Figure 3 The end-to-end speech recognition model is obtained by performing the steps in the above. The end-to-end speech recognition model processing method provided by the embodiment of the present invention can be executed in a hardware device or in the form of a software module. When the end-to-end speech recognition model processing method is implemented in the form of a software module, the embodiment of the present invention also provides a training device for a liveness detection model. Figure 8 , Figure 8 The functional module diagram of the end-to-end speech recognition model processing device provided by the embodiment of the present invention, the end-to-end speech recognition model processing device 500 may include:

[0109] The acquisition module 510 is used to acquire a preset vocabulary and a text corpus; wherein the preset vocabulary is used to maintain hot words in any live broadcast field, the pronunciation paths and modeling units corresponding to the hot words;

[0110] A training module 520 is used to train the model parameters of the initial language model according to the text corpus to obtain a target language model corresponding to the live broadcast field;

[0111] An updating module 530, configured to update the target language model according to the pronunciation path and modeling unit of the hot word, and determine the weight of the hot word in the updated target language model;

[0112] The generating module 540 is used to generate an end-to-end speech recognition model according to the updated target language model.

[0113] It is understandable that the acquisition module 510, the training module 520, the update module 530 and the generation module 540 can be executed in coordination. Figure 3 Each step in the process is performed to achieve the corresponding technical effects.

[0114] In a possible implementation, the update module 530 is further configured to execute Figure 5 , Figure 6 Each step in the process is performed to achieve the corresponding technical effect.

[0115] In a possible implementation manner, the training module 520 is also specifically used in step a1 to step a3 in the above step S202 to achieve corresponding technical effects.

[0116] In a possible implementation, the end-to-end speech recognition model processing device 500 can also be used to execute the above steps b1 to b3 to achieve corresponding technical effects.

[0117] In order to realize the above-mentioned embodiment Figure 7 The various steps in the embodiment of the present invention can achieve the effect of speech recognition. The speech recognition method provided by the embodiment of the present invention can be executed in a hardware device or in the form of a software module. When the speech recognition method is implemented in the form of a software module, the embodiment of the present invention also provides a speech recognition device, see Fig. 9 , Figure 8 The functional module diagram of the speech recognition device provided by the embodiment of the present invention, the speech recognition device 600 may include:

[0118] The acquisition module 610 is used to acquire the speech to be recognized.

[0119] A recognition module 620 is used to determine the live broadcast field corresponding to the speech to be recognized, input the speech to be recognized into an end-to-end speech recognition model corresponding to the live broadcast field, and obtain a text corresponding to the speech to be recognized;

[0120] Among them, the end-to-end speech recognition model corresponding to the live broadcast field includes a language model; the language model is obtained based on a text training set, the weights of hot words in a preset vocabulary, a pronunciation path and a modeling unit, and the preset vocabulary is used to maintain the hot words in the live broadcast field, the pronunciation paths and modeling units corresponding to the hot words.

[0121] It is understandable that the acquisition module 610 and the identification module 620 can be executed in coordination. Figure 7 Each step in the process is performed to achieve the corresponding technical effect.

[0122] It should be noted that the various functional modules in the end-to-end speech recognition model processing device 500 and / or the speech recognition device 600 provided in the embodiment of the present invention can be stored in the memory in the form of software or firmware or solidified in the operating system (OS) of the electronic device 700, and can be executed by the processor in the electronic device 700. At the same time, the data and program codes required to execute the above modules can be stored in the memory.

[0123] In a possible implementation manner, the electronic device 400 may be shaped as follows: Figure 1 The server 120 in the embodiment can be used to deploy the end-to-end speech recognition model processing device 500, and the electronic device 400 can also be in the form of Figure 1 The server 130 is used to deploy the speech recognition device 600.

[0124] like Fig.10 , Fig.10 A block diagram of an electronic device provided in an embodiment of the present invention. The electronic device 700 includes a communication interface 701, a processor 702 and a memory 703. The processor 702, the memory 703 and the communication interface 701 are electrically connected to each other directly or indirectly to achieve data transmission or interaction. For example, these components can be electrically connected to each other through one or more communication buses or signal lines. The memory 703 can be used to store software programs and modules, such as the processing method of the end-to-end speech recognition model and / or the program instructions / modules corresponding to the speech recognition method provided in an embodiment of the present invention. The processor 702 executes the software programs and modules stored in the memory 703 to perform various functional applications and data processing. The communication interface 701 can be used to communicate signaling or data with other node devices. In the present invention, the electronic device 70 can have multiple communication interfaces 701.

[0125] Among them, the memory 703 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0126] The processor 702 may be an integrated circuit chip with signal processing capability. The processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0127] The embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the processing method of the end-to-end speech recognition model and / or the speech recognition method as described in any one of the aforementioned embodiments are implemented. The computer-readable storage medium can be, but is not limited to, various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a PROM, an EPROM, an EEPROM, a magnetic disk, or an optical disk.

[0128] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.< / unk>

Claims

1. An end-to-end speech recognition model processing method, characterized in that: include: Obtain a preset vocabulary and a text corpus; wherein the preset vocabulary is used to maintain hot words in any live broadcast field, pronunciation paths corresponding to the hot words, and modeling units; According to the text corpus, the model parameters of the initial language model are trained to obtain the target language model corresponding to the live broadcast field, including: determining a first type of text corpus, a second type of text corpus and a third type of text corpus from the text corpus; wherein the text corpora in the first type of text corpus and the second type of text corpus have anchor labels; the text corpora in the first type of text corpus are general texts; the text corpora in the second type of text corpus and the third type of text corpus are text corpora in any of the live broadcast fields; respectively using the first type of text corpus, the second type of text corpus and the third type of text corpus to train the initial language model to obtain a first language model, a second language model and a third language model; interpolating and fusing the first language model, the second language model and the third language model according to a preset ratio to obtain the target language model; According to the pronunciation path and modeling unit of the hot word, the target language model is updated, and the weight of the hot word in the updated target language model is determined; An end-to-end speech recognition model is generated based on the updated target language model.

2. The end-to-end speech recognition model processing method according to claim 1, characterized in that: Determine the weight of the hot word in the updated target language model, including: Generate a test set according to the preset vocabulary; the test set includes multiple test texts; Based on the target language model and the preset acoustic model, the multiple test texts are decoded multiple times to determine the decoding error rate after each decoding round and the scores of the hot words in the preset vocabulary and the weights corresponding to the scores; If there are consecutive preset rounds, and the decoding error rate after each round of decoding is less than a preset threshold, the highest score is determined from the scores corresponding to the hot words, and the weight corresponding to the highest score is determined as the weight of the hot word.

3. The end-to-end speech recognition model processing method according to claim 2, characterized in that: Determining the score of each hot word in the preset word list after each round of decoding and the weight corresponding to the score includes: After each round of decoding, determine whether each hot word corresponds to the test text; If yes, the score is determined according to the recall rate, precision rate and classification label corresponding to each hot word. If the score is higher than the score of the previous round, the weight corresponding to the score of the previous round is increased according to the preset rules to obtain the weight corresponding to the score, or, In the case where the score is lower than the score of the previous round, the weight corresponding to the score of the previous round is reduced according to a preset rule to obtain the weight corresponding to the score; wherein the classification label is used to characterize the importance of the hot word; If not, when the number of recognition errors of each hot word is greater than the preset number, the previous weight is reduced according to the preset rule, and the obtained weight is used as the score; or, when it is less than or equal to the preset number, the previous weight is increased according to the preset rule, and the obtained weight is used as the score.

4. The end-to-end speech recognition model processing method according to claim 2 or 3, characterized in that: After determining the decoding error rate of the test set after each round of decoding and the score of each hot word in the preset vocabulary and the weight corresponding to the score, the method further includes: Based on the weights corresponding to the hot words determined after each round of decoding, the target language model is updated; the updated target language model is used to perform the next round of decoding.

5. The end-to-end speech recognition model processing method according to claim 1, characterized in that: Determine the weight of the hot word in the updated target language model, including: In response to the user's weight configuration information, the weight information of the hot words in the preset word list is determined.

6. The end-to-end speech recognition model processing method according to claim 1, characterized in that: After obtaining the preset vocabulary and text corpus, the method further includes: Performing data preprocessing on the text corpus, wherein the preprocessing includes filtering at least one of punctuation marks, special characters, and English characters; Performing word segmentation processing on each of the plurality of text corpora in the text corpus set, and removing words obtained after the word segmentation processing that do not exist in the preset pronunciation dictionary; According to the preset vocabulary, regularization processing is performed on the text corpus.

7. A speech recognition method, characterized in that: The method comprises: Get the speech to be recognized; Determine the live broadcast domain corresponding to the speech to be recognized, and input the speech to be recognized into an end-to-end speech recognition model corresponding to the live broadcast domain to obtain a text corresponding to the speech to be recognized; The end-to-end speech recognition model corresponding to the live broadcast field includes a language model; the end-to-end speech recognition model is obtained based on a text corpus, weights of hot words in a preset vocabulary, and pronunciation paths, and the preset vocabulary is used to maintain hot words in the live broadcast field, pronunciation paths corresponding to the hot words, and modeling units; The training method of the language model is specifically as follows: determining a first type of text corpus set, a second type of text corpus set and a third type of text corpus set from the text corpus; wherein the text corpora in the first type of text corpus set and the second type of text corpus set have anchor labels; the text corpora in the first type of text corpus set are general texts; the text corpora in the second type of text corpus set and the third type of text corpus set are text corpora in any one of the live broadcast fields; respectively using the first type of text corpus set, the second type of text corpus set and the third type of text corpus set to train the initial language model to obtain a first language model, a second language model and a third language model; interpolating and fusing the first language model, the second language model and the third language model according to a preset ratio to obtain the language model.

8. An end-to-end speech recognition model processing device, characterized in that: include: An acquisition module, used to acquire a preset vocabulary and a text corpus; wherein the preset vocabulary is used to maintain hot words in any live broadcast field, pronunciation paths corresponding to the hot words, and modeling units; A training module, used to train the model parameters of the initial language model according to the text corpus, and obtain the target language model corresponding to the live broadcast field, including: determining a first type of text corpus, a second type of text corpus and a third type of text corpus from the text corpus; wherein the text corpora in the first type of text corpus and the second type of text corpus have anchor labels; the text corpora in the first type of text corpus are general texts; the text corpora in the second type of text corpus and the third type of text corpus are text corpora in any one of the live broadcast fields; respectively using the first type of text corpus, the second type of text corpus and the third type of text corpus to train the initial language model to obtain a first language model, a second language model and a third language model; interpolating and fusing the first language model, the second language model and the third language model according to a preset ratio to obtain the target language model; an updating module, used to update the target language model according to the pronunciation path and modeling unit of the hot word, and determine the weight of the hot word in the updated target language model; A generation module is used to generate an end-to-end speech recognition model based on the updated target language model.

9. A speech recognition device, characterized in that: include: An acquisition module, used to acquire the speech to be recognized; A recognition module, used to determine the live broadcast field corresponding to the speech to be recognized, input the speech to be recognized into an end-to-end speech recognition model corresponding to the live broadcast field, and obtain a text corresponding to the speech to be recognized; The end-to-end speech recognition model corresponding to the live broadcast field includes a language model; the end-to-end speech recognition model is obtained based on a text corpus, weights of hot words in a preset vocabulary, a pronunciation path, and a modeling unit, and the preset vocabulary is used to maintain hot words in the live broadcast field, the pronunciation paths corresponding to the hot words, and the modeling units; The training method of the language model is specifically as follows: determining a first type of text corpus set, a second type of text corpus set and a third type of text corpus set from the text corpus; wherein the text corpora in the first type of text corpus set and the second type of text corpus set have anchor labels; the text corpora in the first type of text corpus set are general texts; the text corpora in the second type of text corpus set and the third type of text corpus set are text corpora in any one of the live broadcast fields; respectively using the first type of text corpus set, the second type of text corpus set and the third type of text corpus set to train the initial language model to obtain a first language model, a second language model and a third language model; interpolating and fusing the first language model, the second language model and the third language model according to a preset ratio to obtain the language model.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a computer program executable by the processor, and the processor can execute the computer program to implement the method according to any one of claims 1 to 6 and / or the method according to claim 7.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 and / or the method according to claim 7 are implemented.

Images