Voice recognition method and device, storage medium, and electronic device

Abstract

The present disclosure provides a speech recognition method, a speech recognition device, a storage medium and an electronic device, and relates to the field of computer technology. The speech recognition method includes: obtaining a speech to be recognized, and extracting features of the speech to be recognized to obtain speech features; predicting the speech features by combining an acoustic model constructed by a modeling unit to determine the probability distribution of the modeling unit of the speech to be recognized; the modeling unit includes a main language character and an auxiliary language subword; the speech to be recognized is recognized based on a language model constructed by the main language characters and the auxiliary language subwords to determine the grammatical score of the speech to be recognized; combining the probability distribution and the grammatical score to determine the recognition result of the speech to be recognized. The technical solution of the present disclosure can improve the accuracy of speech recognition.

Description

Background Art

[0002] With the development of speech recognition technology, it has become common for users to communicate in multiple languages. Therefore, single-language speech recognition can no longer meet the recognition needs in many scenarios.

[0003] In related technologies, language recognition is mainly performed in two ways: one is the traditional speech recognition system, which consists of three parts: acoustic model, pronunciation dictionary and language model. Different models need to be trained separately, and then the various parts are integrated together to form a decoding graph, and the recognition result is obtained through decoding. The second is an end-to-end speech recognition system, which only needs to give the input voice and the output text information, train a model, and directly realize the conversion of input voice to output text.

[0004] In the above method, the pronunciation of phonemes requires a lot of linguistic expertise to build a pronunciation dictionary for this mixed language. The operation steps are complicated and there may be deviations, and the accuracy is low. In addition, when Chinese characters and English letters are used as a set of modeling units, there are obvious differences between Chinese characters and English letters, and there may be problems that cannot be fully modeled or cannot be processed by the model. Therefore, the modeling method is unreasonable, resulting in a small application scope of speech recognition and low accuracy of speech recognition.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The present disclosure provides a speech recognition method and device, a computer-readable storage medium and an electronic device, thereby overcoming the problem of low accuracy of speech recognition in the related art at least to a certain extent.

[0007] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0008] According to one aspect of the present disclosure, a speech recognition method is provided, comprising: obtaining a speech to be recognized, and performing feature extraction on the speech to be recognized to obtain speech features; predicting the speech features by combining an acoustic model constructed by a modeling unit, and determining a probability distribution of the modeling unit of the speech to be recognized; the modeling unit comprises a primary language character and an auxiliary language subword; performing recognition processing on the speech to be recognized based on a language model constructed by the primary language characters and the auxiliary language subwords, and determining a grammatical score of the speech to be recognized; and determining a recognition result of the speech to be recognized in combination with the probability distribution and the grammatical score.

[0009] In an exemplary embodiment of the present disclosure, the method further includes: obtaining training corpus and performing feature extraction on the training corpus to determine reference speech features of the training corpus; and training an acoustic neural network model in combination with the reference speech features and the main language characters and the auxiliary language subwords in the modeling unit to obtain an acoustic model.

[0010] In an exemplary embodiment of the present disclosure, the feature extraction of the training corpus to determine the reference speech features of the training corpus includes: performing frame segmentation and windowing operations on the training corpus to obtain multiple speech frames; and transforming the multiple speech frames to obtain the reference speech features.

[0011] In an exemplary embodiment of the present disclosure, the acoustic neural network model is trained in combination with the reference speech feature and the modeling unit to obtain the acoustic model, including: inputting the reference speech feature into the acoustic neural network model to determine the probability distribution of the predicted label corresponding to the reference speech feature; comparing the predicted label with the real label represented by the modeling unit, and determining the loss function according to the distance between the predicted label and the real label; adjusting the model parameters of the neural network model according to the loss function until the loss function is minimized to obtain the acoustic model.

[0012] In an exemplary embodiment of the present disclosure, the method further includes: splitting the training corpus according to the minimum granularity to obtain a character sequence; counting the number of occurrences of each consecutive character pair in the character sequence, merging the character pairs with the highest number of occurrences to generate a new character sequence, until an end condition is met, so as to determine the auxiliary language subword.

[0013] In an exemplary embodiment of the present disclosure, the method further includes: splitting the training corpus according to the main language characters and the auxiliary language subwords in the modeling units to obtain splitting results, and generating the language model according to the splitting results.

[0014] In an exemplary embodiment of the present disclosure, the combination of the probability distribution and the grammatical score to determine the recognition result of the speech to be recognized includes: decoding the probability distribution of the modeling unit to determine the candidate decoding path of the speech to be recognized; evaluating the candidate decoding path in combination with the grammatical score to determine the recognition result of the speech to be recognized.

[0015] In an exemplary embodiment of the present disclosure, the candidate decoding paths are evaluated in combination with the grammatical scores to determine the recognition result of the speech to be recognized, including: combining the acoustic model score of the target character with the grammatical score of the target character to obtain the path score of the candidate decoding path; and taking the candidate decoding path with the highest path score as the recognition result.

[0016] According to one aspect of the present disclosure, a speech recognition device is provided, comprising: a feature extraction module, used to obtain speech to be recognized, and perform feature extraction on the speech to be recognized to obtain speech features; a probability determination module, used to predict the speech features by combining an acoustic model constructed by a modeling unit, and determine the probability distribution of the modeling unit of the speech to be recognized; the modeling unit includes a main language character and an auxiliary language subword; a grammar score determination module, used to perform recognition processing on the speech to be recognized based on a language model constructed by the main language characters and the auxiliary language subwords, and determine the grammar score of the speech to be recognized; and a recognition result determination module, used to determine the recognition result of the speech to be recognized in combination with the probability distribution and the grammar score.

[0017] According to one aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the speech recognition method described in any one of the above is implemented.

[0018] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform any one of the above-mentioned speech recognition methods by executing the executable instructions.

[0019] The speech recognition method, speech recognition device, computer-readable storage medium, and electronic device provided in the embodiments of the present disclosure can determine the label of the speech to be recognized based on the acoustic model constructed by the modeling unit generated by the main language characters and the auxiliary language subwords, and can further determine its grammatical score based on the language model generated by the modeling unit, and then determine the recognition result of the speech to be recognized. On the one hand, due to the use of auxiliary language subwords for modeling, the gap between the length of the modeling units of the main language and the auxiliary language is narrowed, the problem of data sparsity is avoided, and the rationality of the modeling method is improved. On the one hand, due to the use of main language characters and auxiliary language subwords to generate modeling units and then modeling, the limitation of not being able to fully model is avoided, the completeness and accuracy of the model are improved, the operating efficiency is improved, the application scope of language recognition is increased, and the accuracy of speech recognition is improved.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and are used together with the specification to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 The following schematically shows a system architecture diagram for implementing a speech recognition method according to an embodiment of the present disclosure.

[0023] Figure 2 A schematic diagram schematically illustrates a speech recognition method in an embodiment of the present disclosure.

[0024] Figure 3 The specific flow chart of determining auxiliary language subwords in the embodiment of the present disclosure is schematically shown.

[0025] Figure 4 The following schematic diagram shows the process of determining English subwords in an embodiment of the present disclosure.

[0026] Figure 5 The following is a schematic diagram of a process for determining an acoustic model in an embodiment of the present disclosure.

[0027] Figure 6 The specific process diagram of model training in the embodiment of the present disclosure is schematically shown.

[0028] Figure 7 The overall flow chart of model adjustment in the embodiment of the present disclosure is schematically shown.

[0029] Figure 8 The overall process flow diagram for determining the recognition result in the embodiment of the present disclosure is schematically shown.

[0030] Fig. 9 The overall process flow diagram of voice recognition in the embodiment of the present disclosure is schematically shown.

[0031] Fig.10 The block diagram of the speech recognition device in the embodiment of the present disclosure is schematically shown.

[0032] Fig.11 A block diagram of an electronic device in an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0033] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0034] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0035] In the related art, the recognition task of mixed Chinese and English is divided into two types: one is inter-sentence mixing; the other is intra-sentence mixing. Most of the mixed Chinese and English recognition systems are aimed at the situation of inter-sentence mixing, specifically referring to the mixed Chinese and English system constructed by the language recognition model and two single-language speech recognition models. That is, it is necessary to first identify the language of the speech to determine whether it is Chinese or English, and then input the speech into the corresponding single-language speech recognition system for recognition according to the language type. That is, if the speech "hello" is judged as Chinese after language recognition, it is input into the Chinese speech recognition system for recognition; if the speech "How are you" is judged as English after language recognition, it is input into the English speech recognition system for recognition. Although this system can recognize both Chinese and English, it cannot handle the situation of Chinese and English mixed in a sentence, such as "play a song of Justin Bieber's baby", and cannot meet the needs of mixed Chinese and English recognition in more scenarios.

[0036] There are two main types of existing speech recognition systems. The first is the traditional speech recognition system, which consists of three parts: an acoustic model, a pronunciation dictionary, and a language model. Different models need to be trained separately, and then the various parts are fused together through WFST to form a decoding graph, and the recognition result is obtained through decoding. The second is an end-to-end speech recognition system, which only requires the input speech and the output text information to be given, and a model is trained to directly realize the conversion of input speech to output text.

[0037] For the recognition of mixed Chinese and English in sentences, the solution in traditional speech recognition systems is to build a pronunciation dictionary of mixed Chinese and English, and represent Chinese and English words with the same common pronunciation phoneme set. Therefore, when training the acoustic model, both Chinese and English words can be mapped to corresponding phonemes through the pronunciation dictionary, and the acoustic model can learn the pronunciation of these phonemes, so that Chinese and English can be modeled at the same time. However, when this method uses the same set of pronunciation phonemes to represent two different languages, what kind of phonemes can be shared by Chinese and English, what kind of phonemes are unique to a certain language and need to be established separately, and a large number of Chinese and English words need to be annotated with phonemes one by one. These problems require a lot of linguistic expertise to solve, and professional linguists to build such a mixed language pronunciation dictionary, which is relatively expensive, and such a manually constructed dictionary inevitably has a certain degree of error.

[0038] The end-to-end model can directly realize the input and decoding recognition of speech in one step, without introducing the concept of phoneme to word pronunciation dictionary in the traditional system, and directly trains the audio to text model, avoiding the complicated alignment work and pronunciation dictionary production work in the traditional speech recognition system. Therefore, this patent adopts an end-to-end speech recognition system, but similarly, when realizing the mixed recognition of Chinese and English mixed in a sentence, it also faces the problem of how to model Chinese and English at the same time. For mixed Chinese and English tasks, a natural way is to use Chinese characters and English letters as a set of modeling units; but there are obvious differences between Chinese characters and English letters. For acoustic features, Chinese characters are longer than English letters, so this modeling method is not reasonable enough; and if English words are used for modeling, there is a problem of open vocabulary, that is, due to the huge English vocabulary, it is impossible to model all of them. When words other than modeling words appear, the model cannot handle the problem.

[0039] In order to solve the above technical problems, a speech recognition method is provided in an embodiment of the present disclosure. Figure 1 A schematic diagram showing an exemplary system architecture to which the technical solution of the embodiments of the present disclosure can be applied.

[0040] like Figure 1As shown, the system architecture 100 may include a transmitter 101, a network 102, and a receiver 103. The transmitter may be a user or a client capable of sending voice, such as a smart phone, a computer, a tablet computer, a smart speaker, or other terminals. The network 102 is used to provide a medium for a communication link between the transmitter 101 and the receiver 103 so that the receiver can receive the voice. The network 102 may include various connection types, such as a wired communication link, a wireless communication link, and the like. In the embodiment of the present disclosure, the network 102 between the transmitter 101 and the receiver 103 may be a wired communication link, such as a communication link provided by a serial port connection line, or a wireless communication link, providing a communication link through a wireless network. The receiver 103 may be a server or a client with a voice processing function, such as a portable computer, a desktop computer, a smart phone, or other terminal equipment with a voice processing function, which is used to process the voice sent by the transmitter.

[0041] In the disclosed embodiment, the transmitting end can be used to send voice, for example, a user sends voice or a client plays voice. After receiving the voice, the receiving end recognizes the received voice according to the acoustic model, language model and decoder therein to obtain a recognition result.

[0042] It should be noted that the speech recognition method provided in the embodiment of the present disclosure can be completely executed by the receiving end, and accordingly, the speech recognition device can be set in the receiving end 103.

[0043] Based on the above system architecture, the present disclosure provides a speech recognition method for recognizing a mixed speech of Chinese and English. Figure 2 As shown in , the speech recognition method includes steps S210 to S240, which are described in detail as follows:

[0044] In step S210, speech to be recognized is obtained, and feature extraction is performed on the speech to be recognized to obtain speech features.

[0045] In the embodiments of the present disclosure, the speech to be recognized refers to the speech uttered by the user or the speech played by the client. The speech to be recognized can be obtained by mixing different types of languages, for example, it can be a speech combined with a primary language and an auxiliary language. The primary language and the auxiliary language in the speech to be recognized can be any form of language, which is specifically configured according to the application scenario, but the types of the primary language and the auxiliary language are different.

[0046] It should be noted that the primary language and the auxiliary language are used to distinguish the types of languages, and there is no limit to the proportion of the primary language in the speech to be recognized. For example, the proportion of the primary language in the speech to be recognized can be less than or equal to the proportion of the auxiliary language in the speech to be recognized, as long as both the primary language and the auxiliary language exist in the speech to be recognized. For example, the primary language can be Chinese and the auxiliary language can be English; or the primary language can be English and the auxiliary language can be Chinese, and so on.

[0047] Here, the main language is Chinese and the auxiliary language is English. Based on this, the speech to be recognized can be a mixture of Chinese and English, and the mixing mode can be intra-sentence mixing or inter-sentence mixing. Here, intra-sentence mixing is used as an example for description.

[0048] After receiving the speech to be recognized, feature extraction can be performed on the speech to be recognized to obtain its corresponding speech features. Specifically, the speech signal of the speech to be recognized can be determined as a speech frame, and then the speech frame is subjected to signal processing to obtain the speech features. Among them, first, the original waveform of the speech signal of the speech to be recognized can be divided into multiple speech frames through frame segmentation and windowing operations, where the speech frame refers to a small segment of audio; further, the multiple speech frames are subjected to fast Fourier transform, and then filtered through a Mel filter, and the logarithm is calculated, and the calculated preset data is used as the Fbank feature to obtain the speech features of the speech to be recognized. The preset data here can be the first 80-dimensional data or data of any dimensionality, and there is no special limitation here.

[0049] In step S220, the speech features are predicted by combining the acoustic model constructed by the modeling unit to determine the probability distribution of the modeling unit of the speech to be recognized; the modeling unit includes a main language character and an auxiliary language subword.

[0050] In the disclosed embodiment, the modeling unit refers to a character unit used to construct a model. Depending on the length of the time scale, the modeling unit can be selected as a sentence, a phrase, a word, a syllable, a phoneme, or a smaller semiphoneme. The shorter the time scale of the modeling unit, the stronger its trainability and generalizability; and the longer the time scale of the unit, the stronger its accuracy.

[0051] In this example, in order to realize a mixed Chinese and English speech recognition system, the acoustic model needs to be able to learn and recognize the primary language and the auxiliary language, so it is necessary to model the primary language and the auxiliary language at the same time, that is, to model Chinese and English at the same time. At the same time, in order to avoid problems in the related art, the modeling unit can include primary language characters and auxiliary language subwords. Specifically, the primary language characters can be Chinese characters, and specifically, the commonly used 3000 to 8000 Chinese characters can be used as the modeling unit of Chinese.

[0052] In order to realize a mixed Chinese and English speech recognition system, the acoustic model needs to meet the requirements of learning both Chinese and English. For English, letters or words can be used as modeling units in related technologies. However, if letters are used as modeling units, the difference in speech duration between them and the modeling units of Chinese characters is large, and because the granularity of letters is small, the sequence during decoding is too long, thereby increasing the decoding time; and if words are used as modeling units, the number of words in English is huge. If modeling is performed one by one, it will lead to data sparsity problems; if only a part of commonly used words are used for modeling, it will face the problem of open vocabulary.

[0053] In order to solve the above-mentioned problems caused by using letters and words as modeling units, in the embodiments of the present disclosure, a subword is generated as a modeling unit of English, that is, an auxiliary language subword (English subword) is generated as a modeling unit of the auxiliary language. A subword refers to an English unit between a letter and a word. Specifically, an auxiliary language subword can be generated using a BPE (Byte pair encoding, data compression) algorithm, or a WordPiece algorithm and a UnigramLanguage Model algorithm can be used to generate English subwords. In the embodiments of the present disclosure, the generation of English subwords using the BPE algorithm is used as an example for explanation.

[0054] The steps of generating English subwords include: step 1, splitting the training corpus according to the minimum granularity to obtain a character sequence; step 2, counting the number of occurrences of each continuous character pair in the character sequence, merging the character pairs with the highest number of occurrences to generate a new character sequence, until the end condition is met, so as to determine the English subword. Among them, the minimum granularity refers to the smallest character, such as a letter. The split character sequence refers to the sequence composed of each character. The continuous character pair refers to the character pair composed of all adjacent characters in the character sequence. The end condition is used to stop the process of determining the English subword, and the end condition can be determined according to the number of occurrences or the type of subword, and is not specifically limited here.

[0055] Figure 3 A flowchart for determining English subwords is schematically shown in FIG. Figure 3 As shown, it mainly includes the following steps:

[0056] In step S310, the training corpus is split according to the smallest characters to obtain character sequences, that is, all sentences are split into character sequences with the smallest granularity.

[0057] In step S320, the frequency of occurrence of the character pairs is counted.

[0058] In step S330, high-frequency character pairs are merged. That is, by counting the occurrence frequency of each continuous character pair, high-frequency character pairs are selected and merged together as a whole, as a new subword, to obtain a new character sequence, and then the process returns to step S320.

[0059] In step S340, the process of merging high-frequency character pairs is repeated until the frequency of the next highest-frequency character pair is 1 or reaches the set number of symbols (for example, the number of symbols is greater than the set threshold of the number of symbols), and the whole process ends. The symbol set at this time is the final BPE unit set, thereby obtaining all auxiliary language subwords, that is, English subwords.

[0060] refer to Figure 4 As shown in , assuming that the training corpus is three words "hello", "low", and "lower", first split the words in each line according to the smallest letter granularity, and then merge the adjacent characters in pairs, and count the frequency of the merged subwords. At this time, the subword "lo" has a maximum frequency of 3 times, so "l" and "o" in the split corpus are merged as a unit "lo". Repeat the operation of merging adjacent units in pairs and counting the frequency of the merged subwords. This time, the subword with the highest frequency is "low", so "lo" and "w" in the corpus are merged as a unit "low". Merge in pairs again and count the frequency of the merged subwords. At this time, the highest frequency of all subwords is 1, and the end condition is met. Through the above operations, the final BPE unit is "h, e, l, lo, low, r", and the results of the split of the words in the training corpus have also been obtained. It should be noted that in actual use, the number of subwords contained in the training corpus is relatively large. For ease of operation, a target number of subwords can be set. When the number of subwords in the training corpus reaches the target number of subwords, the calculation is stopped.

[0061] The technical solution in the embodiment of the present disclosure uses the BPE algorithm to generate new English modeling units, which can improve the accuracy of the modeling units and facilitate model construction.

[0062] After the Chinese characters and the English subwords are determined, an acoustic model may be trained based on the Chinese characters and the English subwords. Figure 5 The specific flow chart for determining the acoustic model is schematically shown, refer to Figure 5 As shown in , it mainly includes the following steps:

[0063] In step S510, a training corpus is obtained, and feature extraction is performed on the training corpus to determine reference speech features of the training corpus.

[0064] In this step, the training corpus refers to the speech used to train the model. The training corpus can be in the same form as the speech to be recognized. The training corpus can also be a mixture of Chinese and English speech, and can be inter-sentence mixture or intra-sentence mixture. Here, intra-sentence mixture is used as an example for explanation.

[0065] Specifically, the following steps may be included: Step 1, frame and window the training corpus to obtain multiple speech frames; Step 3, transform the multiple speech frames to obtain the reference speech features. Among them, the original waveform of the training corpus can be transformed into small segments of audio as multiple speech frames through frame and window operations, and the multiple speech frames are further transformed to obtain the reference speech features of the training corpus. The transformation processing here includes: obtaining a transformed signal after the speech frame is processed by fast Fourier transform; obtaining a filtered signal after the transformed signal is passed through a Mel filter; obtaining a target signal after taking the logarithm of the filtered signal, and using the pre-preset dimensional data in the target signal as the Fbank feature of the training corpus, that is, the reference speech feature, and then inputting it into the acoustic model to realize the training of the acoustic model. The preset dimensional data can be determined according to actual needs, for example, it can be 80 dimensions.

[0066] In step S520, the acoustic neural network model is trained in combination with the reference speech features and the modeling unit to obtain an acoustic model.

[0067] In this step, the construction unit can be used as the real label corresponding to the reference speech feature, and the reference speech feature and the construction unit can be used to train the acoustic model. The acoustic model is used to determine the label of the speech. The acoustic model can be an acoustic neural network model or other network models. Here, the acoustic neural network model is used as an example for explanation.

[0068] Figure 6 The flowchart of model training is shown schematically in Figure 6 As shown in , it mainly includes the following steps:

[0069] In step S610, the reference speech feature is input into the acoustic neural network model to determine the probability distribution of the prediction label corresponding to the reference speech feature;

[0070] In step S620, the predicted label is compared with the real label represented by the modeling unit, and a loss function is determined according to the distance between the predicted label and the real label;

[0071] In step S630, the model parameters of the acoustic neural network model are adjusted according to the loss function until the loss function is minimized to obtain the acoustic model.

[0072] Among them, the input of the acoustic neural network model is the Fbank feature (reference speech feature) at the frame level, and the output is the probability distribution of the modeling unit at the frame level. The goal of model training is to minimize the loss function. The gradient of the network model is thus calculated to obtain the direction and value of updating the model network parameters. Training the acoustic model requires two processes: one is the forward process, that is, the input reference speech features are calculated with the network parameters to obtain the probability distribution of the predicted label; the other is the reverse process, which compares the obtained predicted label with the true label and calculates the distance between the two. The distance here can be used to represent the loss function, specifically the CTC loss function.

[0073] Further references Figure 7 As shown in , the entire training process is as follows: the reference speech features obtained in the feature extraction stage are input into the speech neural network model to perform predictive processing on them, and the predicted labels of the reference speech features are obtained; the distance between the predicted labels and the true labels (modeling units) is further calculated to determine the loss function, and the model parameters of the acoustic neural network model are adjusted with the goal of minimizing the loss function. After continuous iteration until the value of the loss function no longer decreases, the model can be considered to have converged, and the entire model training process is ended, thereby obtaining an acoustic model.

[0074] In the disclosed embodiment, a modeling method of mixing Chinese characters and English subwords is used, so that the acoustic model has the ability to learn both Chinese and English, and can recognize any form of speech mixed with Chinese and English, so that the acoustic model can recognize both inter-sentence mixing and intra-sentence mixing. The gap between the lengths of modeling units is narrowed, the limitation of not being able to fully model is avoided, the completeness and accuracy of the model is improved, the operating efficiency is improved, and the application scope of language recognition is expanded.

[0075] After generating the acoustic model, the speech features obtained by feature extraction of the speech to be recognized can be input into the acoustic model to obtain the probability distribution of the frame-level modeling units corresponding to the speech to be recognized, that is, the probability distribution of the Chinese characters and English subwords in the modeling units that the speech to be recognized belongs to.

[0076] In step S230, the speech to be recognized is recognized based on the language model constructed by the primary language characters and the auxiliary language subwords to determine a grammatical score of the speech to be recognized.

[0077] In the disclosed embodiment, the language model is used to calculate the probability of a sentence consisting of a symbol sequence. The symbol unit of the language model can be a letter or a subword, depending on the unit of sentence sequence segmentation in the training corpus. For a mixed Chinese and English language model, it is necessary to split the training corpus according to the symbol unit to obtain the split result, and then generate the language model according to the split unit. In order to ensure the accuracy of recognition, the symbol unit can be a modeling unit determined in the acoustic model. In this way, when splitting, the English words in the training corpus can be processed into BPE units (English subwords), and Chinese is also separated according to the Chinese character level (for example, separated according to each word) so that the modeling unit of the language model is consistent with the modeling unit of the acoustic model. Further, after the training corpus is processed according to the modeling unit, the split result is input into the statistical language model training tool to generate the language model. The language model training tool can be a kenlm tool, or a srilm tool or other tools can be used to generate a language model, which is not specifically limited here.

[0078] After determining the language model, the speech to be recognized can be input into the language model to determine the grammatical score of the speech to be recognized. Specifically, the grammatical score can be determined according to the evaluation index corresponding to the language model. The evaluation index can be an index of multiple dimensions, which can be specifically expressed by perplexity. For example, the evaluation index can include but is not limited to whether it is reasonable, fluent, correct, etc.

[0079] In step S240, the recognition result of the speech to be recognized is determined in combination with the probability distribution and the grammar score.

[0080] In the disclosed embodiment, the acoustic model and language model obtained in the above two processes are used in combination with a decoder to decode the speech to be recognized to obtain a recognition result. The decoder gives a possible candidate decoding path for each time step based on the probability distribution of the frame-level modeling unit given by the acoustic model, and then scores all possible candidate decoding paths in combination with the grammatical score given by the language model to obtain the final decoding result, which is the recognition result.

[0081] Figure 8 A flowchart for determining the recognition result is schematically shown in FIG. Figure 8 As shown, it mainly includes the following steps:

[0082] In step S810, the probability distribution of the frame-level modeling unit is decoded to determine a candidate decoding path of the speech to be recognized;

[0083] In step S820, the candidate decoding paths are evaluated in combination with the probability distribution and the grammatical score to determine a recognition result of the speech to be recognized.

[0084] Among them, each time step is each frame. There are several answers in the current time step. The next time step is based on the current time step, and each answer is used to expand a few more words, which can be called a decoding path. There may be multiple candidate decoding paths for the speech to be recognized at each time step. For each time step, each time step corresponds to the probability of all modeling units, that is, the probability on all modeling units. Take the target character from all modeling units, determine the probability score of the target character, and use the probability score as the acoustic model score. The target character can be the modeling unit arranged in the first N positions in descending order of probability.

[0085] Furthermore, evaluating the candidate decoding paths in combination with the grammatical scores refers to scoring each target character again in combination with the grammatical scores of the language model, and combining the acoustic model scores with the grammatical scores to obtain the path scores of the candidate decoding paths. Combining the acoustic model scores and the grammatical scores here may be performing an addition operation or other logical operations on the acoustic model scores and the grammatical scores. After obtaining the path scores of multiple candidate decoding paths, the candidate decoding path with the highest path score is used as the recognition result of the speech to be recognized at each time step, thereby obtaining the recognition result of the speech to be recognized.

[0086] refer to Fig. 9 As shown in , the speech recognition method is mainly completed by three parts: acoustic model, language model and decoder, and the language recognition system can be an end-to-end system. The process of recognizing a speech is as follows: step S901, extracting the features of the speech to be recognized to obtain speech features; step S902, inputting the speech features into the acoustic model to calculate the probability distribution of the modeling unit labels at the frame level of the speech to be recognized; step S903, inputting this probability distribution together with the language model into the decoder, decoding and obtaining the recognition result.

[0087] The technical solution in the disclosed embodiment uses Chinese characters and English subwords for mixed modeling. In the acoustic model training stage of speech recognition, Chinese and English are modeled at the same time, so that the same acoustic model can learn the pronunciation of Chinese and English at the same time; using English subwords for modeling increases the duration of English modeling units and narrows the gap between the duration of Chinese and English modeling units, so that the model can obtain better learning effects, and all words can be represented by English subwords, solving the problem of open vocabulary that cannot be solved when using words for modeling, avoiding the limitations of modeling, increasing the scope of application, and improving the accuracy of the model. On this basis, the acoustic model and language model constructed by English subwords can accurately and comprehensively recognize the mixed speech of Chinese and English, and improve the recognition accuracy.

[0088] In the embodiment of the present disclosure, a speech recognition device is also provided. Fig.10As shown in , the speech recognition device 1000 mainly includes the following modules:

[0089] The feature extraction module 1001 is used to obtain the speech to be recognized and perform feature extraction on the speech to be recognized to obtain speech features;

[0090] The probability determination module 1002 is used to predict the speech features by combining the acoustic model constructed by the modeling unit to determine the probability distribution of the modeling unit of the speech to be recognized; the modeling unit includes a main language character and an auxiliary language subword;

[0091] A grammar score determination module 1003, configured to perform recognition processing on the speech to be recognized based on the language model constructed by the primary language characters and the auxiliary language subwords, and determine a grammar score of the speech to be recognized;

[0092] The recognition result determination module 1004 is used to determine the recognition result of the speech to be recognized by combining the probability distribution and the grammar score.

[0093] In an exemplary embodiment of the present disclosure, the device also includes: a reference feature extraction module, which is used to obtain training corpus and perform feature extraction on the training corpus to determine the reference speech features of the training corpus; a model determination module, which is used to perform model training on the acoustic neural network model in combination with the reference speech features and the main language characters and the auxiliary language subwords in the modeling unit to obtain an acoustic model.

[0094] In an exemplary embodiment of the present disclosure, the reference feature extraction module includes: a speech frame determination module, which is used to perform frame segmentation and windowing operations on the training corpus to obtain multiple speech frames; and a transformation module, which is used to transform the multiple speech frames to obtain the reference speech features.

[0095] In an exemplary embodiment of the present disclosure, the model determination module includes: a prediction module, which is used to input the reference speech feature into the acoustic neural network model to determine the probability distribution of the predicted label corresponding to the reference speech feature; a loss function determination module, which is used to compare the predicted label with the real label represented by the modeling unit, and determine the loss function according to the distance between the predicted label and the real label; and a model training module, which is used to adjust the model parameters of the neural network model according to the loss function until the loss function is minimized to obtain the acoustic model.

[0096] In an exemplary embodiment of the present disclosure, the device also includes: a sequence determination module, which is used to split the training corpus according to the minimum granularity to obtain a character sequence; a subword determination module, which is used to count the number of occurrences of each consecutive character pair in the character sequence, merge the character pairs with the highest number of occurrences to generate a new character sequence, until the end condition is met, so as to determine the auxiliary language subword.

[0097] In an exemplary embodiment of the present disclosure, the device further includes: a language model determination module, configured to split the training corpus according to the main language characters and the auxiliary language subwords in the modeling unit to obtain split results, and generate the language model according to the split results.

[0098] In an exemplary embodiment of the present disclosure, the recognition result determination module includes: a path determination module, which is used to decode the probability distribution of the modeling unit and determine the candidate decoding path of the speech to be recognized; and an evaluation module, which is used to evaluate the candidate decoding path in combination with the grammatical score and determine the recognition result of the speech to be recognized.

[0099] In an exemplary embodiment of the present disclosure, the evaluation module is configured to: integrate the acoustic model score of the target character with the grammatical score of the target character to obtain the path score of the candidate decoding path; and take the candidate decoding path with the highest path score as the recognition result.

[0100] In addition, the specific details of each part of the above-mentioned speech recognition device have been described in detail in the implementation method of the speech recognition method. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0101] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0102] In addition, although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0103] In an embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0104] It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0105] Refer to the following Fig.11 1100 according to this embodiment of the present disclosure is described. Fig.11 The electronic device 1100 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0106] like Fig.11 As shown, the electronic device 1100 is in the form of a general computing device. The components of the electronic device 1100 may include, but are not limited to: the at least one processing unit 1110, the at least one storage unit 1120, a bus 1130 connecting different system components (including the storage unit 1120 and the processing unit 1110), and a display unit 1140.

[0107] The storage unit stores a program code, which can be executed by the processing unit 1110, so that the processing unit 1110 performs the steps described in the "Exemplary Method" section of the present specification according to various exemplary embodiments of the present disclosure. For example, the processing unit 1110 can perform the following steps: Figure 2 Follow the steps shown in .

[0108] The storage unit 1120 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 11201 and / or a cache storage unit 11202 , and may further include a read-only storage unit (ROM) 11203 .

[0109] The storage unit 1120 may also include a program / utility 11204 having a set (at least one) of program modules 11205, such program modules 11205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0110] The bus 1130 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, a graphics acceleration interface, a processing unit, or a local bus using any of a variety of bus architectures.

[0111] The electronic device 1100 may also communicate with one or more external devices 1200 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1100, and / or communicate with any device that enables the electronic device 1100 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1150. Furthermore, the electronic device 1100 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1160. As shown, the network adapter 1160 communicates with other modules of the electronic device 1100 via a bus 1130. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1100, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0112] In an embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present disclosure can also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present disclosure described in the above "Exemplary Method" section of the present specification.

[0113] According to the program product for implementing the above method in the embodiment of the present disclosure, it can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0114] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0115] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0116] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0117] Program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0118] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the embodiment of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0119] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0120] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A speech recognition method, characterized in that: include: Acquire a speech to be recognized, and perform feature extraction on the speech to be recognized to obtain speech features; The speech features are predicted by combining an acoustic model constructed by a modeling unit to determine a probability distribution of the modeling unit of the speech to be recognized; the modeling unit includes a primary language character and an auxiliary language subword; the probability distribution is used to represent the probability distribution of the speech to be recognized belonging to the primary language character and the auxiliary language subword in the modeling unit; Based on the evaluation index corresponding to the language model constructed by the primary language characters and the auxiliary language subwords, the speech to be recognized is recognized and processed to determine the grammatical score of the speech to be recognized; The probability distribution of the modeling unit is decoded to determine a candidate decoding path of the speech to be recognized, and the candidate decoding path is evaluated in combination with the grammatical score to determine a recognition result of the speech to be recognized.

2. The speech recognition method according to claim 1, characterized in that: The method further comprises: Acquire training corpus, perform feature extraction on the training corpus, and determine reference speech features of the training corpus; The acoustic neural network model is trained by combining the reference speech features and the main language characters and the auxiliary language subwords in the modeling unit to obtain an acoustic model.

3. The speech recognition method according to claim 2, characterized in that: The step of extracting features from the training corpus to determine reference speech features of the training corpus includes: Performing frame division and windowing operations on the training corpus to obtain multiple speech frames; The multiple speech frames are transformed to obtain the reference speech features.

4. The speech recognition method according to claim 2, characterized in that: The combining the reference speech feature and the modeling unit to perform model training on the acoustic neural network model to obtain the acoustic model includes: Inputting the reference speech feature into the acoustic neural network model to determine the probability distribution of the predicted label corresponding to the reference speech feature; Comparing the predicted label with the real label represented by the modeling unit, and determining a loss function according to the distance between the predicted label and the real label; The model parameters of the neural network model are adjusted according to the loss function until the loss function is minimized to obtain the acoustic model.

5. The speech recognition method according to claim 1, characterized in that: The method further comprises: Split the training corpus into the smallest granularity to obtain character sequences; The number of occurrences of each continuous character pair in the character sequence is counted, and the character pairs with the highest number of occurrences are merged to generate a new character sequence, until an end condition is met, so as to determine the auxiliary language subword.

6. The speech recognition method according to claim 1, characterized in that: The method further comprises: The training corpus is split according to the main language characters and the auxiliary language subwords in the modeling units to obtain split results, and the language model is generated according to the split results.

7. The speech recognition method according to claim 1, characterized in that: The step of evaluating the candidate decoding paths in combination with the grammatical scores to determine the recognition result of the speech to be recognized includes: Combining the acoustic model score of the target character with the grammatical score of the target character to obtain a path score of the candidate decoding path; The candidate decoding path with the highest path score is taken as the recognition result.

8. A speech recognition device, characterized in that: include: A feature extraction module is used to obtain the speech to be recognized and perform feature extraction on the speech to be recognized to obtain speech features; A probability determination module, used to predict the speech features by combining the acoustic model constructed by the modeling unit to determine the probability distribution of the modeling unit of the speech to be recognized; the modeling unit includes a main language character and an auxiliary language subword; the probability distribution is used to represent the probability distribution of the speech to be recognized belonging to the main language character and the auxiliary language subword in the modeling unit; A grammar score determination module, configured to perform recognition processing on the speech to be recognized based on the evaluation index corresponding to the language model constructed by the primary language characters and the auxiliary language subwords, and determine the grammar score of the speech to be recognized; The recognition result determination module is used to decode the probability distribution of the modeling unit, determine the candidate decoding path of the speech to be recognized, and evaluate the candidate decoding path in combination with the grammatical score to determine the recognition result of the speech to be recognized.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech recognition method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the speech recognition method described in any one of claims 1-7 by executing the executable instructions.

Images