Speech recognition method, system and terminal

Abstract

The present disclosure provides a multi-functional speech recognition method, system and terminal. The present disclosure constructs an end-to-end speech recognition system with pinyin initial and final consonants as modeling units, and adds a modeling unit probability output module, effectively improving non-close sound character replacement errors, and relative to existing end-to-end speech recognition systems, adding various functions such as end-to-end speech recognition system acoustic recognition performance evaluation and pronunciation standard degree evaluation. The method comprises: receiving a speech to be recognized; performing acoustic feature extraction and encoding on the speech to be recognized; decoding the encoded acoustic features using a Chinese character decoder, wherein the Chinese character decoder uses pinyin initial and final consonants as modeling units, and maps the encoded acoustic feature sequence to a Chinese character sequence through initial and final consonants; and outputting a speech recognition result.

Description

TECHNICAL FIELD

[0001] The present application relates to speech recognition technology, and more particularly, to a speech recognition method, system and terminal. BACKGROUND

[0002] Speech recognition technology, also known as automatic speech recognition (ASR), has made great progress with the development of artificial intelligence technology. End-to-end (e2e-asr) speech recognition is a framework developed in recent years that is different from traditional speech recognition and is gradually becoming a trend. Traditional speech recognition is generally divided into an acoustic model and a language model. The acoustic model is responsible for converting an audio sequence into a phoneme sequence, and common phonemes include Chinese pinyin, English phonetic symbols, etc. The language model is responsible for converting the phoneme sequence into a text sequence. The acoustic model and the language model do not need to be coupled during training and can be trained independently. The disadvantage of the traditional speech model is that it requires a pronunciation dictionary and phoneme annotation.

[0003] Unlike traditional speech recognition, an end-to-end speech recognition system directly converts an audio sequence into a text sequence. However, when the end-to-end speech recognition system makes a recognition error, especially a non-close phonetic character replacement error, it is difficult to determine the cause of the recognition error, and the acoustic recognition performance of the system cannot be easily evaluated like the traditional speech recognition system. SUMMARY

[0004] To solve the above technical problems, the present disclosure provides a multi-functional speech recognition method, system and terminal. Compared with the existing Chinese end-to-end speech recognition system that uses Chinese characters as modeling units, the present disclosure uses pinyin initials, finals and medials as modeling units to construct an end-to-end speech recognition system and adds a modeling unit probability output module to effectively improve non-close phonetic character replacement errors. Compared with the existing end-to-end speech recognition system, the end-to-end speech recognition system has improved acoustic recognition performance evaluation and pronunciation standard degree evaluation and other functions.

[0005] In one aspect of the present disclosure, a speech recognition method is provided, including: receiving a speech to be recognized; performing acoustic feature extraction and encoding on the speech to be recognized; decoding the encoded acoustic features using a Chinese character decoder, wherein the Chinese character decoder uses pinyin initials, finals and medials as modeling units, and maps the encoded acoustic feature sequence to a Chinese character sequence through the initials, finals and medials; and outputting a speech recognition result.

[0006] According to a further embodiment of the present disclosure, the Chinese character decoder is a CTC decoder, and wherein the mapping of the encoded acoustic feature sequence into a Chinese character sequence through a syllable mapping further comprises: the CTC decoder fusing a weighted finite state transducer (WFST) with a TLG decoding network, in which the modeling units of tokens T are syllables, the lexicon L is a mapping from Chinese characters to syllables, and the language model G is a Chinese character based N-gram language model.

[0007] According to a further embodiment of the present disclosure, the method further comprises: evaluating the acoustic recognition performance of the Chinese character decoder based on the modeling unit probability distribution generated by the Chinese character decoder during decoding.

[0008] According to a further embodiment of the present disclosure, evaluating the acoustic recognition performance of the Chinese character decoder based on the modeling unit probability distribution generated by the Chinese character decoder during decoding further comprises: inputting a test audio segment into the Chinese character decoder; obtaining the modeling unit probability distribution generated by the Chinese character decoder during decoding of the test audio segment; comparing the real syllable corresponding to the pronunciation contained in the test segment with the obtained modeling unit probability distribution; and determining the acoustic recognition performance evaluation result of the Chinese character decoder based on the ranking of the real syllable in the obtained modeling unit probability distribution.

[0009] According to a further embodiment of the present disclosure, in response to the acoustic recognition performance evaluation result of the Chinese character decoder satisfying a performance requirement, evaluating the pronunciation standard of the received speech.

[0010] According to a further embodiment of the present disclosure, evaluating the pronunciation standard of the received speech further comprises: determining the evaluation result of the pronunciation standard based on the maximum probability value in the modeling unit probability distribution generated by the Chinese character decoder during decoding of the received speech.

[0011] According to a further embodiment of the present disclosure, the method further comprises: outputting the text as a speech recognition result together with its corresponding syllable and probability value.

[0012] According to a further embodiment of the present disclosure, the method further comprises: reordering the plurality of candidate results using an attention decoder; and outputting a speech recognition result based on the reordered candidate results.

[0013] According to another aspect of the present disclosure, there is provided a speech recognition system, comprising: a receiving module configured to receive speech to be recognized; a feature extraction module configured to perform acoustic feature extraction on the speech to be recognized and encode the acoustic features; a Chinese character decoder configured to decode the encoded acoustic features, wherein the Chinese character decoder models with Pinyin syllables as modeling units, and maps a sequence of encoded acoustic features to a sequence of Chinese characters via a syllable mapping; and a result output module configured to output a speech recognition result.

[0014] According to a further embodiment of the present disclosure, the Chinese character decoder is a CTC decoder that fuses a TLG decoding network via a weighted finite state transducer (WFST), in which the modeling units of tokens T are Pinyin syllables, the lexicon L is a mapping from Chinese characters to Pinyin syllables, and the language model G is a Chinese character-based N-gram language model.

[0015] According to a further embodiment of the present disclosure, the speech recognition system further comprises a modeling unit probability output module configured to output a modeling unit probability distribution generated in a decoding process of the Chinese character decoder.

[0016] According to a further embodiment of the present disclosure, the speech recognition system further comprises an acoustic recognition performance evaluation module configured to evaluate an acoustic recognition performance of the Chinese character decoder based on the modeling unit probability distribution output by the modeling unit probability output module.

[0017] According to a further embodiment of the present disclosure, evaluating the acoustic recognition performance of the Chinese character decoder further comprises: obtaining, from the modeling unit probability output module, a modeling unit probability distribution generated by the Chinese character decoder in a decoding process of a test audio segment; comparing a real Pinyin corresponding to a pronunciation contained in the test segment with the obtained modeling unit probability distribution; and determining an acoustic recognition performance evaluation result of the Chinese character decoder based on an ordering of the real Pinyin in the obtained modeling unit probability distribution.

[0018] According to a further embodiment of the present disclosure, the speech recognition system further comprises a pronunciation standard degree evaluation module configured to evaluate a pronunciation standard degree of the received speech.

[0019] According to a further embodiment of the present disclosure, evaluating the pronunciation standard degree of the received speech further comprises: obtaining, from the modeling unit probability output module, a modeling unit probability distribution generated by the Chinese character decoder in a decoding process of the received speech; and determining an evaluation result of the pronunciation standard degree based on a maximum probability value in the modeling unit probability distribution.

[0020] According to a further embodiment of the present disclosure, the result output module is further configured to output the text as the speech recognition result together with its corresponding pinyin and probability value.

[0021] According to a further embodiment of the present disclosure, the speech recognition system further comprises an attention decoder configured to reorder the plurality of candidate results output by the Chinese character decoder, and the result output module is further configured to output the speech recognition result based on the reordered candidate results.

[0022] In yet another aspect of the present disclosure, a terminal is provided, comprising a speech recognition system as described in the present disclosure.

[0023] This Summary is provided to introduce some concepts in a simplified form, which are further described below in the DETAILED DESCRIPTION. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF DRAWINGS

[0024] The above summary of the present disclosure and the following detailed description of the present disclosure will be better understood when read in conjunction with the appended drawings. It is to be noted that the figures are not drawn to scale and that the disclosed embodiments are to be considered merely illustrative and the application is not to be limited by the specific embodiments illustrated. Like reference numerals are used to refer to like elements throughout the various drawings.

[0025] Figure 1 is an example block diagram illustrating a speech recognition system according to an embodiment of the present disclosure;

[0026] Figure 2 is a flowchart illustrating a speech recognition method according to an embodiment of the present disclosure;

[0027] Figure 3 is an example block diagram illustrating a terminal according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] In order to make the above objectives, features and advantages of the present disclosure more clear and comprehensible, a detailed description of the specific embodiments of the present disclosure will be given below with reference to the accompanying drawings.

[0029] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure can be practiced without the specific details given below. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the present disclosure.

[0030] In recent years, with the rise of artificial intelligence, speech recognition technology has made great breakthroughs in theory and application, and has been gradually applied to various aspects of daily life, such as speech recognition dictation, intelligent customer service, real-time subtitles, etc. In recent years, end-to-end (E2E) models have been widely used in the field of speech recognition. Unlike the pronunciation dictionary, acoustic model and language model in traditional speech recognition systems, E2E speech recognition only divides into speech features at the input end and text information at the output end. In the traditional speech recognition framework, the entire automatic speech recognition system is composed of multiple modules, including an acoustic model, a pronunciation dictionary, and language modeling. In order to avoid these complex modeling processes, an end-to-end model directly connects an input end (speech waveform or feature sequence) to an output end (word or character sequence) with a neural network, which takes on the functions of all the original modules. By training to directly optimize the final result, the modeling process of the speech recognition task is greatly simplified.

[0031] In a typical end-to-end speech recognition model structure, it is generally divided into three parts: a front-end feature extractor, a CTC (Connectionist Temporal Classification) branch, and an attention branch. The front-end feature extractor (also known as the encoder) is used to extract and encode features from the speech waveform. Its purpose is to down-sample in the time dimension, thereby reducing computational overhead. The CTC branch (also known as the CTC decoder) is usually a fully connected layer in structure, with the encoded features output by the encoder as input and the probability of each modeling unit (e.g., word) as output, which is constrained by the CTC loss. The attention branch (also known as the attention decoder) contains a series of models with attention mechanisms, such as Transformer, Conformer, etc., which can be arbitrarily selected and are usually constrained by a smooth cross-entropy loss. The CTC branch and the attention branch are not all needed, for example, the recognition result can be directly given based on the probability output by the CTC branch itself, but training the model with both losses of the two branches can achieve better results, such as the reordering of the results of the CTC decoder using the attention decoder in the wenet speech recognition system.

[0032] The recognition accuracy of a speech recognition system can be continuously improved through iterative training, but it is difficult to achieve 100%. Some misrecognitions are represented by homophone or near-homophone substitution errors (e.g., recognizing "fei" as "fei"), and some misrecognitions are represented by non-homophone or near-homophone substitution errors (e.g., recognizing "lixiang" as "li jie"). In a traditional speech recognition framework, which can be divided into two parts, an acoustic model (AM) and a language model (LM), non-homophone or near-homophone substitution errors can be clearly attributed to the acoustic model, and homophone or near-homophone substitution errors can be clearly attributed to the language model. By optimizing the acoustic model, the probability of non-homophone or near-homophone substitution errors can be effectively reduced. Unlike traditional speech recognition technology, an acoustic model is not used in an end-to-end speech recognition framework, and some systems such as wenet incorporate a language model (LM) through WFST (weighted finite state transducer), which makes it impossible to determine whether a non-homophone error is caused by the language model even if it is a non-homophone error.

[0033] To solve this problem, the present disclosure provides a multifunctional speech recognition method and system. Unlike existing Chinese end-to-end speech recognition systems that directly model Chinese characters, the end-to-end speech recognition system of the present disclosure models with pinyin initials, finals, and rimes as modeling units, decodes in combination with an n-gram language model, and can more accurately recognize near-homophone words. At the same time, the probability output of the intermediate product pinyin in the end-to-end recognition process is increased, and the performance of the end-to-end system in acoustic recognition is evaluated based on the probability output. In the case that the performance of acoustic recognition can be effectively evaluated, the speech recognition system can be further used to evaluate the pronunciation standard of the received speech, thereby enriching the functions of the speech recognition system.

[0034] Figure 1 is an example block diagram illustrating a speech recognition system 100 according to an embodiment of the present disclosure. As shown in Figure 1 The speech recognition system 100 can include a receiving module 102 for receiving speech to be recognized, a feature extraction module 104 for performing acoustic feature extraction and encoding on the speech to be recognized, a Chinese character decoder 106 for decoding the encoded acoustic features, wherein the Chinese character decoder models with pinyin initials, finals, and rimes as modeling units, and maps the sequence of encoded acoustic features to a sequence of Chinese characters through initials, finals, and rimes, and a result output module 108 for outputting a speech recognition result.

[0035] The speech to be recognized received by the receiving module 102 can be audio containing Chinese characters received by an audio input device (or a sound collection device), which can be speech dictated by a user or sound played by an audio device, or an audio stream of a media file.

[0036] The feature extraction module 104 can employ various common technical manners in the prior art to extract the acoustic features of the speech to be recognized and encode them, for example, a feature extraction and encoding manner similar to the kaldi or wenet speech recognition system can be used. For an end-to-end speech recognition system, the main features extracted can include FBank features and pitch features. The process of FBank feature extraction generally includes pre-emphasis, framing, windowing, FFT, squaring, Mel filtering, logarithm taking, etc., and finally Fbank features are obtained. The pitch is related to the fundamental frequency (F0) of the sound, and reflects the information of the pitch, i.e., the tone. Any pitch feature collection algorithm in the prior art can be used to extract the pitch features. These acoustic features are converted into digital codes through the corresponding extraction process.

[0037] The Chinese character decoder 106 can be a model trained end to end in an end-to-end speech recognition system, for example, a CTC decoder. According to an embodiment of the present disclosure, the CTC decoder can fuse the TLG decoding network through a weighted finite state transducer WFST.

[0038] The wenet speech recognition system similarly fuses the TLG decoding network through a weighted finite state transducer WFST, and its modeling unit is a Chinese character, such as "ah", "I", etc.

[0039] T (Token, also known as Token) corresponds to the modeling unit during end-to-end training, and the output is the probability that the sound segment belongs to each modeling unit, i.e., the probability of each Chinese character, for example: P(ah) = 0.5, P(I) = 0.1,...

[0040] L is a dictionary, which is a mapping of Chinese characters and words, i.e., splitting a word into a sequence of modeling units, for example: we: I + we, China: China + country,...

[0041] G is a language model, for example, an n-gram language model, which represents the statistical probability of words and word in the language model (i.e., in the training corpus of the language model). Taking 3-gram as an example, the output is P(person | China, is) = 0.1, indicating that the probability of following "is" + "China" with the word "person" is 0.1.

[0042] By contrast, the modeling unit of the token T of the present disclosure is the pinyin initial and final, such as "a1", "o2", "e3", "i4", "b", "p", "m", "f", etc., the dictionary L is a mapping of Chinese characters to pinyin initials and finals, and the language model G is an N-gram language model based on Chinese characters.

[0043] More specifically, T takes in a frame-level CTC label sequence and outputs the probability of the sound segment belonging to each modeling unit, i.e., the probability of each sound syllable, such as: P(a1) = 0.5, P(o2) = 0.1, P(e3) = 0.1, and so on.

[0044] L is a lexicon. The lexicon is a mapping of Chinese characters to sound syllables, i.e., splitting Chinese characters into a sequence of modeling units, for example: w + o3 for "I", m + en2 for "us", and so on. The input of L is a modeling unit, and the output is a Chinese character.

[0045] G is a language model, for example, an n-gram language model, which represents the statistical probability of words and words in the language model (i.e., in the training corpus of the language model). Taking 3-gram as an example, the output is P(person | country, China) = 0.1, indicating that the probability of "person" following "China" + "country" is 0.1.

[0046] Through the construction of the above decoding graph, the mapping of the sound syllable sequence to the Chinese character sequence is realized. According to an embodiment of the present disclosure, a commonly used tool for constructing a decoding graph can be used, such as arpa2fst, fstdeterminizestar, fsttablecompose, fstminimizeencoded, and the like.

[0047] T, L, and G in the decoding graph can each be represented by a WFST, and when fused together, it can be represented as, for example:

[0048] S = T U min(det(L U G))

[0049] where U represents merging, min represents minimization, and det represents determinization. The input of the decoding graph is a frame-level CTC label sequence, and the output is a Chinese character sequence.

[0050] The decoder can use the decoder in Kaldi, such as LatticeFasterDecoder, to use the standard Viterbi beam search algorithm for decoding.

[0051] The CTC decoder built in the above manner can be trained in the existing pure end-to-end manner, for example, trained in a manner similar to wenet.

[0052] According to an embodiment of the present disclosure, a modeling unit probability output module 110 can be added in the speech recognition system 100 to output the modeling unit probability distribution generated in the decoding process of the CTC decoder. As described in the foregoing, the end-to-end of the CTC decoder is the sequence of acoustic features to the sequence of Chinese characters, and the intermediate process or intermediate product includes the initials and finals (which can be understood as the output of T) and the characters (which can be understood as the output of L), and the final result is given after G adjustment. These intermediate products are also expressed in probability. Taking the modeling unit (i.e., the initial and final) as an example, the output is the probability of the acoustic features belonging to different initials and finals, and these probabilities form a probability distribution. The probability can be expressed in different ways, which can be directly expressed by the calculated numerical value, or can be expressed by the processed probability value (for example, after probability smoothing). These probability values can be obtained and output by the modeling unit probability output module 110.

[0053] According to an embodiment of the present disclosure, the speech recognition system 100 can further include an acoustic recognition performance evaluation module 112 for evaluating the acoustic recognition performance of the Chinese character decoder 106 based on the modeling unit probability distribution output by the modeling unit probability output module 110. In one example, the evaluation can further include: obtaining the modeling unit probability distribution generated by the Chinese character decoder 106 in the decoding process of the test audio segment from the modeling unit probability output module 110, comparing the real pronunciation corresponding to the pronunciation contained in the test segment with the obtained modeling unit probability distribution, and determining the acoustic recognition performance evaluation result of the Chinese character decoder 106 based on the ordering of the real pronunciation in the obtained modeling unit probability distribution. As a non-limiting example, the modeling unit probability output module 110 can obtain the probability distribution of the initials and finals generated by the output end of T in the decoding graph of the Chinese character decoder 106. The modeling unit probability output module 110 can output the probability distribution of the initials and finals and compare it with the corresponding real pronunciation (split into initials and finals). Alternatively, the initials and finals output by the modeling unit probability output module 110 can be combined and expressed in the form of combined probability, such as “jin1:0.956”, “jing1:0.012”, “qin1:0.003”, “qing1:0.0009”, …… In this way, it is convenient to compare with the corresponding real pronunciation without the need to split the pronunciation into initials and finals. The comparison can be, for example, to see the ordering of the real pronunciation in the modeling unit output probability distribution, for example, if it is ranked first, it can represent that the Chinese character decoder has good performance in acoustic recognition. In addition, additionally or alternatively, the performance can also be judged based on the specific probability value, for example, in this example, the probability value of “jin1” is 0.956, which is close to 1, indicating that the Chinese character decoder has very strong recognition ability for the pronunciation corresponding to “jin”.

[0054] It can be conceived that the acoustic performance evaluation of the Chinese character decoder is limited if only the recognition accuracy of a single Chinese character is evaluated, and a more scientific and comprehensive evaluation method can be adopted. For example, a corpus containing a large piece of text (e.g., 100 characters or 1000 characters) can be used for testing to see what proportion of the true pinyin is ranked first in the output of the modeling unit of the Chinese character decoder, what proportion is ranked in the top three, or what is the average value of the probability value given by the pinyin decoder for the true pinyin, and the like. It can be understood that the higher the proportion and average value, the better the acoustic recognition performance of the Chinese character decoder.

[0055] In this way, the speech recognition system of the present disclosure can be easily evaluated for its acoustic recognition performance, although it adopts an end-to-end architecture and training method. In this case, if the acoustic recognition performance is good, for example, in the middle link, the pronunciation in the input speech can always be correctly recognized, and if the final output of the Chinese character decoder has recognition errors, especially non-phonetic character errors, the cause of the error is most likely from the language model, and accordingly the overall recognition accuracy of the speech recognition system can be effectively improved by optimizing the language model used.

[0056] According to an embodiment of the present disclosure, the speech recognition system 100 can further comprise a pronunciation standard evaluation module 114 for evaluating the pronunciation standard of the received speech. Pronunciation standard refers to the degree of deviation from the audio pronunciation in the corpus used when the speech recognition system 100 is trained. For example, if the training corpus is standard Mandarin, the pronunciation standard of the Mandarin can be evaluated. Similarly, if the training of the speech recognition system 100 uses a dialect corpus (e.g. Cantonese), the pronunciation standard of the received speech in Cantonese can be evaluated. It can be understood that one of the prerequisites for evaluating the pronunciation standard is that the acoustic recognition performance of the speech recognition system 100 itself is good. Another prerequisite is that the evaluation basis can be based on the results of acoustic recognition, rather than the recognition results of the speech recognition system. It can be understood that nowadays the speech recognition system trained by artificial intelligence technology, especially after the language model is added, many speech pronunciations that are not standard can still give correct recognition results by combining the context, so comparing the recognized text with the known test text cannot judge whether the pronunciation is standard. The speech recognition system of the present disclosure provides a tool that can effectively evaluate the pronunciation standard, because the present disclosure models the initial and final consonants, and can output the recognized initial and final consonants and their probability distribution through the modeling unit probability output module 110. In this way, it is only necessary to observe the probability prediction results of the audio to be evaluated by the modeling unit probability output module 110, if the prediction is correct and the probability value is close to 1, it indicates that the speaker's pronunciation is very standard. Similar to the evaluation of acoustic performance, the evaluation of pronunciation standard can also use a more comprehensive test and evaluation method.

[0057] According to an embodiment of the present disclosure, in order to facilitate observation during development and training, the result output module 108 can be further configured to output the text as the speech recognition result together with its corresponding pinyin and probability value. In this way, the developer can intuitively judge the current acoustic performance of the speech recognition system in training by comparing the input training text and the output pinyin recognition result and the corresponding probability value, and can also directly use this data as a reference basis for pronunciation standard evaluation. Of course, it can be understood that in the terminal product facing users, these parameters can be combined with the front-end interface design to become more friendly and beautiful.

[0058] The above described is an example architecture that can be used as a multi-functional speech recognition system to implement the present disclosure. Those skilled in the art can understand that any appropriate modification can be made on this basis, and other existing architectures can also be modularly cascaded or combined to obtain other additional functions or performance improvements. For example, as adopted in Wenet, the output of the Chinese character decoder can be interfaced with an attention decoder, which re-ranks the n-best candidate results output by the Chinese character decoder to take the re-ranked results as the final speech recognition results.

[0059] Figure 2 is a flowchart illustrating a speech recognition method 200 according to an embodiment of the present disclosure. As shown in Figure 2 the method 200 starts at step 202 by receiving speech to be recognized. At step 204, acoustic feature extraction is performed on the speech to be recognized and encoding is performed. At step 206, the encoded acoustic features are decoded using a Chinese character decoder, wherein the Chinese character decoder models with syllable as modeling unit, and maps the sequence of encoded acoustic features to a sequence of Chinese characters through syllable.

[0060] According to an embodiment of the present disclosure, the Chinese character decoder is a CTC decoder, wherein mapping the sequence of encoded acoustic features to a sequence of Chinese characters can further comprise that the CTC decoder fuses a TLG decoding network through a weighted finite state transducer (WFST), in which the modeling unit of the token T is a syllable, the lexicon L is a mapping from Chinese characters to syllables, and the language model G is a Chinese character based N-gram language model. The construction of the TLG decoding network and the CTC decoder has been described in conjunction with Figure 1 The generated multiple candidate results can be a probability distribution of Chinese characters, usually provided in the form of n-best.

[0061] At step 208, the speech recognition results are output. Usually, the final output speech recognition result is the Chinese character with the highest probability. In some applications, several highest ranked results can also be output as needed. Additionally, the probabilities of these results can be output together.

[0062] According to an embodiment of the present disclosure, the method 200 can further comprise evaluating the acoustic recognition performance of the Chinese character decoder based on the modeling unit probability distribution generated by the Chinese character decoder during the decoding process, which can include inputting a test audio segment into the Chinese character decoder, obtaining the modeling unit probability distribution generated by the Chinese character decoder during the decoding process of the test audio segment, comparing the real syllable corresponding to the pronunciation contained in the test segment with the obtained modeling unit probability distribution, and determining the acoustic recognition performance evaluation result of the Chinese character decoder based on the ranking of the real syllable in the obtained modeling unit probability distribution.

[0063] According to an embodiment of the disclosure, in a case where the acoustic recognition performance evaluation result of the Chinese character decoder satisfies the performance requirement, the method can support evaluation of the pronunciation standard degree of the received speech. The evaluation of the pronunciation standard degree can further include determining an evaluation result of the pronunciation standard degree based on a maximum probability value in the modeling unit probability distribution generated by the Chinese character decoder in the decoding process of the received speech. As a non-limiting example, the evaluation result can be graded, and each grade can correspond to a performance evaluation result range, for example, when the maximum probability value is between [0.95, 1], the evaluation result is “excellent”. Similarly, different evaluation grades and evaluation standards can be designed as needed.

[0064] According to an embodiment of the disclosure, the method can further include outputting the text as the speech recognition result together with its corresponding pinyin and probability value.

[0065] According to an embodiment of the disclosure, the method can further include reordering the plurality of candidate results using an attention decoder; and outputting the speech recognition result based on the reordered candidate results.

[0066] Figure 3 is an example block diagram illustrating a terminal 300 according to an embodiment of the disclosure. As shown in Figure 3 , the terminal 300 can include a speech recognition system 302, which can be the speech recognition system 100 as described in Figure 1 . The terminal 300 can be a user terminal in which sound collection, recognition, and output are integrated, such as a smartphone, a desktop computer, a notebook computer, a tablet computer, a smart wearable device, a smart voice recorder, an automatic translator, and the like. Alternatively, the terminal 300 can also be a system in which sound collection, recognition, and output are separated, such as a system in which sound is collected by a user terminal, a sound file is transmitted to a server, speech recognition is performed by a speech recognition system on the server side, and recognized text is transmitted back to the user terminal.

[0067] The various steps and modules of the speech recognition methods and systems described above can be implemented in hardware, software, or a combination thereof. If implemented in hardware, the various illustrative steps, modules, and circuits described in connection with the present disclosure can be implemented or performed with a general purpose processor, a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA) or other programmable logic component, hardware component, or any combination thereof designed to perform the functions described herein. The general purpose processor can be a microprocessor, a controller, a controller with a state machine, etc. If implemented in software, the various illustrative steps, modules, and circuits described in connection with the present disclosure can be stored in, or transmitted as, one or more instructions or code on a computer-readable medium. A computer-readable medium can be any medium that can be read by a computer, such as a RAM, a flash memory, a ROM, an EPROM, an EEPROM, a register, a hard disk, a removable disk, a CD-ROM, a cloud storage, etc. The computer-readable medium can be coupled to the processor such that the processor can read information from, and write information to, the computer-readable medium. In addition, the software-based embodiment can be uploaded, downloaded, or remotely accessed through a suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, an intranet, software applications, cable (including fiber optic cable), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such communication means.

[0068] Various embodiments of the present disclosure are described in a progressive manner, and the same or similar parts among the various embodiments can be referred to each other, and each embodiment focuses on the difference from other embodiments. For the system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiments.

[0069] It should also be noted that the embodiments can be described as a process which is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. Although a flowchart can describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently. In addition, the order of the operations can be re-arranged.

[0070] The disclosed methods, apparatus, and systems should not be limited in any way. Rather, the present disclosure encompasses all novel and non-obvious features and aspects of the various disclosed embodiments, alone and in various combinations and sub-combinations with one another. The disclosed methods, apparatus, and systems are not limited to any specific aspect or feature or combination of aspects and features, nor do any of the disclosed embodiments require the presence of any particular advantage or address any particular technical problem.

[0071] The above describes particular embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

[0072] The above describes particular embodiments of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order in which they are recited and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous or necessary.

Claims

1. A speech recognition method, comprising: Receive the voice to be recognized; The acoustic features of the speech to be recognized are extracted and encoded. The encoded acoustic features are decoded using a CTC decoder, mapping the encoded acoustic feature sequence to Chinese character sequences via initials and finals. The CTC decoder integrates a weighted finite-state converter (WFST) with a TLG decoding network. In the TLG decoding network, the modeling unit of the token T is a pinyin initial and final, the dictionary L is a mapping from Chinese characters to pinyin initials and finals, and the language model G is an N-gram language model based on Chinese characters. Output the speech recognition results.

2. The method as described in claim 1, characterized in that, The method further includes: The acoustic recognition performance of the CTC decoder is evaluated based on the probability distribution of the modeling units generated during the decoding process.

3. The method as described in claim 2, characterized in that, The evaluation of the acoustic recognition performance of the CTC decoder based on the probability distribution of the modeling units generated during the decoding process further includes: Input the test audio clip into the CTC decoder; Obtain the modeling unit probability distribution generated by the CTC decoder during the decoding process of the test audio segment; The actual pinyin corresponding to the pronunciation contained in the test audio segment is compared with the obtained probability distribution of the modeling unit; and Based on the ranking of the actual pinyin in the obtained modeling unit probability distribution, the acoustic recognition performance evaluation result of the CTC decoder is determined.

4. The method as described in claim 2 or 3, characterized in that, The method further includes: In response to the acoustic recognition performance evaluation result of the CTC decoder meeting the performance requirements, the pronunciation standard of the received speech is evaluated.

5. The method as described in claim 4, characterized in that, The evaluation of the pronunciation accuracy of the received speech further includes: The evaluation result of the pronunciation standard is determined based on the maximum probability value in the modeling unit probability distribution generated by the CTC decoder during the decoding process of the received speech.

6. The method as described in claim 1, characterized in that, The method further includes: The text, which is the result of speech recognition, along with its corresponding pinyin and probability value, will be output together.

7. The method as described in claim 1, characterized in that, The method further includes: The attention decoder is used to reorder multiple candidate results output by the CTC decoder; and The speech recognition result is output based on the reordered candidate results.

8. A speech recognition system, comprising: A receiving module, configured to receive the speech to be recognized; A feature extraction module is configured to extract and encode acoustic features of the speech to be recognized. A CTC decoder is configured to decode encoded acoustic features, mapping the encoded acoustic feature sequence to Chinese character sequences via initials and finals. The CTC decoder integrates a weighted finite-state converter (WFST) with a TLG decoding network. In the TLG decoding network, the modeling unit of the token T is a pinyin initial and final, the dictionary L is a mapping from Chinese characters to pinyin initials and finals, and the language model G is an N-gram language model based on Chinese characters. The result output module is configured to output the speech recognition result.

9. The speech recognition system as described in claim 8, characterized in that, Further includes: The modeling unit probability output module is used to output the modeling unit probability distribution generated during the decoding process of the CTC decoder.

10. The speech recognition system as described in claim 9, characterized in that, Further includes: The acoustic recognition performance evaluation module is used to evaluate the acoustic recognition performance of the CTC decoder based on the modeling unit probability distribution output by the modeling unit probability output module.

11. The speech recognition system as described in claim 9, characterized in that, The evaluation of the acoustic recognition performance of the CTC decoder further includes: The modeling unit probability distribution generated by the CTC decoder during the decoding process of the test audio segment is obtained from the modeling unit probability output module; The actual pinyin corresponding to the pronunciation contained in the test audio segment is compared with the obtained probability distribution of the modeling unit; and Based on the ranking of the actual pinyin in the obtained modeling unit probability distribution, the acoustic recognition performance evaluation result of the CTC decoder is determined.

12. The speech recognition system as described in claim 10, characterized in that, Further includes: The pronunciation standard assessment module is used to evaluate the pronunciation standard of the received speech.

13. The speech recognition system as described in claim 12, characterized in that, The evaluation of the pronunciation accuracy of the received speech further includes: The modeling unit probability distribution generated by the CTC decoder during the decoding process of the received speech is obtained from the modeling unit probability output module; and The evaluation result of the pronunciation standard is determined based on the maximum probability value in the probability distribution of the modeling unit.

14. The speech recognition system as described in claim 8, characterized in that, The result output module is further configured to: The text, which is the result of speech recognition, along with its corresponding pinyin and probability value, will be output together.

15. The speech recognition system as described in claim 8, characterized in that, Further includes: An attention decoder, configured to reorder multiple candidate results output by the CTC decoder. The result output module is further configured to output speech recognition results based on the reordered candidate results.

16. A terminal, comprising: The speech recognition system as described in any one of claims 8-15.

Images