Speech recognition methods, systems and terminals

Abstract

This disclosure provides a speech recognition method, system, and terminal. The method may include: receiving speech to be recognized; extracting and encoding acoustic features from the speech to be recognized; decoding the encoded acoustic features using a pinyin decoder to output pinyin; and decoding the encoded acoustic features using a Chinese character decoder to output Chinese characters, wherein the pinyin output by the pinyin decoder is embedded and superimposed on the encoded acoustic features input to the Chinese character decoder. The method further includes: evaluating the performance of the pinyin decoder; and, before the performance of the pinyin decoder is evaluated to be satisfactory, at least partially replacing the pinyin output by the pinyin decoder with the actual pinyin corresponding to the acoustic features.

Description

Technical Field

[0001] This invention mainly relates to speech recognition technology, and more specifically, to speech recognition methods, systems and terminals. Background Technology

[0002] Speech recognition technology, also known as Automatic Speech Recognition (ASR), has made significant progress with the development of artificial intelligence. End-to-end (e2e-ASR) speech recognition, a framework developed in recent years that differs from traditional speech recognition, is gradually becoming a trend. Traditional speech recognition generally consists of an acoustic model and a language model. The acoustic model is responsible for converting audio sequences into phoneme sequences, such as Chinese Pinyin and English phonetic symbols. The language model is responsible for converting these phoneme sequences into text sequences. The acoustic and language models do not need to be coupled during training and can be trained independently. The disadvantage of traditional speech models is that they require a pronunciation dictionary and phoneme annotations.

[0003] Unlike traditional speech recognition, end-to-end speech recognition systems directly convert audio sequences into text sequences. However, current end-to-end speech recognition systems that directly model text suffer from numerous errors in replacing non-near-phonetic characters. The more non-near-phonetic character errors a speech recognition system has, the worse its acoustic performance and the worse the user experience. Summary of the Invention

[0004] To address the aforementioned technical problems, this disclosure provides a speech recognition method and system. Compared to existing Chinese end-to-end speech recognition systems, the system of this disclosure adds a pinyin decoder in addition to the encoder and Chinese character decoder, and superimposes the output of the pinyin decoder as a feature onto the input of the Chinese character decoder, which can effectively improve the error of non-near-phonetic character replacement.

[0005] In one aspect of this disclosure, a speech recognition method is provided, comprising: receiving speech to be recognized; extracting and encoding acoustic features of the speech to be recognized; decoding the encoded acoustic features using a pinyin decoder to output pinyin; and decoding the encoded acoustic features using a Chinese character decoder to output Chinese characters, wherein the pinyin output by the pinyin decoder is embedded and superimposed on the encoded acoustic features input to the Chinese character decoder, wherein the method further comprises: evaluating the performance of the pinyin decoder; and, before the performance of the pinyin decoder is evaluated to be satisfactory, at least partially replacing the pinyin output by the pinyin decoder with the actual pinyin corresponding to the acoustic features.

[0006] According to a further embodiment of this disclosure, the Pinyin decoder and the Chinese character decoder are trained end-to-end as a whole.

[0007] According to a further embodiment of this disclosure, evaluating the performance of the pinyin decoder further includes: comparing the pinyin obtained from the dictionary with the corresponding output of the trained pinyin decoder; and if the pinyin obtained from the dictionary is ranked within the top threshold number of candidate results with the highest probability among the multiple candidate results output by the pinyin decoder, then the performance of the pinyin decoder is evaluated as meeting the requirements.

[0008] According to a further embodiment of this disclosure, using the real pinyin corresponding to the acoustic features instead of the pinyin output by the pinyin decoder further includes: performing alignment on the audio-text pairs used for training to delimit the audio character by character and align it with the corresponding Chinese character in the text; obtaining the real pinyin corresponding to the Chinese character in the text by looking up a dictionary; and embedding the obtained real pinyin onto the corresponding encoded acoustic features input to the Chinese character decoder.

[0009] According to a further embodiment of this disclosure, the method further includes: after the performance of the pinyin decoder meets the requirements, replacing some of the pinyin obtained from the dictionary with pinyin output by the pinyin decoder when training the Chinese character decoder.

[0010] According to a further embodiment of this disclosure, embedding the pinyin onto the encoded acoustic features input to the Chinese character decoder further includes: converting the pinyin into one-hot encoding; and superimposing the one-hot encoding corresponding to the Chinese character onto the encoded acoustic features input to the Chinese character decoder.

[0011] According to a further embodiment of this disclosure, decoding the encoded acoustic features using a Chinese character decoder to output Chinese characters further includes: performing probability smoothing on similar pronunciations in the output of the Chinese character decoder.

[0012] According to a further embodiment of this disclosure, the pinyin is further divided into initials and finals.

[0013] In another aspect of the invention, a speech recognition system is provided, comprising: a receiving module configured to receive speech to be recognized; a feature extraction module configured to extract and encode acoustic features from the speech to be recognized; a pinyin decoder configured to decode the encoded acoustic features to output pinyin, wherein, before the performance of the pinyin decoder is evaluated to meet requirements, at least partially using the actual pinyin corresponding to the acoustic features to replace the pinyin output by the pinyin decoder; and a Chinese character decoder configured to decode the encoded acoustic features to output Chinese characters, wherein the pinyin output by the pinyin decoder is embedded and superimposed on the encoded acoustic features input to the Chinese character decoder.

[0014] According to a further embodiment of this disclosure, the Pinyin decoder and the Chinese character decoder are trained end-to-end as a whole.

[0015] According to a further embodiment of this disclosure, embedding the pinyin further includes: converting the pinyin into one-hot encoding; and superimposing the one-hot encoding corresponding to the Chinese character onto the input of the Chinese character decoder.

[0016] According to a further embodiment of this disclosure, the output of the Chinese character decoder is probabilistically smoothed for similar pronunciations.

[0017] According to a further embodiment of this disclosure, the pinyin is further divided into initials and finals.

[0018] In another aspect of the invention, a terminal is provided, comprising: a voice recognition system as described in this disclosure.

[0019] This overview is provided to introduce, in a simplified form, some of the concepts further described in the detailed description below. This overview is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0020] The above-described invention and the following detailed description will be better understood when read in conjunction with the accompanying drawings. It should be noted that the drawings are merely examples of the claimed invention. In the drawings, the same reference numerals represent the same or similar elements.

[0021] Figure 1 An example construction process of a speech recognition system according to an embodiment of the present disclosure is shown;

[0022] Figure 2 This is a flowchart illustrating a speech recognition method according to an embodiment of the present disclosure;

[0023] Figure 3 This is an example block diagram illustrating a speech recognition system according to an embodiment of the present disclosure;

[0024] Figure 4 This is an example block diagram illustrating a terminal according to an embodiment of the present disclosure. Detailed Implementation

[0025] To make the above-mentioned objects, features and advantages of this disclosure more apparent and understandable, the specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings.

[0026] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure. However, this disclosure may be practiced in other ways different from those described herein, and therefore is not limited to the specific embodiments disclosed below.

[0027] In recent years, with the rise of artificial intelligence, speech recognition technology has made significant breakthroughs in both theory and application, and has gradually been applied to various aspects of daily life, such as speech recognition dictation devices, intelligent customer service, and real-time captioning. In recent years, end-to-end (E2E) models have begun to be widely used in the field of speech recognition. Unlike traditional speech recognition systems that use pronunciation dictionaries, acoustic models, and language models, E2E speech recognition only separates the input speech features from the output text information. In the traditional speech recognition framework, the entire automatic speech recognition system consists of multiple modules, including acoustic models, pronunciation dictionaries, and language modeling. To avoid these complex modeling processes, end-to-end models directly connect the input (speech waveform or feature sequence) to the output (word or character sequence) using a single neural network, which performs the functions of all the original modules. The final result is directly optimized through training, greatly simplifying the modeling process for speech recognition tasks.

[0028] In a typical end-to-end speech recognition model structure, it is generally divided into three parts: a front-end feature extractor, a CTC (Connectionist Temporal Classification) branch, and an attention branch. The front-end feature extractor (also known as the encoder) is used to extract and encode features from the speech waveform, aiming to downsample in the time dimension to reduce the computational overhead. The CTC branch (also known as the CTC decoder) is usually a fully connected layer in structure. The input is the encoded features output by the encoder, and the output is the probability of each modeling unit (such as a word), which is constrained by the CTC loss. The attention branch (also known as the attention decoder) includes a series of models with the attention mechanism, such as Transformer, Conformer, etc., which can be arbitrarily selected and is usually constrained by the smoothed cross-entropy loss. Both the CTC branch and the attention branch are not required. For example, the recognition result can be directly given based on the probability output by the CTC branch itself. However, training the model with the two losses of these two branches together can achieve better results. For example, in the wenet speech recognition system, the attention decoder is used to reorder the results of the CTC decoder. In addition, some end-to-end speech recognition systems (such as wenet) have also proposed a framework to integrate the language model (LM) through WFST (weighted finite state transducer).

[0029] The recognition accuracy of a speech recognition system can be continuously improved through continuous iterative training, but it is also difficult to achieve 100%. Some misrecognitions are manifested as homophone or near-homophone substitution errors (for example, recognizing "非" as "飞"), and some misrecognitions are manifested as non-homophone or non-near-homophone substitution errors (for example, recognizing "理想" as "理解"). In the traditional speech recognition framework, it is usually divided into two parts: an acoustic model (AM) and a language model (LM). It can clearly attribute non-homophone or non-near-homophone substitution errors to the acoustic model and homophone or near-homophone substitution errors to the language model. By optimizing the acoustic model, the probability of non-homophone or non-near-homophone substitution errors can be effectively reduced. Different from traditional speech recognition technologies, the acoustic model is not used in the end-to-end speech recognition framework. Therefore, even with continuous training, the overall recognition accuracy of the system can only be slowly improved, and the probability of non-homophone or non-near-homophone errors cannot be effectively controlled.

[0030] To address this issue, this disclosure provides an improved speech recognition method and system. Existing Chinese end-to-end speech recognition systems typically model Chinese characters directly. There are approximately 6000 common Chinese characters, but only about 1300 corresponding pinyin characters. The relationship between Chinese characters and pinyin is many-to-one. When using deep neural networks for classification modeling, fewer output types make the model easier to train and improve its performance. Therefore, this disclosure adds a pinyin decoder to the system. The input acoustic features are first decoded using the pinyin decoder to obtain the pinyin. Then, the pinyin is embedded and superimposed onto the features as input to the Chinese character decoder. This effectively provides a more easily modelable pinyin feature for modeling complex Chinese characters. Since pinyin is a strong feature of Chinese characters, it can significantly reduce the probability of incorrect character prediction, especially for non-homophones or near-homophones (hereinafter collectively referred to as "non-near-homophones").

[0031] Figure 1 An example construction process for a speech recognition system according to an embodiment of the present disclosure is shown. Figure 1 As shown, a training corpus for speech recognition can first be prepared, which contains multiple Chinese audio-text pairs that can be used for training. The training corpus can be any open-source corpus, such as aishell1, aishell2, wenetspeech, etc., or a specific corpus can be prepared for a specific application scenario.

[0032] As is well known in the art, audio features in audio-text pairs can be extracted using an encoder. Depending on the requirements, examples that can be extracted include, but are not limited to, formant / envelope MFCC, pitch period / fine structure, and FBank features. After compression, transformation, normalization, and other processing, the sound waveform in the original audio is converted into an encoded acoustic feature.

[0033] In existing end-to-end speech recognition systems, the encoder output is directly input to the Chinese character decoder for recognition. For example, in Wenet, the Chinese character decoder is a CTC-based decoder, which outputs n candidate results, which are then reordered by an attention decoder to obtain the final recognition result. In the example disclosed herein, such as... Figure 1 As shown, while the encoder's output is provided to the Chinese character decoder, this output is also provided to an additional Pinyin decoder to recognize the acoustic features as Pinyin. Subsequently, the Pinyin output from the Pinyin decoder is also used as a feature of the corresponding Chinese character and superimposed with the corresponding acoustic features input to the Chinese character decoder, i.e., embedding. The Chinese character decoder then recognizes the embedded features and outputs the recognition result. Based on this framework, the Chinese character decoder is trained end-to-end until its loss function converges, thus obtaining the trained Chinese character decoder.

[0034] It's important to note that the Pinyin decoder and the Chinese character decoder can be trained separately, or they can be trained end-to-end as a whole through a connection. However, when training them as a whole, the Pinyin decoder's performance is initially poor, and its predictions are likely to be inaccurate. Therefore, directly superimposing the Pinyin predictions onto the Chinese character decoder's input for end-to-end training from the beginning may lead to slow training or failure to converge. For this reason, in the early stages of training, real Pinyin can be temporarily used to embed the input features of the Chinese character decoder instead of the Pinyin decoder's output.

[0035] More specifically, an aligner can be used to align the audio input to the Chinese character decoder. The audio-text pairs in the training corpus are typically sentences. The aligner can segment these sentences character by character, also known as demarcation, generating character boundary information. After segmentation, it can be determined which Chinese character in the corresponding sentence text each audio segment corresponds to. Based on the Chinese characters, a dictionary lookup can determine the actual pinyin of each character, thus associating the actual pinyin with each audio segment and embedding it onto the input features of the corresponding Chinese character decoder.

[0036] Through several generations of training, the performance of the Pinyin decoder may have significantly improved compared to the initial stage. The performance of the Pinyin decoder can be evaluated. If the performance requirements are met, the system can be switched to the Pinyin decoder's predicted Pinyin output, or the predicted Pinyin output can be used initially at a certain proportion before gradually transitioning to the later stage. Ultimately, after training is complete, the input features to be recognized can be extracted and encoded, first output as Pinyin by the Pinyin decoder, then superimposed with the feature encoding before being input into the Chinese character decoder to output the recognized Chinese character.

[0037] Figure 2 This is a flowchart illustrating a speech recognition method 200 according to an embodiment of the present disclosure. Figure 2 As shown, method 200 begins at step 202, receiving the speech to be recognized. The speech to be recognized can be audio containing Chinese characters received by an audio input device (or sound acquisition device). The audio can be spoken speech by a user or sound played by an audio device, or it can be an audio stream from a media file.

[0038] In step 204, acoustic features of the speech to be recognized are extracted and encoded. The acoustic features of the speech to be recognized can be extracted by using various common technical methods in the prior art. For example, feature extraction and encoding methods similar to those of the kaldi or wenet speech recognition systems can be used. For an end-to-end speech recognition system, the main features to be extracted may include FBank features and pitch features. The process of FBank feature extraction usually includes steps such as pre-emphasis, framing, windowing, FFT, squaring, Mel filtering, and logarithm taking. Finally, the Fbank features are obtained. Pitch is related to the fundamental frequency (F0) of the sound and reflects the pitch information, that is, the tone. Any pitch feature acquisition algorithm in the prior art can be used to extract the pitch features. These acoustic features are converted into digital codes through the corresponding extraction processes.

[0039] In step 206, a pinyin decoder is used to decode the encoded acoustic features to output pinyin. For example, when the encoding of the acoustic features of the audio corresponding to "ni hao" is input into the pinyin decoder, the pinyin decoder can output the recognition result of "ni3" + "hao3" through calculation.

[0040] In step 208, a Chinese character decoder is used to decode the encoded acoustic features to output Chinese characters, where the pinyin output by the pinyin decoder is superimposed on the encoded acoustic features input to the Chinese character decoder through embedding.

[0041] Most Chinese characters have only a unique pinyin, and a small number of Chinese characters have a limited number (usually two or three) of different pinyins. Chinese characters and pinyins have a many-to-one relationship. Therefore, from the perspective of machine learning and training, pinyin is undoubtedly a strong feature for Chinese characters. Therefore, it is very effective to be used as a feature for describing the target learning object (that is, Chinese characters), and it is very helpful to make the training process converge quickly, that is, to shorten the training time.

[0042] In order to superimpose pinyin as a feature with other acoustic features, the output pinyin can also be encoded. In an embodiment of the present disclosure, one-hot encoding can be used to encode the pinyin. One-hot encoding converts the pinyin into a string of binary codes, most of which are 0, and the only 1 represents the corresponding pinyin. This encoded string can be appended after the acoustic feature encoding of the sound segment corresponding to the Chinese character, and used as the input for the training and recognition of the Chinese character decoder, that is, the end-to-end input.

[0043] The pinyin decoder can be trained independently. For example, using pinyin as the modeling unit, a common audio-text corpus can be converted into an audio-pinyin corpus, and a pinyin decoder can be trained end-to-end. Alternatively, in one embodiment of this disclosure, the pinyin decoder and the Chinese character decoder can be trained end-to-end as a whole. This joint training of two cascaded decoders is common in the art. Cascading two separately trained decoders can easily lead to error propagation. Jointly training the two decoders allows them to learn together during training, requiring only joint constraints on the loss function, similar to the joint training of the CTC decoder and attention decoder in the Wenet speech recognition system.

[0044] As mentioned earlier, end-to-end training of the Pinyin decoder and Chinese character decoder as a whole differs from typical end-to-end training. In the initial training phase, the Pinyin decoder's prediction output error probability is relatively high. Therefore, real Pinyin can be used to replace the Pinyin decoder's output during the initial training phase. More specifically, the performance of the Pinyin decoder can be evaluated, and until its performance is deemed satisfactory, the real Pinyin corresponding to the received acoustic features can be used, at least partially, to replace the Pinyin output of the Pinyin decoder.

[0045] According to one embodiment of this disclosure, the training corpus undergoes acoustic feature extraction and encoding, and is then fed into a Pinyin decoder and a Chinese character decoder. Simultaneously, alignment is performed via an aligner. The purpose of alignment is to divide the corpus, which is organized by sentence, into audio segments defined by Chinese characters. After delimitation, the segments can be easily compared with the text in the original audio-text pair, and the text is also divided into segments corresponding to individual Chinese characters. In one example, this can be achieved by indexing the characters in the segments and the text for matching. Subsequently, the text can retrieve the corresponding real Pinyin by querying a Chinese character-Pinyin dictionary. The retrieved real Pinyin is then bound to the audio segments, the characters in the text, and subsequently to the acoustic features input into the decoder—that is, aligned. Thus, the real Pinyin can replace the output of the Pinyin decoder, and the real Pinyin can also be one-hot encoded and superimposed on the corresponding acoustic features as an embedding.

[0046] After a certain number of training generations, the system can gradually transition back to using the output of the Pinyin decoder. According to one embodiment of this disclosure, the timing of this transition may still depend on the performance of the Pinyin decoder. Therefore, the performance of the Pinyin decoder can be evaluated periodically. This evaluation can be performed every few training generations.

[0047] According to an embodiment of the present disclosure, the performance of the pinyin decoder can be evaluated by comparing the pinyin obtained from the dictionary with the corresponding output results of the trained pinyin decoder. If the ranking of the pinyin obtained from the dictionary is within the top threshold number of candidate results with the highest probabilities among the multiple candidate results output by the pinyin decoder, the performance of the pinyin decoder is evaluated as meeting the requirements. As an end-to-end structured pinyin decoder, with pinyin as the modeling unit, during the decoding process, the probabilities of all modeling units (i.e., pinyin) are calculated, and only the one with the highest probability can be output when outputting. For example, when evaluating the performance of the pinyin decoder, a segment of audio can be selected. Suppose the Chinese character contained in it is "今", the true pinyin obtained by looking up the dictionary is "jin1", the output result of the pinyin decoder is "jin1", and the probability results before output are "jin1: 0.956", "jing1: 0.012", "qin1: 0.003", "qing1: 0.0009",... Therefore, in this example, the true pinyin "jin1" ranks first in the output probabilities of the pinyin decoder, which can represent that the performance of the pinyin decoder is very good. In addition, the probability value of "jin1", 0.956, is very close to 1, which indicates that the pinyin decoder has a very strong recognition ability for the pronunciation corresponding to "今".

[0048] It can be envisioned that evaluating the performance of the pinyin decoder only by the recognition accuracy rate of a single Chinese character would be rather limited, and a more scientific and comprehensive evaluation method can be adopted. For example, a corpus containing a large amount of text (such as 100 characters or 1000 characters) can be used for testing to see what proportion of the true pinyin ranks first in the output of the pinyin decoder, what proportion ranks among the top three, and what the average value of the probability values of the pinyin decoder for giving the true pinyin is. It can be understood that the higher these proportions and average values are, the better the performance of the pinyin decoder is.

[0049] Corresponding thresholds can be set for the performance of the pinyin decoder. For example, the proportion of the true pinyin ranking among the top three in the output of the pinyin decoder is 95%. If this threshold is exceeded, it can be considered that the performance of the pinyin decoder meets the requirements. After the performance of the pinyin decoder meets the requirements, when training the Chinese character decoder, some of the pinyin obtained from the dictionary can be replaced by the pinyin output by the pinyin decoder. For example, 80% of the true pinyin can be replaced by the pinyin output by the pinyin decoder. This replacement ratio can be dynamically adjusted as needed, for example, depending on the specific results of the performance evaluation of the pinyin decoder and the subsequent training effects. For example, as the performance of the pinyin decoder gets better and better, the replacement ratio can be higher and higher until the predicted output of the pinyin decoder is completely used.

[0050] The output of the Chinese character decoder can be the final result of a single Chinese character or the probabilities (n-best) of multiple candidate results. For example, in a speech recognition system similar to wenet, the result of the Chinese character decoder is further provided to the attention decoder for reordering. At this time, the output of the Chinese character decoder is the n Chinese characters with the highest probabilities. According to an embodiment of the present disclosure, probability smoothing can be performed on the output result. Optionally, probability smoothing can be performed for similar pronunciations, which also helps to reduce non-close phonetic errors and can avoid the probability of Chinese characters with similar pronunciations being 0. As an example, probability smoothing can adopt the weighted sum method, such as calculating the weighted sum with pinyin embedding.

[0051] Return to step 208. After the above-trained Chinese character decoder receives the encoded acoustic features and superimposes the pinyin output by the pinyin decoder thereon, it performs recognition and outputs the Chinese character recognition result, such as "Hello".至此流程结束。

[0052] Figure 3 is an example block diagram showing a speech recognition system 300 according to an embodiment of the present disclosure. As Figure 3 shown, the speech recognition system 300 may include a receiving module 302 for receiving the speech to be recognized; a feature extraction module 304 for extracting acoustic features from the speech to be recognized and encoding them; a pinyin decoder 306 for decoding the encoded acoustic features to output pinyin, where at least part of the true pinyin corresponding to the acoustic features is used to replace the pinyin output by the pinyin decoder before the performance of the pinyin decoder is evaluated to meet the requirements; and a Chinese character decoder 308 for decoding the encoded acoustic features to output Chinese characters, where the pinyin output by the pinyin decoder 306 is superimposed on the encoded acoustic features input to the Chinese character decoder 308 through embedding.

[0053] According to an embodiment of the present disclosure, the pinyin decoder 306 and the Chinese character decoder 308 are trained end-to-end as a whole. For example, the pinyin decoder 306 and the Chinese character decoder 308 can adopt the system construction method and training method described above in combination with Figure 1 and Figure 2 and are trained end-to-end in a cascaded manner. As mentioned before, during the training process, the performance of the pinyin decoder can be evaluated. Before the performance meets the requirements, at least part of the true pinyin can be used to replace the output of the pinyin decoder, which helps to accelerate the training convergence speed. As an example, both the pinyin decoder 306 and the Chinese character decoder 308 can be constructed using the CTC mechanism, and the Chinese character decoder 308 can further incorporate the n-gram language model through WFST. As a non-limiting example, the system 300 can be trained and implemented by adding a pinyin decoder based on the architecture of the wenet speech recognition system with a language model LM.

[0054] According to an embodiment of the present disclosure, further embedding the pinyin may include: converting the pinyin into one-hot encoding; and superimposing the one-hot encoding corresponding to the Chinese character on the input of the Chinese character decoder 308. Probability smoothing may be performed for similar pronunciations in the output result of the Chinese character decoder.

[0055] In addition, it can be understood that although in the above-described examples, the pinyin decoder directly uses the pinyin as the modeling unit for modeling, the pinyin can also be further disassembled into initials and finals, that is, using the initials and finals as the modeling units for modeling. Correspondingly, the output of the pinyin decoder corresponding to "你好" will be "n, i3, h, ao3". Correspondingly, when replacing with the real pinyin during the training process, the real pinyin will also be split into initials and finals.

[0056] Figure 4 is an example block diagram showing a terminal 400 according to an embodiment of the present disclosure. As Figure 4 shown, the terminal 400 may include a speech recognition system 402, and the speech recognition system 402 may be the speech recognition system 300 as Figure 3 described. The terminal 400 may be a user terminal where sound collection, recognition, and output are integrated, such as a smart phone, a desktop computer, a laptop computer, a tablet computer, a smart wearable device, a smart voice recorder, an automatic translator, etc. Alternatively, the terminal 400 may also be a system where sound collection, recognition, and output are located at different places. For example, after the sound is collected by the user terminal, the sound file is transmitted to the server, and the speech recognition system on the server side performs speech recognition, and then the recognized text is sent back to the user terminal.

[0057] The various steps and modules of the speech recognition methods and systems described above can be implemented in hardware, software, or a combination thereof. If implemented in hardware, the various illustrative steps, modules, and circuits described in this disclosure can be implemented or executed using a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic components, hardware components, or any combination thereof. The general-purpose processor can be a processor, microprocessor, controller, microcontroller, or state machine, etc. If implemented in software, the various illustrative steps and modules described in this disclosure can be stored as one or more instructions or codes on a computer-readable medium or transmitted. Software modules implementing the various operations of this disclosure can reside in storage media such as RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disks, removable disks, CD-ROMs, cloud storage, etc. The storage medium can be coupled to a processor so that the processor can read and write information from / to the storage medium and execute corresponding program modules to implement the various steps of this disclosure. Moreover, software-based embodiments can be uploaded, downloaded, or remotely accessed through appropriate communication means. Such appropriate means of communication include, for example, the Internet, the World Wide Web, intranets, software applications, cables (including fiber optic cables), magnetic communication, electromagnetic communication (including RF, microwave and infrared communication), electronic communication, or other such means of communication.

[0058] The various embodiments disclosed herein are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. For system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant details can be found in the descriptions of the method embodiments.

[0059] It should also be noted that these embodiments may be described as processes depicted as flowcharts, flow diagrams, structure diagrams, or block diagrams. Although a flowchart may describe the operations as a sequential process, many of these operations can be executed in parallel or concurrently. Furthermore, the order of these operations can be rearranged.

[0060] The disclosed methods, apparatuses, and systems should not be limited in any way. Rather, this disclosure covers all novel and non-obvious features and aspects of the various disclosed embodiments (individually and in various combinations and sub-combinations of each other). The disclosed methods, apparatuses, and systems are not limited to any particular aspect or feature or combination thereof, and no disclosed embodiment is required to have any one or more specific advantages or to solve any particular or all technical problems.

[0061] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0062] The embodiments of this disclosure have been described above with reference to the accompanying drawings. However, this disclosure is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many modifications under the guidance of this disclosure without departing from the spirit and scope of the claims. All of these modifications fall within the scope of protection of this disclosure.

Claims

1. A speech recognition method, comprising: Receive the voice to be recognized; The acoustic features of the speech to be recognized are extracted and encoded. The encoded acoustic features are decoded using a pinyin decoder to output pinyin. as well as The encoded acoustic features are decoded using a Chinese character decoder to output Chinese characters, wherein the pinyin output by the pinyin decoder is embedded and superimposed on the encoded acoustic features input to the Chinese character decoder. The method further includes: The performance of the Pinyin decoder was evaluated; as well as Before the performance of the pinyin decoder is evaluated to be satisfactory, the pinyin output by the pinyin decoder is replaced at least partially with the actual pinyin corresponding to the acoustic features, wherein the actual pinyin refers to the pinyin corresponding to the Chinese characters obtained by looking up the dictionary.

2. The method as described in claim 1, wherein the pinyin decoder and the Chinese character decoder are trained end-to-end as a whole.

3. The method of claim 1, wherein evaluating the performance of the pinyin decoder further comprises: The pinyin obtained from the dictionary is compared with the corresponding output of the trained pinyin decoder; as well as If the pinyin obtained from the dictionary is ranked within the top threshold number of candidate results output by the pinyin decoder, then the performance of the pinyin decoder is evaluated as meeting the requirements.

4. The method of claim 1, wherein using the actual pinyin corresponding to the acoustic feature instead of the pinyin output by the pinyin decoder further comprises: Alignment is performed on the audio-text pairs used for training to delimit the audio characters one by one and align them with the corresponding Chinese characters in the text. Obtain the actual pinyin corresponding to the Chinese characters in the text by looking up the dictionary; as well as The obtained real pinyin is embedded and superimposed on the corresponding encoded acoustic features input to the Chinese character decoder.

5. The method of claim 1, wherein the method further comprises: After the performance of the Pinyin decoder meets the requirements, during the training of the Chinese character decoder, some of the Pinyin obtained from the dictionary is replaced with Pinyin output by the Pinyin decoder.

6. The method of claim 1, wherein embedding the pinyin onto the encoded acoustic features input to the Chinese character decoder further comprises: Convert Pinyin to one-hot encoding; as well as The one-hot encoding corresponding to the Chinese character is superimposed on the encoded acoustic features input to the Chinese character decoder.

7. The method of claim 1, wherein decoding the encoded acoustic features using a Chinese character decoder to output Chinese characters further comprises: In the output of the Chinese character decoder, probability smoothing is performed for similar pronunciations.

8. The method of claim 1, wherein the pinyin is further divided into initials and finals.

9. A speech recognition system, comprising: A receiving module, configured to receive the speech to be recognized; A feature extraction module is configured to extract and encode acoustic features of the speech to be recognized. A pinyin decoder configured to decode encoded acoustic features to output pinyin, wherein, before the performance of the pinyin decoder is evaluated to be satisfactory, at least partially the pinyin output by the pinyin decoder is replaced by the actual pinyin corresponding to the acoustic features, wherein the actual pinyin refers to the pinyin corresponding to the Chinese characters obtained by looking up a dictionary; as well as A Chinese character decoder, configured to decode encoded acoustic features to output Chinese characters, wherein the pinyin output by the pinyin decoder is embedded and superimposed on the encoded acoustic features input to the Chinese character decoder.

10. The speech recognition system of claim 9, wherein the pinyin decoder and the Chinese character decoder are trained end-to-end as a whole.

11. The speech recognition system of claim 9, wherein embedding the pinyin further comprises: Convert Pinyin to one-hot encoding; as well as The one-hot encoding corresponding to the Chinese character is superimposed on the input of the Chinese character decoder.

12. The speech recognition system of claim 9, wherein the output of the Chinese character decoder is subjected to probability smoothing for similar pronunciations.

13. The speech recognition system of claim 9, wherein the pinyin is further divided into initials and finals.

14. A terminal, comprising: The speech recognition system as described in any one of claims 9-13.

Images