Speech recognition method and apparatus

Abstract

The application relates to a speech recognition method and device. The speech recognition method comprises: inputting a phoneme to be recognized into a first multi-task neural network model by a terminal device; outputting a first prediction result by the first multi-task neural network model, the first prediction result comprising a character prediction result and a punctuation prediction result corresponding to the phoneme to be recognized; and displaying at least part of the first prediction result on a display screen of the terminal device. By constructing a neural network model for simultaneously predicting characters and punctuation corresponding to a phoneme, the characters and punctuation corresponding to the phoneme can be simultaneously output, and the neural network model has a small size and can be deployed on an end side.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a speech recognition method and apparatus. Background Technology

[0002] Speech recognition, also known as Automatic Speech Recognition (ASR), is a technology that converts speech into corresponding text using a computer. With the development of terminal device technology, speech recognition technology has become an important means of human-computer interaction and is applied in many different fields. Speech recognition technology is needed in many scenarios of electronic device applications, such as translation between different languages, voice interaction between smart electronic devices and users, and the conversion of real-time voice signals to text information in instant messaging software, etc. Summary of the Invention

[0003] The embodiments of this application propose a speech recognition method and apparatus.

[0004] In a first aspect, embodiments of this application provide a speech recognition method, the method comprising:

[0005] The terminal device inputs the phoneme to be identified into a first multi-task neural network model, and the terminal device outputs a first prediction result using the first multi-task neural network model. The first prediction result includes the character prediction result and punctuation prediction result corresponding to the phoneme to be identified. The terminal device displays at least a portion of the first prediction result on the display screen of the terminal device based on the first prediction result.

[0006] The first multi-task neural network model can be deployed on the edge (such as on a terminal device) or the cloud.

[0007] The speech recognition method of the embodiments of the present application can simultaneously predict the characters and the punctuation corresponding to the phonemes by constructing a neural network model (i.e., a first multi-task neural network model) for simultaneously predicting the characters and the punctuation corresponding to the phonemes. The converted phonemes (vectors) of the speech to be recognized are taken as the input of the neural network model, and the forward reasoning is performed, so that the characters and the punctuation corresponding to the phonemes can be simultaneously output, and the neural network model has a small size and can be deployed on the terminal side. In this document, "simultaneously", "simultaneous output", and the like can be understood as being able to obtain two kinds of information (such as character information corresponding to the phonemes and punctuation information corresponding to the phonemes) from the output of the neural network model, and do not limit the time sequence of obtaining the two kinds of information. In other words, "simultaneously" in this document does not necessarily mean the same time.

[0008] According to a first possible implementation manner of the first aspect, the first multi-task neural network model is obtained by training a second multi-task neural network model using training samples, and the training samples include: a sample sentence including characters, and the training samples further include: phonemes corresponding to the characters in the sample sentence and punctuation.

[0009] The second multi-task neural network model can be deployed on the terminal side (such as a terminal device) or the cloud side.

[0010] By constructing a neural network model (i.e., a second multi-task neural network model) for simultaneously predicting the characters and the punctuation corresponding to the phonemes, and constructing a training sample set to train the neural network model, a trained neural network model (i.e., a first multi-task neural network model) is obtained. In the training process, no word segmentation processing is required. The converted phonemes (vectors) of the speech to be recognized are taken as the input of the trained neural network model, and the forward reasoning is performed, so that the characters and the punctuation corresponding to the phonemes can be simultaneously output.

[0011] According to the second possible implementation manner of the first possible implementation manner of the first aspect, the length of the characters in the sample sentence is the same as the length of the phonemes and the length of the punctuation. In the process of constructing the training sample set, the length of the characters in the sample sentence is aligned with the length of the phonemes after phonetic transcription and the length of the punctuation. After the neural network model is trained using the training sample set constructed by the embodiments of the present application, the neural network model can simultaneously perform phoneme-to-character conversion and punctuation prediction, so that the predicted characters and punctuation results can be simultaneously output.

[0012] In a third possible implementation manner of the first aspect, the terminal device inputs the phonemes to be recognized into the first multi-task neural network model, and outputs the first prediction result by using the first multi-task neural network model, including: the terminal device inputs the phonemes to be recognized into the first multi-task neural network model in a loop, and outputs the first prediction result by using the first multi-task neural network model based on the length of the phonemes to be recognized currently input.

[0013] In the fourth possible implementation manner of the third possible implementation manner of the first aspect, the terminal device inputs the phonemes to be recognized into the first multi-task neural network model in a loop, and outputs the first prediction result by using the first multi-task neural network model based on the length of the phonemes to be recognized currently input, including:

[0014] Before all the phonemes to be recognized are input into the first multi-task neural network model, if the length of the phonemes currently input is less than the receptive field, the terminal device continues to input the next phonemes;

[0015] Before all the phonemes to be recognized are input into the first multi-task neural network model, if the length of the phonemes currently input is not less than the receptive field, the terminal device obtains the second prediction result of the first phoneme of the phonemes currently input according to the character and the punctuation of the phonemes currently input, and stores the second prediction result; the terminal device inputs the feature vector of the first phoneme, the phonemes currently input except the first phoneme, and the next phonemes to be recognized into the first multi-task neural network model;

[0016] When all the phonemes to be recognized are input into the first multi-task neural network model, the terminal device obtains the second prediction result of the phonemes currently input according to the character and the punctuation of the phonemes currently input;

[0017] If there is no stored second prediction result, the second prediction result of the phonemes currently input is taken as the first prediction result of the phonemes to be recognized;

[0018] If there is the stored second prediction result, the first prediction result of the phonemes to be recognized is obtained according to the second prediction result of the phonemes currently input and the stored second prediction result.

[0019] According to the speech recognition method in the above-mentioned embodiments of the present application, the phonemes to be recognized output by the acoustic model are input into the first multi-task neural network model in a loop, so that the prediction result of the phonemes to be recognized refers to the previous phonemes and the subsequent phonemes, and the prediction accuracy is improved.

[0020] In a fifth possible implementation manner of the first aspect, the first multi-task neural network model is a non-streaming network structure.

[0021] The first prediction result is output by using the first multi-task neural network model, including:

[0022] The first prediction result is output by using the first multi-task neural network model based on a relationship between a total length of the phoneme to be recognized and a phoneme length threshold.

[0023] In the sixth possible implementation manner of the fifth possible implementation manner of the first aspect, the first prediction result is output by using the first multi-task neural network model based on a relationship between a total length of the phoneme to be recognized and a phoneme length threshold, including:

[0024] If the total length of the phoneme to be recognized is less than the phoneme length threshold, the first prediction result is output by using the first multi-task neural network model based on all the phonemes to be recognized.

[0025] If the total length of the phoneme to be recognized is not less than the phoneme length threshold, before all the phonemes to be recognized are input into the first multi-task neural network model, if a length of a currently input phoneme is less than the phoneme length threshold, the terminal device continues to input a next phoneme; if the length of the currently input phoneme is not less than the phoneme length threshold, the terminal device obtains a second prediction result of a first phoneme of the currently input phoneme based on a character and a punctuation of the currently input phoneme and stores the second prediction result, and the terminal device inputs phonemes other than the first phoneme in the currently input phoneme and a next phoneme in the phonemes to be recognized into the first multi-task neural network model.

[0026] If the total length of the phoneme to be recognized is not less than the phoneme length threshold, when all the phonemes to be recognized are input into the first multi-task neural network model, a second prediction result of a currently input phoneme is obtained based on a character and a punctuation of the currently input phoneme.

[0027] If there is no stored second prediction result, the second prediction result of the currently input phoneme is taken as the first prediction result of the phoneme to be recognized.

[0028] If there is the stored second prediction result, the first prediction result of the phoneme to be recognized is obtained based on the second prediction result of the currently input phoneme and the stored second prediction result.

[0029] According to the speech recognition method of the above-mentioned embodiment of the present application, a non-stream network structure is adopted, and there is no need to re-input the phonemes whose results have been predicted into the network model. Compared with the stream network structure, the non-stream network structure does not need to cache the historical results that have been predicted, reduces the occupied memory space, can further reduce the size of the neural network model, and is easy to deploy on the edge side. Moreover, during the calculation process, there is no need to perform splicing, segmentation and other operations on the historical results and the phonemes input currently, which can accelerate the inference speed, and in long speech recognition, the effect of real-time output is remarkable.

[0030] In a second aspect, an embodiment of the present application provides a neural network model training method, and the method comprises:

[0031] A training sample is constructed, and the training sample comprises a sample sentence, wherein the sample sentence comprises characters, and the training sample further comprises phonemes corresponding to the characters in the sample sentence and punctuation marks.

[0032] The training sample is used to train a second multi-task neural network model to obtain a first multi-task neural network model; wherein the second multi-task neural network model and the first multi-task neural network model can output a first prediction result and display at least a part of the first prediction result, and the first prediction result comprises character prediction results and punctuation mark prediction results.

[0033] The neural network training method of the embodiment of the present application can obtain a trained neural network model by constructing a neural network model for simultaneously predicting characters corresponding to phonemes and punctuation marks and constructing a training sample set to train the neural network model. During the training process, no word segmentation processing is needed, the converted phonemes (vectors) of the speech to be recognized are used as the input of the trained neural network model, and the forward inference is performed, so that the characters corresponding to the phonemes and the punctuation marks can be simultaneously output, and the neural network model has a small size and can be deployed on the edge side.

[0034] According to the first possible implementation manner of the second aspect, the training sample can be constructed in the following manner.

[0035] According to the phonetic dictionary, the characters in the sample sentence are phonetized to obtain the phonemes corresponding to the characters, and the phonemes corresponding to the characters are aligned with the characters and the punctuation marks, and the length of the characters in the sample sentence is the same as the length of the phonemes and the length of the punctuation marks.

[0036] According to the first possible implementation manner of the second aspect, in a second possible implementation manner, the alignment of the phonemes corresponding to the characters with the characters and the punctuation marks comprises the following.

[0037] For a multi-sound Chinese character, an optional phoneme from the multiple phonemes corresponding to the multi-sound Chinese character is taken as the phoneme corresponding to the multi-sound Chinese character; that is, the phoneme corresponding to the multi-sound Chinese character in the aligned Chinese character is any one of the multiple phonemes corresponding to the multi-sound Chinese character;

[0038] For an English character, an alignment character is added in the character to align the length of the phoneme corresponding to the character; the aligned English character includes the alignment character, and the length of the aligned English character is the same as the length of the phoneme corresponding to the character; if there is no punctuation after the character, the punctuation corresponding to the character is set as blank, so that the length of the punctuation is aligned with the length of the character; for the character without punctuation before alignment, the punctuation after alignment is blank.

[0039] By aligning the length of the character in the sample sentence and the length of the phoneme after annotation, and the length of the punctuation in the process of constructing the training sample set, the training sample set constructed by the embodiment of the present application is used to train the neural network model, and the neural network model can simultaneously perform phoneme-to-character conversion and punctuation prediction, so as to simultaneously output the predicted character and punctuation results.

[0040] In a third possible implementation manner of the second aspect, training the second multi-task neural network model by using the training sample to obtain the first multi-task neural network model comprises:

[0041] inputting the training sample into the second multi-task neural network model to determine a character probability matrix and a punctuation probability matrix corresponding to the training sample;

[0042] calculating a character cross-entropy loss and a punctuation cross-entropy loss according to the character probability matrix and the punctuation probability matrix, respectively;

[0043] calculating a weighted cross-entropy loss according to the character cross-entropy loss, a first weight corresponding to the character cross-entropy loss, the punctuation cross-entropy loss, and a second weight corresponding to the punctuation cross-entropy loss;

[0044] adjusting the parameters of the second multi-task neural network model according to the weighted cross-entropy loss to obtain the trained first multi-task neural network model.

[0045] The training method of the multi-task neural network model can realize training of the tasks of character prediction and punctuation prediction at the same time. In addition, since the training sample set constructed includes multiple languages, the training method of the multi-task neural network model can also realize training of the task of multiple language recognition (prediction). The multi-task neural network model trained by the training method of the multi-task neural network model according to the embodiment of the present application can simultaneously perform prediction of multiple languages and punctuation, and the multi-task neural network model is smaller in size than a traditional acoustic model and can be deployed on an end side.

[0046] In a third aspect, embodiments of the present application provide a speech recognition device, the device comprising:

[0047] an input module configured to input phonemes to be recognized into a first multi-task neural network model;

[0048] an inference module configured to output a first prediction result from the first multi-task neural network model, the first prediction result comprising character prediction results and punctuation prediction results corresponding to the phonemes to be recognized;

[0049] a display module configured to display at least part of the first prediction result on a display screen of the terminal device according to the first prediction result.

[0050] The speech recognition device of the embodiments of the present application can simultaneously output characters and punctuation corresponding to phonemes by constructing a neural network model for simultaneously predicting characters and punctuation corresponding to phonemes, inputting converted phonemes (vectors) of speech to be recognized into the neural network model as input, and performing forward inference.

[0051] According to a first possible implementation manner of the third aspect, the first multi-task neural network model is obtained by training a second multi-task neural network model using training samples, and the training samples comprise: a sample sentence, the sample sentence comprising characters, and the training samples further comprising: phonemes corresponding to the characters in the sample sentence and punctuation.

[0052] The neural network model (i.e., the second multi-task neural network model) for simultaneously predicting characters and punctuation corresponding to phonemes is constructed, and a training sample set is constructed to train the neural network model to obtain a trained neural network model (i.e., the first multi-task neural network model). In the training process, no word segmentation processing is required. The converted phonemes (vectors) of speech to be recognized are input into the trained neural network model as input, and forward inference is performed to simultaneously output characters and punctuation corresponding to phonemes.

[0053] According to the first possible implementation manner of the third aspect, in a second possible implementation manner, the length of the characters in the sample sentence is the same as the length of the phonemes and the length of the punctuation.

[0054] In the process of constructing the training sample set, the length of the characters in the sample sentence is aligned with the length of the phonemes after annotation and the length of the punctuation. After the neural network model is trained using the training sample set constructed by the embodiments of the present application, the neural network model can simultaneously perform phoneme-to-character conversion and punctuation prediction, so as to simultaneously output the predicted character and punctuation results.

[0055] In a third possible implementation manner of the third aspect, the first multi-task neural network model is a streaming network structure, and the input module comprises a first input unit configured to input the phonemes to be recognized into the first multi-task neural network model in a loop; and the inference module comprises a first inference unit configured to output the first prediction result based on a length of a current input phoneme to be recognized by using the first multi-task neural network model. In this way, the prediction result of the phonemes to be recognized is referenced based on both the previous phonemes and the subsequent phonemes, and the prediction accuracy is improved.

[0056] In a fourth possible implementation manner of the third possible implementation manner of the third aspect, the first input unit is further configured to: before all the phonemes to be recognized are input into the first multi-task neural network model, if the length of the current input phoneme is less than the receptive field, the terminal device continues to input the next phoneme; and before all the phonemes to be recognized are input into the first multi-task neural network model, if the length of the current input phoneme is not less than the receptive field, the first inference unit is configured to obtain a second prediction result of a first phoneme of the current input phoneme based on the character and the punctuation of the current input phoneme, and store the second prediction result; and the first input unit is further configured to input a feature vector of the first phoneme, phonemes other than the first phoneme in the current input phoneme, and the next phoneme in the phonemes to be recognized into the first multi-task neural network model. The first inference unit is further configured to: when all the phonemes to be recognized are input into the first multi-task neural network model, obtain a second prediction result of the current input phoneme based on the character and the punctuation of the current input phoneme; if there is no stored second prediction result, take the second prediction result of the current input phoneme as the first prediction result of the phonemes to be recognized; and if there is a stored second prediction result, obtain the second prediction result of the phonemes to be recognized based on the second prediction result of the current input phoneme and the stored second prediction result.

[0057] According to the speech recognition apparatus in the above-mentioned embodiments, the phonemes to be recognized output by the acoustic model are input into the first multi-task neural network model in a loop, so that the prediction result of the phonemes to be recognized is referenced based on both the previous phonemes and the subsequent phonemes, and the prediction accuracy is improved.

[0058] In a fifth possible implementation manner of the third aspect, the first multi-task neural network model is a non-streaming network structure, and the inference module comprises a second inference unit configured to output the first prediction result based on a relationship between a total length of the phonemes to be recognized and a phoneme length threshold by using the first multi-task neural network model.

[0059] In a sixth possible implementation manner of the fifth possible implementation manner of the third aspect, the second inference unit is further configured to: if the total length of the phonemes to be recognized is less than the phoneme length threshold, output the first prediction result according to all the phonemes to be recognized by using the first multi-task neural network model;

[0060] If the total length of the phonemes to be recognized is not less than the phoneme length threshold, before all the phonemes to be recognized are input into the first multi-task neural network model: if the length of the currently input phoneme is less than the phoneme length threshold, the next phoneme is continuously input; if the length of the currently input phoneme is not less than the phoneme length threshold, the second prediction result of the first phoneme of the currently input phoneme is obtained according to the character and the punctuation of the currently input phoneme, and the second prediction result is stored, and the phonemes other than the first phoneme in the currently input phoneme and the next phoneme in the phonemes to be recognized are continuously input into the first multi-task neural network model;

[0061] If the total length of the phonemes to be recognized is not less than the phoneme length threshold, when all the phonemes to be recognized are input into the first multi-task neural network model, the second prediction result of the currently input phoneme is obtained according to the character and the punctuation of the currently input phoneme;

[0062] If there is no stored second prediction result, the second prediction result of the currently input phoneme is taken as the first prediction result of the phonemes to be recognized;

[0063] If there is a stored second prediction result, the first prediction result of the phonemes to be recognized is obtained according to the second prediction result of the currently input phoneme and the stored second prediction result.

[0064] According to the speech recognition device in the above-mentioned embodiments of the present application, a non-stream network structure is adopted, and the phonemes whose results have been predicted do not need to be re-input into the network model. Compared with a stream network structure, the non-stream network structure does not need to cache the historical results that have been predicted, reduces the occupied memory space, can further reduce the size of the neural network model, and is easy to deploy on the terminal side. Moreover, during the calculation process, the historical results and the currently input phonemes do not need to be spliced or cut, which can accelerate the inference speed and significantly realize the effect of real-time output in long speech recognition.

[0065] In a fourth aspect, an embodiment of the present application provides a neural network model training device, and the device comprises:

[0066] The construction module is configured to construct a training sample, and the training sample comprises: a sample sentence, the sample sentence comprising characters, and the training sample further comprising: phonemes corresponding to the characters in the sample sentence and punctuation;

[0067] The training module is configured to train the second multi-task neural network model to obtain a first multi-task neural network model by using the training samples; wherein the second multi-task neural network model and the first multi-task neural network model can output a first prediction result and display at least part of the first prediction result, and the first prediction result includes a character prediction result and a punctuation prediction result.

[0068] The neural network training apparatus of the embodiments of the present application can simultaneously predict characters and punctuation corresponding to phonemes by constructing a neural network model for simultaneously predicting characters and punctuation corresponding to phonemes, and constructing a training sample set to train the neural network model to obtain a trained neural network model. In the training process, no word segmentation processing is required. The converted phonemes (vectors) of the to-be-recognized speech are used as inputs of the trained neural network model, and forward reasoning is performed. The characters and punctuation corresponding to the phonemes can be simultaneously output, and the neural network model has a small size and can be deployed on an end side.

[0069] According to a first possible implementation manner of the fourth aspect, the constructing module comprises:

[0070] The alignment unit is configured to phonate characters in a sample sentence to obtain phonemes corresponding to the characters according to a phonetic dictionary, and perform alignment processing on the phonemes corresponding to the characters, the characters and punctuation. The length of the characters in the sample sentence is the same as the length of the phonemes and the length of the punctuation. According to the first possible implementation manner of the fourth aspect, in a second possible implementation manner, the alignment unit is further configured to:

[0071] For a multi-sound character in Chinese, any one of a plurality of phonemes corresponding to the multi-sound character is selected as the phoneme corresponding to the multi-sound character; the phoneme corresponding to the multi-sound character in the aligned Chinese is any one of the plurality of phonemes corresponding to the multi-sound character;

[0072] For an English character, an alignment character is added in the character to align the length of the alignment character with the phoneme corresponding to the character; the aligned English character includes the alignment character, and the length of the aligned English character is the same as the length of the phoneme corresponding to the English character;

[0073] If there is no punctuation after the character, the punctuation corresponding to the character is set as blank, so that the length of the punctuation is aligned with the length of the character; for the character without punctuation before alignment, the punctuation after alignment is blank.

[0074] In the process of constructing the training sample set, the length of the characters in the sample sentence is aligned with the length of the phonemes after phonation and the length of the punctuation. After the neural network model is trained by using the training sample set constructed by the embodiments of the present application, the neural network model can simultaneously perform phoneme-to-character conversion and punctuation prediction, so that the predicted character and punctuation results can be simultaneously output.

[0075] In a third possible implementation manner of the fourth aspect, the training module comprises:

[0076] a determination unit, configured to input a training sample into the second multi-task neural network model, and determine a character probability matrix and a punctuation probability matrix corresponding to the training sample;

[0077] a first calculation unit, configured to calculate a character cross-entropy loss and a punctuation cross-entropy loss according to the character probability matrix and the punctuation probability matrix, respectively;

[0078] a second calculation unit, configured to calculate a weighted cross-entropy loss according to the character cross-entropy loss, a first weight corresponding to the character cross-entropy loss, the punctuation cross-entropy loss, and a second weight corresponding to the punctuation cross-entropy loss;

[0079] an adjustment unit, configured to adjust parameters of the second multi-task neural network model according to the weighted cross-entropy loss, to obtain a trained first multi-task neural network model.

[0080] The training apparatus of the multi-task neural network model can train the tasks of character prediction and punctuation prediction simultaneously. In addition, since the training sample set constructed includes multiple languages, the training method of the multi-task neural network model can also train the task of recognizing (predicting) multiple languages. The multi-task neural network model trained by the training apparatus of the multi-task neural network model according to the embodiments can simultaneously predict multiple languages and punctuation, and the multi-task neural network model is smaller in size than a traditional acoustic model, and can be deployed on an end side.

[0081] In a fifth aspect, the embodiments of the present application provide a speech recognition apparatus, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the speech recognition method of the first aspect or one or more of the possible implementation manners of the first aspect when executing the instructions.

[0082] In a sixth aspect, the embodiments of the present application provide a neural network model training apparatus, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the neural network model training method of the second aspect or one or more of the possible implementation manners of the second aspect when executing the instructions.

[0083] In a seventh aspect, an embodiment of the present application provides a nonvolatile computer readable storage medium, having stored thereon computer program instructions, which when executed by a processor implement the speech recognition method of the first aspect or one or more of the multiple possible implementation manners of the first aspect, or implement the neural network model training method of the second aspect or one or more of the multiple possible implementation manners of the second aspect.

[0084] These and other aspects of the present application will become more fully understood from the following (multiple) embodiment descriptions. BRIEF DESCRIPTION OF DRAWINGS

[0085] The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the exemplary embodiments, features and aspects of the present application and, together with the description, further serve to explain the principles of the present application.

[0086] Figure 1 An application scenario of the speech recognition method according to an embodiment of the present application is shown.

[0087] Figure 2 A component structure schematic diagram of the speech recognition model training apparatus provided by an embodiment of the present application is shown.

[0088] Figure 3 A block diagram of part of the structure of the mobile phone provided by an embodiment of the present application is shown.

[0089] Figure 4 A software structure schematic diagram of the mobile phone 100 of an embodiment of the present application is shown.

[0090] Figure 5a A block diagram of the neural network model according to an embodiment of the present application is shown.

[0091] Figure 5b A schematic diagram of an encoder-decoder model according to an example of the present application is shown.

[0092] Figure 5c A schematic diagram of an encoder model according to an example of the present application is shown.

[0093] Figure 6 A schematic diagram of the process of constructing a training sample set according to an embodiment of the present application is shown.

[0094] Figure 7 An example of the process of constructing a training sample set according to an embodiment of the present application is shown.

[0095] Figure 8 A flowchart of the multi-task neural network model training method according to an embodiment of the present application is shown.

[0096] Figure 9aFIG. 1 shows a schematic diagram of an application scenario of terminal device side voice recognition according to an embodiment of the present application.

[0097] Figure 9b FIG. 2 shows a schematic diagram of a process of prior art voice recognition according to an example of the present application.

[0098] Figure 10 FIG. 3 shows a flowchart of a voice recognition method according to an embodiment of the present application.

[0099] Figure 11 FIG. 4 shows a flowchart of a voice recognition method according to an embodiment of the present application.

[0100] Figure 12 FIG. 5 shows a flowchart of a voice recognition method according to an embodiment of the present application.

[0101] Figure 13 FIG. 6 shows a flowchart of a voice recognition method according to an embodiment of the present application.

[0102] Figure 14 FIG. 7 shows a block diagram of a voice recognition apparatus according to an embodiment of the present application.

[0103] Figure 15 FIG. 8 shows a block diagram of a neural network model training apparatus according to an embodiment of the present application. DETAILED DESCRIPTION

[0104] Various exemplary embodiments, features, and aspects of the present application will be described herein below with reference to the accompanying drawings. Identical reference numerals in the drawings denote elements having the same or similar functions. Although various aspects of the embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0105] The word "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0106] In addition, for the purpose of convenience and brevity, detailed descriptions of well-known functions and structures incorporated herein can not be described in detail herein. It should be appreciated that the present application can be practiced with the exact details as

[0107] Traditional voice recognition implements phoneme-to-character conversion (Chinese refers to pinyin-to-Chinese character conversion) through a statistical-based N-Gram language model. This method requires a large model, generally GB level, which cannot be deployed on the terminal side.

[0108] The traditional punctuation prediction is performed after the speech recognition ends, especially in long speech recognition, and cannot output punctuation while outputting converted characters. In the related art, punctuation marks are constructed as part of a word to train a text and a dictionary file, and a language model is trained to output punctuation marks while outputting text. However, the acoustic model uses a trigram model, and needs to perform word segmentation on a sentence in the training process, and the acoustic model uses a Gaussian mixture model and a hidden Markov model to align phonemes, and the like, and the processing process is relatively complex, resulting in that the acoustic model also has a large model and cannot be deployed on an end side. Moreover, since the acoustic model is used for punctuation prediction, the punctuation cannot be adjusted according to the context, and the prediction accuracy is not high.

[0109] Therefore, the related speech recognition technology has the technical problems that the model cannot be deployed on an end side, and the prediction accuracy of the punctuation predicted by using the acoustic model is not high.

[0110] To solve the above technical problems, the present application provides a speech recognition method. Figure 1 The application scenario of the speech recognition method according to an embodiment of the present application is shown. As shown in Figure 1 The terminal device (including terminal device 10-1 and terminal device 10-2) is provided with a client of a speech recognition software, and a user can input a corresponding speech recognition statement through the set speech recognition software client. The chat client can also receive a corresponding speech recognition result, and display the received speech recognition result to the user, or execute a task matched with the speech instruction. The terminal device is connected to the server 200 through the network 300, the network 300 can be a wide area network or a local area network, or a combination of the two, and can use a wired or wireless link to realize data transmission, Figure 1 The way of transmitting data by using a wireless link in the above is only an example of the present application, and does not limit the present application in any way.

[0111] As an example, the server 200 is used to deploy a speech recognition model and train the speech recognition model, and deploy the trained speech recognition model in a corresponding terminal device, and process speech information in a media usage environment by using the deployed speech recognition model through the terminal device. The speech recognition model can be the second multi-task neural network model or the first multi-task neural network model provided by the embodiments of the present application. The speech recognition model deployed on the server 200 before training can be the second multi-task neural network model, and the speech recognition model deployed in the terminal device after training can be the first multi-task neural network model. The second multi-task neural network model and the first multi-task neural network model both integrate multiple tasks that can accurately predict characters and punctuation at the same time, and the model size is small, and can be deployed on an end side.

[0112] Of course, before processing the voice information through the voice recognition model to generate the corresponding voice recognition result, the voice recognition model also needs to be trained, specifically including: constructing a training sample, the training sample including: a sample sentence, the sample sentence including characters, the training sample also including: phonemes corresponding to the characters in the sample sentence, punctuation; using the training sample to train the second multi-task neural network model to obtain the first multi-task neural network model.

[0113] Among them, the voice recognition method provided by the embodiment of the application is realized based on artificial intelligence. Artificial intelligence (AI) is to use digital computers or digital computer controlled machines to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science, which tries to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0114] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software level technology. Artificial intelligence basic technology generally includes technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, etc.

[0115] In the embodiment of the application, the main artificial intelligence software technology involved includes the above-mentioned speech processing technology and machine learning, etc. For example, it can involve the speech recognition technology (Automatic Speech Recognition, ASR) in the speech technology (Speech Technology), which includes speech signal preprocessing (Speech signal preprocessing), speech signal frequency analysis (Speech signal frequency analyzing), speech signal feature extraction (Speech signal feature extraction), speech signal feature matching / recognition (Speech signal feature matching / recognition), speech training (Speech training), etc.

[0116] For example, it can involve machine learning (ML), which is a multidisciplinary field that involves probability theory, statistics, approximation theory, convex analysis, algorithmic complexity theory, etc. It is a specialized study of how computers simulate or implement human learning behavior to acquire new knowledge or skills, reorganize existing knowledge structure, and continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent, and its applications are widespread in various fields of artificial intelligence. Machine learning usually includes technologies such as deep learning, which includes artificial neural networks such as convolutional neural networks (CNN), recurrent neural networks (RNN), deep neural networks (DNN), etc.

[0117] It can be understood that the voice recognition model training method and voice recognition provided by the present application can be applied to an intelligent device. The intelligent device can be any device with voice command recognition function, such as an intelligent terminal, an intelligent home device (such as a smart speaker, a smart washing machine, etc.), a smart wearable device (such as a smart watch), a vehicle-mounted intelligent central control system (which wakes up a small program in the terminal for executing different tasks through voice commands), or an AI intelligent medical device (which is awakened and triggered through voice commands).

[0118] The structure of the voice recognition model training device of the embodiments of the present application will be described in detail below. The voice recognition model training device can be implemented in various forms, such as a special terminal with voice recognition model training function, or a server with voice recognition model training function, such as the server 200 in Figure 1 . Figure 2 The composition structure diagram of the voice recognition model training device provided by the embodiments of the present application can be understood as follows: Figure 2 Only an exemplary structure of the voice recognition model training device is shown, not all structures. The part or all of the structures shown can be implemented as needed. Figure 2 .

[0119] The speech recognition model training apparatus provided in the embodiments of the present application includes at least one processor 201, a storage unit 202, a user interface 203, and at least one network interface 204. The various components in the speech recognition model training apparatus are coupled together through a bus system 205. It can be understood that the bus system 205 is used to realize the connection and communication between the components. In addition to including a data bus, the bus system 205 also includes a power supply bus, a control bus, and a status signal bus. However, for the purpose of clear illustration, all the buses are marked as the bus system 205 in Figure 2

[0120] The user interface 203 can include a display, a keyboard, a mouse, a trackball, a click wheel, a key, a button, a touchpad, or a touch screen, etc.

[0121] It can be understood that the storage unit 202 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. The storage unit 202 in the embodiments of the present application can store data to support the operation of the terminal device (such as 10-1). Examples of the data include any computer programs for operating on the terminal device (such as 10-1), such as an operating system and an application program. The operating system contains various system programs, for example, a framework layer, a core library layer, a driver layer, etc., for realizing various basic services and processing hardware-based tasks. The application program can include various application programs.

[0122] In some embodiments, the speech recognition model training apparatus provided in the embodiments of the present application can be realized in a combination of software and hardware. As an example, the speech recognition model training apparatus provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the speech recognition model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can use one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic elements.

[0123] ​As an example of the speech recognition model training apparatus provided in the embodiments of the present application being implemented in a combination of software and hardware, the speech recognition model training apparatus provided in the embodiments of the present application can be directly embodied as a combination of software modules executed by the processor 201. The software modules can be located in the storage medium, and the storage medium is located in the storage unit 202. The processor 201 reads executable instructions included in the software modules in the storage unit 202, and completes the speech recognition model training method provided in the embodiments of the present application in combination with the necessary hardware (for example, including the processor 201 and other components connected to the bus 205).

[0124] As an example, the processor 201 can be an integrated circuit chip with a processing capability of signals, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor.

[0125] As an example of the speech recognition model training apparatus provided in the embodiments of the present application being implemented in hardware, the apparatus provided in the embodiments of the present application can be directly implemented in the form of a hardware decoding processor, such as one or more application specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or other electronic elements, to execute and implement the speech recognition model training method provided in the embodiments of the present application.

[0126] The storage unit 202 in the embodiments of the present application is used to store various types of data to support the operation of the speech recognition model training apparatus. Examples of these data include any executable instructions for operating on the speech recognition model training apparatus.

[0127] In some other embodiments, the speech recognition model training apparatus provided in the embodiments of the present application can be implemented in software, Figure 2The voice recognition model training apparatus stored in the storage unit 202, which can be software in the form of programs and plug-ins, etc., and includes a series of modules, as an example of the program stored in the storage unit 202, can include the voice recognition model training apparatus, and the voice recognition model training apparatus includes the following software modules: a construction module for constructing a training sample, the training sample including: a sample sentence, the sample sentence including characters, the training sample further including: phonemes corresponding to the characters in the sample sentence, punctuation; a training module for training the second multi-task neural network model to obtain the first multi-task neural network model using the training sample; wherein the second multi-task neural network model and the first multi-task neural network model can output a first prediction result, display at least part of the first prediction result, and the first prediction result includes character prediction results and punctuation prediction results.

[0128] The voice recognition method provided by the embodiments of the present application can be applied to terminal devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc., and can also be applied to databases, servers, and terminal-based artificial intelligence service response systems for responding to voice recognition requests. The embodiments of the present application do not limit the specific type of terminal device.

[0129] For example, the terminal device can be a station (STATION, ST) in a WLAN, a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Wireless Local Loop (WLL) station, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication function, a computing device, or other processing devices connected to a wireless modem, a computer, a laptop, a handheld communication device, a handheld computing device, and / or other devices for communicating on a wireless system, and a next-generation communication system, such as a mobile terminal in a 5G network or a mobile terminal in a future evolved Public Land Mobile Network (PLMN) network, etc.

[0130] As an example but not limitation, when the terminal device is a wearable device, the wearable device can also be a general term of application of wearable technology to the intelligent design of daily wear, and development of wearable devices, such as glasses, gloves, watches, clothing and shoes, etc. The wearable device is a portable device directly worn on the body, or integrated into the user's clothes or accessories, which collects the user's atrial fibrillation signal by attaching to the user. The wearable device is not only a hardware device, but also a powerful function through software support and data interaction, cloud interaction. The general wearable smart device includes a full function, large size, and can realize complete or partial functions without relying on a smart phone, such as a smart watch or smart glasses, etc., and only focuses on a certain application function, and needs to cooperate with other devices such as a smart phone, such as various types of smart bracelets, smart jewelry, etc.

[0131] Taking the terminal device as a mobile phone as an example. Figure 3 The figure shows a block diagram of part of the structure of the mobile phone provided by the embodiments of the application. Referring to Figure 3 , the mobile phone includes: radio frequency (RF) circuit 110, memory 120, input unit 130, display unit 140, sensor 150, audio circuit 160, near field communication module 170, processor 180, and power supply 190, etc. Those skilled in the art can understand that Figure 3 The structure of the mobile phone shown in the figure does not constitute a limitation on the mobile phone, and can include more or less components than the figure, or combine certain components, or different component arrangements.

[0132] The following will be specifically introduced in combination with Figure 3 each component of the mobile phone:

[0133] The RF circuit 110 can be used for receiving and sending signals in the process of information or communication, in particular, receiving the downlink information from the base station and sending the uplink data to the base station. Generally, the RF circuit includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 110 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc. The voice signal collected by other terminals is received through the RF circuit 110, and the voice signal is recognized to output corresponding text information.

[0134] The memory 120 can be used to store software programs and modules, and the processor 180 executes various functions of the mobile phone and data processing by running the software programs and modules stored in the memory 120, for example, storing the trained real-time voice recognition algorithm in the memory 120. The memory 120 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 120 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0135] The input unit 130 can be used to receive inputted digital or character information, and to generate key signal input related to user settings and function control of the mobile phone 100. Specifically, the input unit 130 can include a touch panel 131 and other input devices 132. The touch panel 131, also known as a touch screen, can collect touch operations of a user thereon or adjacent thereto (such as operations of a user using a finger, a stylus, or any suitable object or accessory on or adjacent to the touch panel 131), and drive corresponding connection devices according to pre-set programs.

[0136] The display unit 140 can be used to display information inputted by a user or provided to the user, as well as various menus of the mobile phone, such as outputted text information after voice recognition. The display unit 140 can include a display panel 141, which can be configured in the form of a Liquid Crystal Display (LCD), an Organic Light-Emitting Diode (OLED), or the like. Further, the touch panel 131 can cover the display panel 141, and when the touch panel 131 detects a touch operation thereon or adjacent thereto, transmits to the processor 180 to determine the type of the touch event, and then the processor 180 provides corresponding visual output on the display panel 141 according to the type of the touch event. Although in the above description, the touch panel 131 and the display panel 141 are implemented as two independent components to realize the input and output functions of the mobile phone, in some embodiments, the touch panel 131 and the display panel 141 can be integrated to realize the input and output functions of the mobile phone. Figure 3

[0137] The mobile phone 100 can further include at least one sensor 150, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor can include an ambient light sensor and a proximity sensor, wherein the ambient light sensor can adjust the brightness of the display panel 141 according to the brightness of ambient light, and the proximity sensor can turn off the display panel 141 and / or the backlight when the mobile phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (generally three axes), and when at rest, can detect the magnitude and direction of gravity, and can be used for applications of identifying the posture of the mobile phone (such as switching between landscape and portrait, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), and the like. As for other sensors that can be configured in the mobile phone, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, they will not be described here.

[0138] ​The audio circuit 160, the speaker 161 and the microphone 162 can provide an audio interface between the user and the mobile phone. The audio circuit 160 can convert the received audio data into an electrical signal, transmit the electrical signal to the speaker 161, and convert the electrical signal into a sound signal by the speaker 161 for output; on the other hand, the microphone 162 converts the collected sound signal into an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then output to the processor 180 for processing, and then transmitted to another mobile phone through the RF circuit 110, or output to the memory 120 for further processing. For example, the terminal device can collect the target voice signal of the user through the microphone 162, and send the converted electrical signal to the processor of the terminal device for voice recognition.

[0139] The terminal device can receive the target voice signal sent by other devices through the near field communication module 170, for example, the near field communication module 170 integrates a Bluetooth communication module, and establishes a communication connection with the wearable device through the Bluetooth communication module, and receives the target voice signal fed back by the wearable device. Although Figure 3 The near field communication module 170 is shown, but it can be understood that it does not belong to the necessary components of the mobile phone 100, and can be omitted as needed without changing the essence of the application.

[0140] The processor 180 is the control center of the mobile phone, which connects all parts of the mobile phone through various interfaces and lines, executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 120 and calling data stored in the memory 120, and thus monitors the whole mobile phone. Optionally, the processor 180 can include one or more processing units; preferably, the processor 180 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 180.

[0141] The mobile phone 100 further includes a power supply 190 (such as a battery) for supplying power to various components, and preferably, the power supply can be logically connected to the processor 180 through a power management system, so as to realize the functions of managing charging, discharging and power consumption management through the power management system.

[0142] Figure 4 is a software structure diagram of the mobile phone 100 of the embodiment of the application. Taking the Android system as the operating system of the mobile phone 100 as an example, in some embodiments, the Android system is divided into four layers, which are an application layer, an application framework layer (FWK), a system layer and a hardware abstraction layer, and the layers communicate with each other through software interfaces.

[0143] As shown in Figure 4 The application layer can include a series of application packages, which can include short message, calendar, camera, video, navigation, gallery, call, etc. In particular, the voice recognition algorithm can be embedded in the application, and the voice recognition process can be started through the relevant control in the application, and the collected target voice signal can be processed to obtain the corresponding text information.

[0144] The application framework layer provides the application programming interface (API) and programming framework for the application of the application layer. The application framework layer can include some pre-defined functions, such as functions for receiving events sent by the application framework layer.

[0145] As shown in Figure 4 The application framework layer can include a window manager, a resource manager, and a notification manager, etc.

[0146] The window manager is used to manage the window program. The window manager can obtain the size of the display screen, determine whether there is a status bar, lock the screen, and intercept the screen, etc. The content provider is used to store and obtain data, and make the data accessible to the application. The data can include video, image, audio, dialed and received call, browsing history and bookmark, phonebook, etc.

[0147] The resource manager provides various resources for the application, such as localized strings, icons, pictures, layout files, video files, etc.

[0148] The notification manager makes the application display notification information in the status bar, which can be used to convey the type of message, which can automatically disappear after a short stay without user interaction. For example, the notification manager is used to inform the download completion, message reminder, etc. The notification manager can also be a notification in the form of a chart or a scroll bar text appearing in the top status bar of the system, such as the notification of the application running in the background, and can also be a notification in the form of a dialog window appearing on the screen. For example, the text information is prompted in the status bar, the prompt sound is emitted, the electronic device is vibrated, the indicator light is blinked, etc.

[0149] The application framework layer can also include:

[0150] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build an application. The display interface can be composed of one or more views. For example, the display interface including the short message notification icon can include a view for displaying text and a view for displaying pictures.

[0151] The phone manager is used to provide the communication function of the mobile phone 100. For example, the management of the call state (including the connection, hang-up, etc.).

[0152] The system layer can include a plurality of function modules. For example: a sensor service module, a physical state identification module, a three-dimensional graphics processing library (for example: OpenGL ES), etc.

[0153] The sensor service module is used to monitor the sensor data uploaded by various sensors of the hardware layer, and determine the physical state of the mobile phone 100.

[0154] The physical state identification module is used to analyze and identify the user gestures, faces, etc.

[0155] The three-dimensional graphics processing library is used to realize three-dimensional graphics drawing, image rendering, synthesis, and layer processing, etc.

[0156] The system layer can also include:

[0157] The surface manager is used to manage the display subsystem, and provides the fusion of 2D and 3D layers for a plurality of application programs.

[0158] The media library supports a plurality of commonly used audio, video format playback and recording, and static image files, etc. The media library can support a plurality of audio and video coding formats, for example: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0159] The hardware abstraction layer is the layer between the hardware and the software. The hardware abstraction layer can include a display driver, a camera driver, a sensor driver, a microphone driver, etc., which are used to drive the related hardware of the hardware layer, such as the display screen, the camera, the sensor, and the microphone, etc. In particular, the microphone module is started through the microphone driver, and the target voice information of the user is collected, so as to linearly follow the subsequent voice recognition process.

[0160] It should be noted that the voice recognition method provided by the embodiments of the present application can be executed in any of the above-mentioned levels, which is not limited herein.

[0161] The voice recognition method of the embodiments of the present application constructs a neural network model for simultaneously predicting the characters and the punctuation corresponding to the phonemes, constructs a training sample set to train the neural network model, obtains the trained neural network model, and does not need to perform word segmentation processing in the training process. The converted phonemes (vectors) of the voice to be recognized are taken as the input of the trained neural network model, forward reasoning is performed, the characters and the punctuation corresponding to the phonemes can be simultaneously output, the size of the neural network model is small, and the neural network model can be deployed on the terminal side.

[0162] The "simultaneously", "simultaneously output" and the like expressed in the present text can be understood as being able to obtain two kinds of information (such as character information corresponding to phonemes and punctuation information corresponding to phonemes) from the output of the neural network model, not just one kind of information, and does not limit the time sequence of obtaining the two kinds of information, in other words, the "simultaneously" described in the present text does not limit the time to be the same time.

[0163] Figure 5a A block diagram of a neural network model according to an embodiment of the present application is shown. As shown, the input of the neural network model is the label sequence corresponding to phonemes converted from the speech to be recognized, and the neural network model can perform feature extraction on the label sequence. Specifically, the label sequence can be feature extracted by an embedding layer to obtain a corresponding feature vector, and then the character and punctuation corresponding to the phonemes can be predicted according to the feature vector, and the character and punctuation corresponding to the speech can be output simultaneously. In the embodiments of the present application, the neural network model can complete multiple tasks simultaneously, and therefore, the neural network model is referred to as a multi-task neural network model hereinafter. Figure 5a

[0164] Specifically, in the embodiments of the present application, when predicting the character and punctuation corresponding to the phonemes according to the feature vector, the neural network model can use a classifier to predict the character and punctuation corresponding to each phoneme, so as to realize simultaneous output of the character and punctuation, and the multi-task neural network model that simultaneously realizes character and punctuation prediction can be deployed on the terminal side.

[0165] In the embodiments of the present application, the punctuation can include blank, as well as comma, period, question mark and exclamation mark, etc. The comma, period, question mark and exclamation mark can also be divided into two forms of full angle in Chinese and half angle in English.

[0166] One Chinese character can have multiple pinyin, and one English character corresponds to multiple English phonemes, which can cause the lengths of phonemes and characters to be inconsistent, and the number of punctuation in a sentence can also be inconsistent with the lengths of characters and phonemes, that is, the lengths of the input sequence and the output sequence are inconsistent, and the prior art cannot simultaneously output the prediction results. If an encoder-decoder is used, the problem of inconsistent lengths of the input sequence and the output sequence can be solved, but the current output must depend on the previous output. Figure 5b A schematic diagram of an encoder-decoder model according to an example of the present application is shown. As shown, Figure 5b ​As shown, for example, the input sequence of the encoder in the encoder-decoder model is "X1X2X3X4", the encoder encodes to output a vector C to the decoder, and the decoder decodes to obtain an output sequence "Y1Y2Y3" with a length of 3. Before outputting "Y2", "Y1" must be output first, and "Y1Y2Y3" cannot be output simultaneously, which leads to poor real-time output of recognition results.

[0167] Figure 5c A schematic diagram of an encoder model according to an example of the present application is shown. Although Figure 5c The encoder model shown can include an encoder and a Softmax classifier, where the encoder model is used to encode an input sequence to obtain a feature vector C, and the Softmax classifier can obtain an output sequence according to the feature vector C. In Figure 5c In the example, "Y1Y2Y3Y4" can be output simultaneously according to the input sequence "X1X2X3X4", but this can only be applied to scenarios where the input sequence and the output sequence have the same length.

[0168] The embodiments of the present application provide a method for constructing a training sample set. The method for constructing a training sample set according to the embodiments of the present application aligns the length of characters in a sample sentence with the length of phonemes after annotation and the length of punctuation. The above neural network structure constructed according to the embodiments of the present application can use Figure 5c The encoder model shown implements conversion of phonemes to characters and punctuation. Since the encoder model is suitable for scenarios where the input sequence and the output sequence have the same length, after the neural network model is trained using the training sample set constructed according to the embodiments of the present application, the neural network model can simultaneously perform conversion of Chinese and English phonemes to characters and punctuation prediction, and solves the technical problem that in the related art, the input and output do not have the same length, and the results cannot be output simultaneously.

[0169] The embodiments of the present application also provide a method for training a multi-task neural network model. The training sample in the training sample set is input into a second multi-task neural network model for training to obtain a trained first multi-task neural network model. The second multi-task neural network model and the first multi-task neural network model incorporate punctuation prediction and character prediction, and generate punctuation in real time while generating characters in real time, implement simultaneous training of multiple tasks, and the size of the first multi-task neural network model is small, and the first multi-task neural network model can be deployed on an end side.

[0170] The voice recognition method of the present application is described below according to the process of training sample set construction, neural network model training and neural network model inference. In order to clearly describe the embodiments provided by the present application, the neural network model before training is referred to as a second multi-task neural network model, and the neural network model obtained after training is referred to as a first multi-task neural network model. Among them, "first" and "second" are only used to distinguish different features, and do not represent a specific order or size relationship.

[0171] Training sample set construction

[0172] Figure 6 A schematic diagram showing the process of constructing a training sample set according to an embodiment of the present application is shown. As shown in Figure 6 The phonetic dictionary can be constructed, and the constructed phonetic dictionary can include a dictionary and a phoneme character mapping table. Among them, the dictionary can include one language or multiple languages, such as a Chinese dictionary or an English dictionary, or a mixed Chinese-English dictionary, or a mixed dictionary of other multiple languages, which is not limited by the present application. For a dictionary of one language, it can also include multiple different dictionaries, which can be obtained by classifying according to the characteristics of the language. Taking a Chinese dictionary as an example, it can be further subdivided into rare character dictionary, multi-sound character dictionary, idiom dictionary, personal name dictionary, etc. Further subdividing the dictionary according to the characteristics of the language helps to improve the training effect and improve the accuracy of prediction.

[0173] The phoneme character mapping table is used to store the correspondence between characters and corresponding phonemes. One character can correspond to one or more phonemes, and the processor can phonetize the character according to the phoneme character mapping table to obtain the phonemes corresponding to the character. For example, for Chinese characters, due to the existence of multi-sound characters, one Chinese character can correspond to one or more phonemes; for English characters, since some English words include multiple syllables, one English character can also correspond to one or more phonemes; the processor can look up the phoneme character mapping table according to the character to determine one or more phonemes corresponding to the character.

[0174] The corpus used to construct the training sample set can be a single language or a mixed corpus including multiple languages. The processor can phonetize the characters in the corpus according to the phonetic dictionary to obtain the phonemes corresponding to the characters, and align the phonemes corresponding to the characters with the characters and punctuation, the length of the characters and the length of the punctuation being the same as the length of the corresponding phonemes.

[0175] In a possible implementation, the processor can transcribe each character in the corpus to obtain a phoneme corresponding to the character, and determine whether the length of the character and the length of the phoneme corresponding to the character are the same. If not, the processor can perform alignment processing on the length of the character and the length of the phoneme corresponding to the character. Alternatively, the processor can first transcribe all characters in the corpus to obtain corresponding phonemes, and then perform alignment processing on the phonemes corresponding to the characters and the characters. The sequence of the transcription and alignment steps is not limited in the embodiments of the present application.

[0176] In a possible implementation, the alignment processing manner can be different for different languages. For example, for a multi-sound character in Chinese, the processor can select one phoneme from multiple phonemes as the phoneme corresponding to the character, that is, the phoneme corresponding to the multi-sound character in the aligned Chinese is any one of the multiple phonemes corresponding to the multi-sound character; for an English character, the processor can add an alignment character to the character to perform alignment, the English character after alignment includes the alignment character, and the length of the English character after alignment and the length of the phoneme corresponding to the English character are the same. When adding the alignment character to the character, the position of the alignment character can be before or after the character, which is not limited in the present application. The alignment character can be any symbol other than English letters, for example, the alignment character can be “@”, “*”, “&” or “%” and the like.

[0177] In a possible implementation, when transcribing, if there is no character in the transcription dictionary that is the same as the English character in the corpus, the processor can split the English character in the corpus to obtain multiple independent sub-characters, and transcribe the sub-characters in the transcription dictionary.

[0178] In a possible implementation, one character corresponds to one punctuation mark. In the embodiments of the present application, in addition to aligning the phonemes and characters, the processor can also align the characters and punctuation marks. As described above, the punctuation marks can include blank and commas, periods, question marks and exclamation marks, and the like. If there is no punctuation mark after the original character, the punctuation mark corresponding to the character can be set as blank. For the character without a punctuation mark before alignment, the punctuation mark after alignment is blank, so that the length of the punctuation mark is aligned with the length of the character, and the punctuation mark corresponding to the character is blank when output. The processor can simultaneously align the characters, phonemes and punctuation marks, or perform the alignment in steps, which is not limited in the present application.

[0179] In a possible implementation, the training sample set construction method of the present application can also align the lengths of different sentences trained simultaneously. For example, if multiple sentences are trained simultaneously, that is, batch size > 1, if the lengths of the multiple sentences trained simultaneously are different, then the characters, phonemes and punctuation marks corresponding to the shorter sentences can be padded with Null, and the lengths of the characters, phonemes and punctuation marks corresponding to the shorter sentences are the same as the length of the longest sentence after padding with Null.

[0180] For example, there are two sentences for training, which are:

[0181] Hello!

[0182] Very good.

[0183] The sentence "Hello!" has only two characters, which is relatively short, and the characters, phonemes and punctuation marks corresponding to the sentence can be padded with "Null", so in this embodiment, the punctuation marks corresponding to the punctuation sentence are "[BLANK]! [Null]", and the punctuation marks of "Very good." are "[BLANK] [BLANK].". In this way, the lengths of the two sentences are the same, and they can be used for training.

[0184] In a possible implementation, the corpus can be preprocessed before phonetic transcription and alignment. The specific content of preprocessing can be determined according to the constructed dictionary and the specific language type. For example, if the dictionary does not include numbers, preprocessing the corpus can include converting numbers in the corpus to Chinese characters (such as Figure 6 regularization of numbers shown); if the English words in the dictionary are in uppercase, preprocessing the corpus can also include converting English in the corpus from lowercase to uppercase (such as Figure 6 conversion of English letters shown). Preprocessing can also include simplifying traditional Chinese characters, removing special characters, and other processing. The specific way of preprocessing can be determined according to the phonetic dictionary and language characteristics, and the present application does not limit the specific preprocessing method.

[0185] Figure 7 An example of the process of constructing a training sample set according to an embodiment of the present application is shown. For example, as Figure 7 shown, taking the mixed Chinese-English sentence "use P30 to open CCTV to watch NBA video." as an example, the corpus can be preprocessed to convert numbers to Chinese characters and convert English from lowercase to uppercase, and "use P three zero to open CCTV to watch NBA VIDEO." can be obtained.

[0186] Phonetic transcription is performed on "use P three zero to open CCTV to watch NBA VIDEO.", Chinese is transcribed into pinyin, and English is transcribed into corresponding English phonemes, as Figure 7As shown, the English characters "NBA" correspond to three phonemes "en bi ei", and the English characters "VIDEO" correspond to two phonemes "vi diu". Since the English characters "CCTV" are not in the phonetic dictionary, "CCTV" can be split into four independent sub - characters, and the corresponding phonemes "see see ti vi" can be obtained by phonetically annotating the sub - characters according to the phonetic dictionary. The final phonemes can be "yong4 pi san1 ling2 da3 kai1 see see ti vi kan4 en bi ei vidiu".

[0187] Alignment processing: The processor can perform alignment processing during the phonetic annotation process or uniformly perform alignment processing after phonetic annotation. This application does not limit this. As Figure 7 shown, the alignment character used in the example of this application can be "@". After performing alignment processing on the character "NBA", "@@NBA" can be obtained because "NBA" corresponds to three phonemes; the character "VIDEO" corresponds to two phonemes, and the processor can obtain "@VIDEO" after performing alignment processing on the character "VIDEO". For the scenario of Chinese polyphonic characters, for example: "long hair", the phonetic annotation can obtain "chang2|zhang3 tou2 fa4", which will result in the Chinese character "长" corresponding to two pinyins. For such Chinese polyphonic characters, in the implementation of this application, the processor can randomly select one as the final Chinese character pinyin to achieve alignment processing. The result of the character after alignment with the phoneme is as Figure 7 shown in the last step of "yong4 pi san1 ling2 da3 kai1 see see ti vi kan4 en bi ei vidiu".

[0188] In the above example, there is a punctuation mark at the end of the corpus: a full stop. The character VIDEO and the phoneme diu correspond to the full stop. For other characters without punctuation, all punctuation is set to blank. As Figure 7 shown, it includes 15 blanks and one full stop, a total of 16 punctuations, which are aligned with 16 phonemes.

[0189] By aligning the lengths of the characters, the lengths of the punctuations, and the lengths of the phonemes corresponding to the characters, the training sample set constructed by the implementation of this application is used to train the second multi - task neural network model to obtain the first multi - task neural network model. For the scenario where the lengths of the input sequence and the output sequence are different, the second neural network model obtained after training through the alignment processing of this application can also output prediction results according to the input phonemes to be recognized. The prediction results can include the characters and punctuations corresponding to the phonemes to be recognized. That is to say, it can achieve real - time generation of characters while also generating punctuations in real - time, and the size of the first multi - task neural network model is small and can be deployed on the edge side.

[0190] Model training

[0191] Figure 8 A flowchart of a multi-task neural network model training method according to an embodiment of the present application is shown. The multi-task neural network model training method provided by the embodiments of the present application can be applied to Figure 2 The device shown.

[0192] In the embodiments of the present application, before training, a training sample can be selected from a training sample set as the input of the multi-task neural network, and the size of the training sample can be represented as (B, U), where B can represent the number of samples for one training, and U can represent the length of the phonemes corresponding to the longest sample in the batch of training samples. For example, B can be 128, indicating that the number of samples for one training is 128 sentences corresponding to phonemes, and the length of the phonemes corresponding to the longest sentence in the 128 sentences is U.

[0193] It should be noted that multiple batches of training samples can be selected to be input to the second multi-task neural network model for training. The larger the amount of data for training, the higher the accuracy of the predicted characters and punctuation of the first multi-task neural network model when it is used for inference.

[0194] The input of the neural network model must be a numerical value, and cannot be a string, so the training sample can be converted into numerical representation data before training. In the embodiments of the present application, the numerical value corresponding to each phoneme in the dictionary can be pre-set as the label of the phoneme. Before training, the corresponding label can be found according to the phoneme, so as to convert the training sample into a label sequence, that is, into a numerical representation vector as the input data of the neural network, and train the neural network.

[0195] For example, taking a training sample as an example, (jin1 tian1 tian1 qi4 bu2 cuo4), the input is (1, 6), and the training sample can be converted into a label sequence to obtain (10, 148, 148, 2456, 30, 40). That is, each phoneme has a corresponding label, and the label corresponding to the phoneme can be a number. By converting the sample sequence into a label sequence, the sample sequence can be converted into a vector representation to participate in the subsequent calculation process.

[0196] As Figure 8 shown, the multi-task neural network model training method provided by the present application can include:

[0197] Step S801, inputting the input data to the second multi-task neural network model, and determining the character probability matrix and the punctuation probability matrix corresponding to the input data;

[0198] Step S802, according to the character probability matrix and the punctuation probability matrix, respectively calculating the character cross-entropy loss and the punctuation cross-entropy loss;

[0199] Step S803, according to the character cross-entropy loss and the punctuation cross-entropy loss, calculating the weighted cross-entropy loss.

[0200] Step S804, adjusting the parameters of the second multi-task neural network model according to the weighted cross-entropy loss, to obtain the trained first multi-task neural network model.

[0201] In step S801, the input data can be a label sequence converted from phonemes, that is, a vector corresponding to the phonemes to be recognized.

[0202] In one possible implementation, in step S801, the training device can operate the input data through the second multi-task neural network model to obtain a feature vector of the input data; then, the training device can operate the feature vector through the second multi-task neural network model to predict the characters and punctuation corresponding to the training sample, to obtain the character probability matrix and the punctuation probability matrix. The input data can be the training sample described above.

[0203] In another possible implementation, the second multi-task neural network model can include an encoder in the encoder model as shown in Figure 5c The encoder is used to extract features from the phonemes to be recognized (input data) to obtain a feature vector.

[0204] For example, the encoder can include an embedding layer, and the training device can operate the input data through the embedding layer to extract the feature vector. Specifically, the training device can operate the input data according to the encoding method adopted by the embedding layer to obtain the feature vector, such as the vector C as shown in Figure 5c

[0205] In one possible implementation, each phoneme can be represented by a one-dimensional vector after encoding, and the length of the vector can be determined according to the number of phonemes in the dictionary, for example, in the example of the present application, 512 data can be used to represent one phoneme. In the embodiments of the present application, the correspondence between the labels corresponding to the phonemes and the vectors corresponding to the phonemes after encoding can be recorded.

[0206] Still taking the above example as an example, the dimension of the input data is (1, 6), that is, the labels converted from 6 phonemes, and the feature vector obtained after processing by the encoder can be (1, 6, 512).

[0207] ​In one possible implementation, the second multi-task neural network model may also include a classifier (such as...). Figure 5c The Softmax classifier shown can be used by the training device to classify feature vectors through the classifier in the second multi-task neural network model, resulting in a character probability matrix and a punctuation probability matrix. The character probability matrix represents the first probability of a character corresponding to a phoneme, and the punctuation probability matrix represents the second probability of a punctuation mark corresponding to each phoneme. Based on the character probability matrix and the punctuation probability matrix, the corresponding character and punctuation mark can be obtained. In one possible implementation, a correspondence between characters and their corresponding first index values, and between punctuation marks and their corresponding second index values, can be pre-established to form a vocabulary. Thus, when training the second multi-task neural network model, or when performing inference using the first multi-task neural network model, the neural network model can obtain the corresponding character and punctuation mark based on the obtained character probability matrix, punctuation probability matrix, and vocabulary.

[0208] Specifically, the character probability matrix yields the first index value of the character with the highest first probability. Based on this first index value and the vocabulary, the corresponding character for the phoneme can be determined. Similarly, the punctuation probability matrix yields the second index value of the punctuation mark with the highest second probability. Based on this second index value and the vocabulary, the corresponding punctuation mark for the phoneme can be obtained. In other words, the Softmax classifier produces a probability matrix of characters corresponding to the phonemes to be identified (input data). The first probability in the matrix represents the probability that the character corresponding to the phoneme is the character with that first probability. Therefore, the character with the highest first probability is the corresponding character for the phoneme. The punctuation mark corresponding to the phoneme can be determined in the same way.

[0209] In one possible implementation, in step S802, the training device can calculate the character cross-entropy loss based on the cross-entropy loss function and the character probability matrix. The specific calculation formula is as follows:

[0210]

[0211] Where y(C) represents the cross-entropy loss of all characters, P(c i ) can represent the character c i The corresponding first probability is given by , where i represents the character index, and the value of i ranges from 1 to n, where n is a positive integer. Based on the above formula and the character probability matrix, the character cross-entropy loss can be calculated.

[0212] Similarly, in step S802, the training device can calculate the punctuation cross-loss based on the cross-entropy loss function and the punctuation probability matrix. The specific calculation formula is as follows:

[0213]

[0214] where y(P) represents the cross-entropy loss of all punctuations, P(p i ) can represent the corresponding second probability of the punctuation p i According to the above formula and the punctuation probability matrix, the punctuation cross-entropy loss can be calculated.

[0215] In a possible implementation, according to different requirements of the accuracy of character prediction and the accuracy of punctuation prediction, the first weight corresponding to the character cross-entropy loss and the second weight corresponding to the punctuation cross-entropy loss can be set. In step S803, the weighted cross-entropy loss can be calculated according to the character cross-entropy loss, the first weight, and the punctuation cross-entropy loss, the second weight. The cross-entropy loss can be calculated according to the following formula:

[0216] y(C+P) = w1 x y(C) + w2 x y(P)

[0217] where y(C+P) can represent the weighted cross-entropy loss of characters and punctuations, w1 can represent the first weight, and w2 can represent the second weight. In a possible implementation, the sum of the first weight and the second weight is 1, that is, w2 = 1-w1. For example, assuming that the first weight is 0.7, the second weight can be 0.3.

[0218] It should be noted that the above setting method and example of the first weight and the second weight are only some examples of the present application, and do not limit the present application in any way.

[0219] After obtaining the weighted cross-entropy loss, in step S804, the training device can update the weights of the second multi-task neural network model according to the weighted cross-entropy by using the back propagation algorithm, to obtain the trained first multi-task neural network model. In a possible implementation, the Adam optimizer can be used to update the weights.

[0220] The training method of the multi-task neural network model of the present application can realize the training of the tasks of character prediction and punctuation prediction at the same time. In addition, since the training sample set constructed includes multiple languages, the training method of the multi-task neural network model of the present application can also realize the training of the task of recognizing (predicting) multiple languages. The multi-task neural network model trained by the training method of the multi-task neural network model according to the embodiments of the present application can simultaneously predict multiple languages and punctuations, and the multi-task neural network model has a smaller size compared with a traditional acoustic model, and can be deployed on an end side.

[0221] Model inference

[0222] After the first multi-task neural network model is obtained by training the second multi-task neural network model, the phonemes to be recognized can be input into the first multi-task neural network model for forward inference to simultaneously predict and output the characters and punctuation corresponding to the phonemes.

[0223] Therefore, the present application also provides a speech recognition method, which can be applied to a terminal device as shown in Figure 1 or Figure 3 After the first multi-task neural network model is obtained, the first multi-task neural network model can be deployed in the terminal device.

[0224] Figure 9a A schematic diagram showing an application scenario of speech recognition on the terminal device side according to an embodiment of the present application is shown. As shown in Figure 9a The acoustic model and the neural network model (the first multi-task neural network model) can be deployed in the terminal device. The terminal device can input the collected or received speech signal into the acoustic model, and the phonemes corresponding to the speech signal can be obtained by processing the speech signal through the acoustic model and output to the first multi-task neural network model.

[0225] Figure 10 A flowchart showing a speech recognition method according to an embodiment of the present application is shown. As shown in Figure 10 The speech recognition method according to an embodiment of the present application can include the following steps:

[0226] Step S901, input the phonemes to be recognized into the first multi-task neural network model, wherein the first multi-task neural network model is obtained by training the second multi-task neural network model using training samples.

[0227] The training samples include sample sentences, and the sample sentences include characters. The training samples include the phonemes corresponding to the characters in the sample sentences and punctuation. In a possible implementation, the lengths of the phonemes, characters and punctuation are the same.

[0228] The second multi-task neural network model and the first multi-task neural network model can output the first prediction result and display at least part of the first prediction result. The first prediction result includes character prediction results and punctuation prediction results. That is, the second multi-task neural network model and the first multi-task neural network model can simultaneously predict the characters and punctuation corresponding to the phonemes to be recognized according to the phonemes to be recognized.

[0229] The process of constructing the training samples and obtaining the first multi-task neural network model by training the second multi-task neural network model using the training samples can be referred to the description above, and will not be repeated here.

[0230] AsFigure 9a As shown, the phonemes to be recognized can be obtained by processing the speech signal to be recognized by using an acoustic model. The speech signal to be recognized can be a signal collected by the terminal device or a signal received by the terminal device, which is not limited in the present application.

[0231] For example, in one example, the terminal device opens a social APP, detects that the microphone is opened, and collects a speech signal; if the terminal device detects a conversion request for converting the speech signal into text, the terminal device can input the speech signal into the acoustic model. In another example, the terminal device opens a social APP, receives a speech signal sent by another terminal device, and detects a conversion request, and the terminal device can input the speech signal into the acoustic model. After receiving the speech signal, the acoustic model can process the speech signal to obtain phonemes to be recognized. The terminal device can input the phonemes to be recognized into the first multi-task neural network model.

[0232] In the embodiments of the present application, the phonemes to be recognized output by the acoustic model can be a label sequence corresponding to the phonemes.

[0233] As Figure 10 shown, the speech recognition method of one embodiment of the present application can further include:

[0234] Step S902, the terminal device outputs a first prediction result by using the first multi-task neural network model, the first prediction result including a character prediction result and a punctuation prediction result corresponding to the phonemes to be recognized;

[0235] Step S903, the terminal device displays at least part of the first prediction result on the display screen of the terminal device according to the first prediction result.

[0236] According to Figure 5a As shown in the block diagram of the neural network model, the first multi-task neural network model can extract a feature vector of the phonemes to be recognized, and then the classifier can classify according to the feature vector, and can predict the character and punctuation corresponding to each phoneme to be recognized. For example, the classifier can obtain the classification according to the input phonemes to be recognized, obtain the corresponding character and punctuation, and output the predicted result (first prediction result). After the first multi-task neural network model outputs the first prediction result, the terminal device can display the predicted character and punctuation at the same time. Alternatively, the first multi-task neural network model can process the input phonemes to be recognized by using the encoder model shown in Figure 5c

[0237] Figure 9b A schematic diagram showing the process of speech recognition according to the prior art of one example of the present application is shown. As​Figure 9b As shown, in the traditional phoneme-to-character and punctuation method, the phonemes can be first mapped to characters, and then the corresponding punctuation is predicted. In an example, the phonemes can be first mapped to characters by an N-Gram language model, and then the punctuation is obtained by a punctuation prediction model after the characters are obtained. The prediction of characters and punctuation needs to be performed by two models respectively, and the characters and punctuation cannot be output simultaneously. Moreover, the model is large and cannot be deployed on the edge side. The speech recognition method adopted in the present application can simultaneously output characters and punctuation, and the model can be deployed on the edge side due to its simplicity. Figure 9a As shown, one neural network model can simultaneously output characters and punctuation, and the model can be deployed on the edge side due to its simplicity.

[0238] According to the speech recognition method provided by the embodiments of the present application, since the character prediction and punctuation prediction are integrated in the neural network model, the neural network model is trained by a specially constructed training sample set, and the multi-task neural network model obtained after training is deployed on the edge side, so that the simultaneously output and display of the predicted characters and punctuation can be realized.

[0239] In a possible implementation, the first multi-task neural network model can be a streaming network structure, and the terminal device inputs the phonemes to be recognized into the first multi-task neural network model, and outputs the first prediction result by using the first multi-task neural network model, which can include that the terminal device can input the phonemes to be recognized into the first multi-task neural network model in a loop, and output the first prediction result based on the length of the currently input phonemes to be recognized by using the first multi-task neural network model.

[0240] In a possible implementation, the terminal device inputs the phonemes to be recognized into the first multi-task neural network model in a loop, and outputs the first prediction result based on the length of the currently input phonemes to be recognized by using the first multi-task neural network model, which can include that:

[0241] Before completing the input of all the phonemes to be recognized into the first multi-task neural network model, if the length of the currently input phonemes is less than the receptive field, the terminal device continues to input the next phonemes;

[0242] Before completing the input of all the phonemes to be recognized into the first multi-task neural network model, if the length of the currently input phonemes is not less than the receptive field, the terminal device obtains the second prediction result of the first phoneme of the currently input phonemes according to the characters and punctuation of the currently input phonemes, and stores the second prediction result; and the terminal device inputs the feature vector of the first phoneme, the phonemes other than the first phoneme in the currently input phonemes, and the next phonemes to be recognized into the first multi-task neural network model;

[0243] After inputting all the phonemes to be recognized into the first multi-task neural network model, the terminal device obtains a second prediction result of the current input phoneme according to the character and punctuation of the current input phoneme;

[0244] If there is no stored second prediction result, the terminal device takes the second prediction result of the current input phoneme as the first prediction result of the phonemes to be recognized;

[0245] If there is a stored second prediction result, the terminal device obtains the first prediction result of the phonemes to be recognized according to the second prediction result of the current input phoneme and the stored second prediction result.

[0246] The second prediction result is the final result of one or more phonemes to be recognized, and the prediction result of the phonemes other than the first phoneme in the current input phoneme is a temporary prediction result. Therefore, the terminal device stores the second prediction result, and finally fuses all the second prediction results to obtain the first prediction result (the final result of all the phonemes to be recognized). Figure 11 A flowchart of a speech recognition method according to an embodiment of the present application is shown. As shown in the figure, Figure 11 In the speech recognition method of the embodiment of the present application, the phonemes to be recognized can be circulated into the first multi-task neural network model for character and punctuation prediction according to the relationship between the length of the phonemes to be recognized and the receptive field of the first multi-task neural network model. Specifically, the following processes can be included:

[0247] Step S1100: It is determined whether the input of all the phonemes to be recognized has been completed. If the input of all the phonemes to be recognized has not been completed, step S1101 is performed; if the input of all the phonemes to be recognized has been completed, step S1104 is performed.

[0248] Step S1101: It is determined whether the length of the current input phoneme is less than the receptive field. If the length of the current input phoneme is less than the receptive field, step S1102 is performed; if the length of the current input phoneme is not less than the receptive field, step S1103 is performed.

[0249] Step S1102: The character and punctuation of the current input phoneme are predicted to obtain a temporary result of the current input phoneme, and the next phoneme is inputted, and the process returns to step S1100.

[0250] Step S1103: The character and punctuation of the current input phoneme are predicted to obtain the final result of the first phoneme of the current input phoneme, and the terminal device can store the final result. The terminal device inputs the feature vector of the first phoneme, the phonemes other than the first phoneme in the current input phoneme, and the next phoneme in the phonemes to be recognized into the first multi-task neural network model, and the process returns to step S1100.

[0251] Step S1104, the character and punctuation of the current input phoneme are predicted to obtain the final result of the current input phoneme, and it is judged whether there is a stored final result; if there is a stored final result, step S1105 is executed; if there is no stored final result, step S1106 is executed.

[0252] Step S1105, the stored final result and the final result of the current input phoneme are fused to obtain the final result of the phoneme to be recognized, and the loop is ended.

[0253] Step S1106, the final result of the current input phoneme is taken as the final result of the phoneme to be recognized, and the loop is ended.

[0254] In step S1100, the terminal device can determine whether the input of all phonemes to be recognized has been completed according to the output of the previously connected acoustic model. If the acoustic model no longer outputs new phonemes, the terminal device can determine that all phonemes have been input into the first multi-task neural network model. Otherwise, the terminal device can determine that the input of all phonemes to be recognized has not been completed.

[0255] In a possible implementation, the terminal device can also be provided with a VAD (Voice Activity Detection). The VAD can detect when there is a human voice in an audio and when the human voice ends. After detecting the end of the human voice in the audio, the acoustic model can be controlled to no longer output.

[0256] The length of the current input phoneme is 1 from the beginning of input, and gradually increases as more phonemes are input. If the length of all phonemes to be recognized is greater than or equal to the receptive field, the length of the current input phoneme no longer changes when it increases to the size of the receptive field. If there is a new input phoneme, the first phoneme of the current input phoneme is no longer input into the first multi-task neural network model. If the length of all phonemes to be recognized is less than the receptive field, the maximum length of the current input phoneme is less than the receptive field.

[0257] For example, assuming that the receptive field of the first multi-task neural network model is 8 and the length of the phoneme to be recognized is 15. When the first 7 phonemes to be recognized are input, the length of the current input phonemes is 1, 2, 3, 4, 5, 6, and 7 respectively, and the length of the current input phonemes is less than the receptive field. From the 8th phoneme to be recognized, the length of the current input phonemes is 8, which is not less than the receptive field. When the 9th phoneme to be recognized is input, the length of the current input phonemes is still 8, and the current input phonemes are 2, 3, 4, 5, 6, 7, 8, and 9 respectively. The same is true for the 10th phoneme to be recognized and the phonemes to be recognized thereafter. Assuming that the receptive field of the first multi-task neural network model is 8 and the length of the phoneme to be recognized is 7, the length of the current input phonemes is at most 7, which is less than the receptive field.

[0258] If it is determined that the length of the current input phonemes is less than the receptive field, the terminal device can perform step S1102 to predict the characters and punctuation of the current input phonemes to obtain a temporary result of the current input phonemes. The length of the current input phonemes is less than the receptive field, which means that the characters and punctuation predicted from the current input phonemes can still change based on the phonemes input thereafter. Therefore, when the length of the current input phonemes is less than the receptive field, the terminal device can take the result of the prediction of the current input phonemes as the temporary result. The terminal device can input the next phoneme to be recognized predicted by the acoustic model into the first multi-task neural network model, and then return to step S1100 to continue to determine whether the input of all phonemes to be recognized has been completed.

[0259] For example, still taking the above example, the current input phonemes are the first, second, third, fourth, and fifth phonemes, and the length of the current input phonemes is 5, which is less than the receptive field 8. Therefore, the terminal device can take the prediction result of the characters and punctuation of the first, second, third, fourth, and fifth phonemes as the temporary result, and input the next (sixth) phoneme to be recognized, that is, the current input phonemes are the first, second, third, fourth, fifth, and sixth phonemes.

[0260] If it is determined that the length of the current input phonemes is not less than the receptive field, the terminal device can perform step S1103 to predict the characters and punctuation of the current input phonemes to obtain the final result of the first phoneme of the current input phonemes, and the terminal device can store the final result. The prediction result of the phonemes other than the first phoneme in the current input phonemes is the temporary result. The terminal device can continue to input the feature vector of the first phoneme extracted in the current prediction process, the phonemes other than the first phoneme in the current input phonemes, and the next phoneme to be recognized into the first multi-task neural network model. Then return to step S1100 to continue to determine whether the input of all phonemes to be recognized has been completed.

[0261] For example, still taking the above example, the current input phonemes are the first to the eighth phonemes, the length of the current input phonemes is 8, which is equal to the receptive field 8, that is, not less than the receptive field 8. Therefore, the terminal device can take the result of the prediction of the first phoneme as the final result and store the final result. The terminal device can take the result of the prediction of the second to the eighth phonemes as temporary results. The terminal device can input the feature vector of the first phoneme extracted in the current prediction process, the second to the eighth phonemes, and the ninth phoneme into the first multi-task neural network model for further inference. Continuing the inference, the result of the prediction of the second (the first phoneme of the current input phonemes) is taken as the final result and stored. The terminal device can input the feature vector of the second phoneme extracted in the current prediction process, the third to the ninth phonemes, and the tenth phoneme into the first multi-task neural network model for further inference. The above process is repeated until the input of all to-be-recognized phonemes is completed.

[0262] For step S1103, when the input of the first multi-task neural network model is the feature vector of the current input phonemes and the first phoneme of the previous input phonemes, the feature vector of the current input phonemes can be extracted, the feature vector of the current input phonemes and the feature vector of the first phoneme of the previous input phonemes can be concatenated, and the terminal device can perform convolution operation to further extract the feature vector and predict the result according to the extracted feature vector. For example, the current input phonemes are the second to the ninth phonemes and the feature vector of the first phoneme. The terminal device can extract the feature vector of the second to the ninth phonemes and concatenate the feature vector of the first phoneme and the feature of the second to the ninth phonemes. For the concatenated feature vector, the terminal device can perform convolution operation to further extract the feature vector and predict the result according to the extracted feature vector. The terminal device can also perform a cutting operation on the concatenated feature vector to cut out the feature vector corresponding to the second phoneme as the input for the next prediction.

[0263] Returning to step S1100, if the input of all to-be-recognized phonemes has been completed, the terminal device can perform step S1104 to predict the characters and punctuation of the current input phonemes to obtain the final result of the current input phonemes. At this time, the terminal device can determine whether there is a stored final result. If the length of all to-be-recognized phonemes is not less than the receptive field, the terminal device has stored the final result of the previous part of the phonemes. If the length of all to-be-recognized phonemes is less than the receptive field, the terminal device has not stored the final result.

[0264] If the terminal device judges that there is a stored final result, the terminal device can perform step S1105, fuses the stored final result and the final result of the current input phoneme to obtain the final result of the phoneme to be recognized, and ends the loop. The specific fusion method can be to splice the final result of the current input phoneme and the stored final result to obtain the final result of the phoneme to be recognized. If the terminal device judges that there is no stored final result, the terminal device can perform step S1106, takes the final result of the current input phoneme as the final result of the phoneme to be recognized, and ends the loop.

[0265] For example, still taking the above example, the current input phonemes are the eigenvectors of the 8th, 9th, 10th, 11th, 12th, 13th, 14th, 15th phonemes and the 7th phoneme. The terminal device can judge that the input of all phonemes to be recognized has been completed, perform step S1104, and predict the characters and punctuation of the 8th-15th phonemes to obtain the final results of the 8th-15th phonemes. The terminal device can judge that the final results of the 1st-7th phonemes have been stored. Therefore, the terminal device can fuse the final results of the 1st-7th phonemes and the final results of the 8th-15th phonemes to obtain the final results of the 1st-15th phonemes.

[0266] According to the speech recognition method of the above-mentioned embodiments of the present application, the phonemes to be recognized output by the acoustic model are sent into the first multi-task neural network model with a streaming network structure in a loop, so that the prediction result of the phonemes to be recognized not only refers to the previous phonemes but also refers to the subsequent phonemes, and the prediction accuracy is improved.

[0267] In addition, the previous input is sent into the network as a buffer by using the streaming network structure, the model calculation amount is reduced, and fast inference is realized. Specifically, since the CNN (Convolutional Neural Networks) has a receptive field, for example, it is assumed that the total number of convolutional layers is 7, and the actual receptive field is 15. The receptive field is referred to the center position, and 7 on the left and right are needed. The streaming network structure caches all the previous 7, and caches the historical features through the buffer of each layer. Therefore, 8 can be calculated each time, and the actual receptive field is 8, which can reduce the calculation amount compared with the receptive field of 15.

[0268] Moreover, the punctuation prediction and the character prediction are integrated into one model, which can ensure that the characters are generated in real time while the punctuation is also generated in real time, and the punctuation prediction does not need to wait until all the speech recognition results are completed, and the characters and punctuation can be output simultaneously. In addition, the multi-task neural network model is smaller in size than the traditional acoustic model, and can be deployed on the terminal side.

[0269] The speech recognition method of the present application will be further described below in combination with a specific application example.

[0270] Taking the input of "chun1 mian2 bu4 jue2 xiao3 chu4 chu4 wen2 ti2 niao3" as an example, when inputting cyclically, first input 'chun1' into the neural network model. The length of the currently input phoneme is 1, which is less than the receptive field of 8, and the predicted temporary result is '春。'. Input 'chun1 mian2' into the neural network model. The length of the currently input phoneme is 2, which is less than the receptive field of 8, and the predicted temporary result is '春眠。'. Input 'chun1 mian2 bu4 jue2 xiao3' into the neural network model. The length of the currently input phoneme is 5, which is less than the receptive field of 8, and the predicted temporary result is '春眠不觉晓。'.

[0271] Since the receptive field of the model is 8, the result obtained when the receptive field is not satisfied is a temporary result. When the input meets (is not less than) the receptive field, that is, when inputting 'chun1 mian2 bu4 jue2 xiao3 chu4 chu4 wen2' into the neural network model, the length of the currently input phoneme is 8, which is equal to the receptive field of 8, and the output is '春眠不觉晓,处处闻。'. At this time, the result predicted for the first input phoneme 'chun1' is the final result, and the character and punctuation corresponding to the phoneme 'chun1' are stored as the final result.

[0272] During the next inference, the feature vector of the phoneme 'chun1' is used as the buffer and input into the neural network model. When inputting'mian2bu4 jue2 xiao3 chu4 chu4 wen2 ti2' into the neural network model, the length of the currently input phoneme is 8, which is equal to the receptive field of 8, and the output is '眠不觉晓,处处闻啼。'. At this time, the character and punctuation predicted for the first input phoneme'mian2' are stored as the final result.

[0273] The feature vector of the phoneme'mian' is used as the buffer and input into the neural network model. The currently input phoneme is 'bu4jue2 xiao3 chu4 chu4 wen2 ti2 niao3', and the prediction result is '不觉晓,处处闻啼鸟。'. Since there are no more phonemes generated at this time, therefore, the character and punctuation predicted for the currently input phoneme are the final result, which is fused with the previously saved result to obtain the final output "春眠不觉晓,处处闻啼鸟。".

[0274] In a possible implementation, the terminal device can store the predicted temporary result in a buffer, and the terminal device can be pre-configured with a number of temporary buffers (preset number), and the preset number can be the same as the receptive field size. In this way, the terminal device can also determine whether the length of the current input phoneme is less than the receptive field by determining whether the preset number of buffers is full. If the preset number of buffers is not full, the length of the current input phoneme is less than the receptive field. If the preset number of buffers is full, the length of the current input phoneme is not less than the receptive field. In this implementation, the process of determining whether the length of the current input phoneme is less than the receptive field can be performed after the prediction of the current input phoneme to obtain the prediction result.

[0275] Figure 12 A flowchart of a speech recognition method according to an embodiment of the present application is shown. In the speech recognition method of the present embodiment, the following steps can be included:

[0276] In step S1200, it is determined whether the input of all phonemes to be recognized has been completed. If the input of all phonemes to be recognized has not been completed, step S1201 is performed. If the input of all phonemes to be recognized has been completed, step S1204 is performed.

[0277] In step S1201, the characters and punctuation of the current input phoneme are predicted to obtain a temporary result of the current input phoneme, the temporary result is stored in a temporary buffer, and it is determined whether the temporary buffer is full. If the temporary buffer is not full, step S1202 is performed. If the temporary buffer is full, step S1203 is performed.

[0278] In step S1202, the next phoneme is input, and the process returns to step S1200.

[0279] In step S1203, the prediction result of the first phoneme of the current input phoneme is taken as the final result, and the terminal device can store the final result. The terminal device inputs the feature vector of the first phoneme, the phonemes in the current input phoneme except the first phoneme, and the next phoneme in the phonemes to be recognized into the first multi-task neural network model, and the process returns to step S1100.

[0280] In step S1204, the characters and punctuation of the current input phoneme are predicted to obtain a final result of the current input phoneme, and it is determined whether there is a stored final result. If there is a stored final result, step S1205 is performed. If there is no stored final result, step S1206 is performed.

[0281] In step S1205, the stored final result and the final result of the current input phoneme are fused to obtain a final result of the phonemes to be recognized, and the loop is ended.

[0282] Step S1206, the final result of the current input phoneme is taken as the final result of the phoneme to be recognized, and the loop is ended.

[0283] In the above embodiment, steps S1201-S1203 and steps S1101-S1103 are different in part judging whether the order of the receptive field and the predicted order are satisfied. Other contents can be referred to Figure 11 Part of the explanation.

[0284] In another possible implementation, the first multi-task neural network model can also be a non-streaming network structure. The terminal device can sequentially input the phonemes to be recognized, and no longer loop input the phonemes whose results have been predicted. Compared with the streaming network structure, the non-streaming network structure does not need to cache the historical results that have been predicted, reduces the occupied memory space, and can further reduce the size of the neural network model.

[0285] For the non-streaming network structure, outputting the first predicted result by using the first multi-task neural network model can include: outputting the first predicted result by using the first multi-task neural network model based on the relationship between the total length of the phonemes to be recognized and the phoneme length threshold. Specifically, it can include the following steps:

[0286] 1. If the total length of the phonemes to be recognized is less than the phoneme length threshold, output the first predicted result by using the first multi-task neural network model according to all the phonemes to be recognized;

[0287] 2. If the total length of the phonemes to be recognized is not less than the phoneme length threshold, before completing the input of all the phonemes to be recognized into the first multi-task neural network model: 2.1 if the length of the current input phoneme is less than the phoneme length threshold, the terminal device continues to input the next phoneme; 2.2 if the length of the current input phoneme is not less than the phoneme length threshold, the terminal device obtains the second predicted result of the first phoneme of the current input phoneme according to the characters and punctuation of the current input phoneme and stores the second predicted result, and the terminal device continues to input the phonemes other than the first phoneme in the current input phoneme and the next phoneme in the phonemes to be recognized into the first multi-task neural network model; 2.3 if the total length of the phonemes to be recognized is not less than the phoneme length threshold, when completing the input of all the phonemes to be recognized into the first multi-task neural network model, the second predicted result of the current input phoneme is obtained according to the characters and punctuation of the current input phoneme; 2.4 if there is no stored second predicted result, the second predicted result of the current input phoneme is taken as the first predicted result of the phonemes to be recognized; 2.5 if there is a stored second predicted result, the first predicted result of the phonemes to be recognized is obtained according to the second predicted result of the current input phoneme and the stored second predicted result.

[0288] The terminal device can set a phoneme length threshold. When the total length of the phonemes to be recognized is less than the phoneme length threshold, the terminal device can input the phonemes to be recognized into the first multi-task neural network model for inference, and the obtained prediction result is taken as the final result. When the total length of the phonemes to be recognized is greater than the phoneme length threshold, the terminal device can input the phonemes to be recognized into the first multi-task neural network model one by one for inference. When the length of the currently input phoneme is not less than the phoneme length threshold, the prediction result of the first phoneme of the currently input phoneme is taken as the final result and stored, and the next phoneme to be recognized is continuously input into the first multi-task neural network model for inference. Until the last phoneme to be recognized is input, the prediction result of the currently input phoneme is taken as the final result. The final result of the currently input phoneme and the stored final result are fused to obtain the final result of the phonemes to be recognized.

[0289] Figure 13 A flowchart of a speech recognition method according to an embodiment of the present application is shown. As shown in the figure, the speech recognition method of the present embodiment can include the following steps: Figure 13

[0290] Step S1300: Determine whether the input of all phonemes to be recognized has been completed. If the input of all phonemes to be recognized has not been completed, execute step S1301. If the input of all phonemes to be recognized has been completed, execute step S1304.

[0291] Step S1301: Determine whether the length of the currently input phoneme is less than the phoneme length threshold. If the length of the currently input phoneme is less than the phoneme length threshold, execute step S1302. If the length of the currently input phoneme is not less than the phoneme length threshold, execute step S1303.

[0292] Step S1302: Predict the characters and punctuation of the currently input phoneme to obtain a temporary result of the currently input phoneme, and continue to input the next phoneme, and return to step S1300.

[0293] Step S1303: Predict the characters and punctuation of the currently input phoneme to obtain the final result of the first phoneme of the currently input phoneme, and the terminal device can store the final result. The terminal device continues to input the phonemes other than the first phoneme in the currently input phoneme and the next phoneme to be recognized into the first multi-task neural network model. Return to step S1300.

[0294] Step S1304: Predict the characters and punctuation of the currently input phoneme to obtain the final result of the currently input phoneme, and determine whether there is a stored final result. If there is a stored final result, execute step S1305. If there is no stored final result, execute step S1306.​

[0295] In step S1305, the final result of the stored final result and the final result of the current input phoneme are fused to obtain the final result of the phoneme to be recognized, and the cycle is ended.

[0296] In step S1306, the final result of the current input phoneme is taken as the final result of the phoneme to be recognized, and the cycle is ended.

[0297] Compared with the embodiment of Figure 11 In the embodiment of Figure 13 In the embodiment of the application, the terminal device judges whether the length of the current input phoneme is less than a phoneme length threshold value, and when performing character and punctuation prediction on the phoneme, the phonemes after the phoneme are referred to, and the number of the referred phonemes is the phoneme length threshold value. In the embodiment of the application, the terminal device can set the phoneme length threshold value to 32. It can be understood that the size of the phoneme length threshold value can be set according to actual needs, and the application does not make specific limitations thereto.

[0298] In step S1303, if the length of the current input phoneme is not less than the phoneme length threshold value, the terminal device saves the final result of the first phoneme of the current input phoneme, but does not take the feature vector of the first phoneme as the input of the next inference. Instead, the phonemes in the current input phoneme except the first phoneme and the next phoneme in the phoneme to be recognized are input into the first multi-task neural network model for inference.

[0299] For example, assuming that the phoneme length threshold value is 32, when the total length of the phonemes to be recognized is less than 32, the terminal device inputs the phonemes to be recognized into the first multi-task neural network model one by one. When the length of all the current input phonemes is less than 32, when a new phoneme to be recognized is input, the terminal device uses the first multi-task neural network model to determine the temporary result of the current input phoneme to be recognized, and refreshes the temporary result of the phoneme before the current input phoneme to be recognized according to all the current input phonemes to be recognized. The above process is repeated until all the phonemes to be recognized are input into the first multi-task neural network model, and the result obtained by inference is the final result.

[0300] If the input is 'chun1 mian2 bu3 jue2 xiao1 chu1 chu1 wen1 ti2 niao0', when the input is 'chun1', the actual vector input to the first multi-task neural network model is [chun1, 0, 0, 0…, 0], and 0 is padded to 32 positions. The output is [chun, 0, 0…, 0]. When the next phoneme'mian2' is input, 'chun1 mian2' can be input into the first multi-task neural network model together, the input is [chun1, mian2, 0, 0,…, 0], and the output is [chun, mian, 0, 0…, 0], the result obtained will refresh the original temporary result of the phoneme "chun"… Repeat the above process until [chun1, mian2, bu3, jue2, xiao1, chu1, chu1, wen1, ti2, niao0, 0,…, 0] is input into the first multi-task neural network model, and the final result is obtained.

[0301] When the total length of the phonemes to be recognized is not less than 32, the phonemes to be recognized are input into the first multi-task neural network model one by one. When the length of the currently input phonemes is not less than 32, the process is the same as the process when the total length is less than 32, which will not be repeated. When the length of the currently input phonemes is not less than 32, the prediction result of the first phoneme of the currently input phonemes is saved as the final result, and the phonemes other than the first phoneme in the currently input phonemes and the next phonemes in the phonemes to be recognized are input into the first multi-task neural network model for further inference… Repeat this process until the last phoneme of the phonemes to be recognized is input and the final result is obtained by inference. The final result saved before is fused with the last 32 final results to obtain the final result of the phonemes to be recognized.

[0302] According to the speech recognition method of the above-mentioned embodiments of the present application, a non-streaming network structure is adopted, and there is no need to re-input the phonemes whose results have been predicted into the network model. Compared with the streaming network structure, the non-streaming network structure does not need to cache the historical results that have been predicted, reduces the occupied memory space, can further reduce the size of the neural network model, and is easy to deploy on the end side. Moreover, compared with the streaming network structure, although the non-streaming network structure has a large amount of calculation, there is no splicing, splitting and other operators in the network, and there is no time-consuming operation such as memory transfer, which can quickly infer in a high-parallel computing device such as a GPU.

[0303] According to the above construction from the training sample set to the model inference process, it can be known from the introduction of the speech recognition method of the present application that, in order to solve the technical problems that the model cannot be deployed on the end side and the prediction accuracy of the acoustic model for predicting punctuation is not high in related speech recognition technologies, the speech recognition method of an embodiment provided by the present application specifically includes the following steps:

[0304] The terminal device inputs the phonemes to be identified into a first multi-task neural network model, wherein the first multi-task neural network model is obtained by training a second multi-task neural network model using training samples. The training samples include: sample sentences, the sample sentences include characters, and the training samples include: phonemes and punctuation marks corresponding to the characters in the sample sentences.

[0305] The terminal device uses the first multi-task neural network model to output a first prediction result, which includes the character prediction result and punctuation prediction result corresponding to the phoneme to be identified.

[0306] The terminal device displays at least a portion of the first prediction result on its display screen based on the first prediction result.

[0307] For details, please refer to the above text. Figure 10 The process of the aforementioned speech recognition method will not be described in detail here.

[0308] The speech recognition method of this application constructs a neural network model for simultaneously predicting the characters and punctuation corresponding to phonemes, and trains the neural network model by constructing a training sample set to obtain the trained neural network model. During the training process, word segmentation is not required. The phonemes (vectors) converted from the speech to be recognized are used as the input of the trained neural network model for forward inference, which can simultaneously output the characters and punctuation corresponding to the phonemes. Furthermore, the neural network model is small in size and can be deployed on the edge.

[0309] To further address the technical problem in related technologies where the lengths of the input and output are different, preventing the simultaneous output of prediction results, in the embodiments of this application, the lengths of characters, phonemes, and punctuation marks in the sample sentences of the constructed training samples are the same. By aligning the lengths of characters in the sample sentences with the lengths of the phonemes and punctuation marks after phonetic transcription during the construction of the training sample set, the neural network model, after being trained using the training sample set constructed according to the embodiments of this application, can simultaneously perform phoneme-to-character conversion and punctuation prediction, thereby simultaneously outputting the predicted character and punctuation results.

[0310] In a possible implementation, the first multi-task neural network model is a streaming network structure, the terminal device inputs the phonemes to be recognized into the first multi-task neural network model, and outputs the first prediction result by using the first multi-task neural network model, which can include that the terminal device inputs the phonemes to be recognized into the first multi-task neural network model in a loop, and outputs the first prediction result by using the first multi-task neural network model based on the length of the phonemes to be recognized that are currently input. In this way, the prediction result of the phonemes to be recognized not only refers to the previous phonemes, but also refers to the subsequent phonemes, thereby improving the accuracy of prediction.

[0311] In a possible implementation, the terminal device inputs the phonemes to be recognized into the first multi-task neural network model in a loop, and outputs the first prediction result by using the first multi-task neural network model based on the length of the phonemes to be recognized that are currently input, which can include that:

[0312] Before all the phonemes to be recognized are input into the first multi-task neural network model, if the length of the phonemes that are currently input is less than the receptive field, the terminal device continues to input the next phonemes;

[0313] Before all the phonemes to be recognized are input into the first multi-task neural network model, if the length of the phonemes that are currently input is not less than the receptive field, the terminal device obtains the second prediction result of the first phoneme of the phonemes that are currently input according to the character and the punctuation of the phonemes that are currently input, and stores the second prediction result; and the terminal device inputs the feature vector of the first phoneme, the phonemes other than the first phoneme in the phonemes that are currently input, and the next phonemes in the phonemes to be recognized into the first multi-task neural network model.

[0314] In a possible implementation, the terminal device inputs the phonemes to be recognized into the first multi-task neural network model in a loop, and outputs the first prediction result by using the first multi-task neural network model based on the length of the phonemes to be recognized that are currently input, which can further include that:

[0315] When all the phonemes to be recognized are input into the first multi-task neural network model, the terminal device obtains the second prediction result of the phonemes that are currently input according to the character and the punctuation of the phonemes that are currently input;

[0316] If there is no stored second prediction result, the second prediction result of the phonemes that are currently input is taken as the first prediction result of the phonemes to be recognized;

[0317] If there is the stored second prediction result, the first prediction result of the phonemes to be recognized is obtained according to the second prediction result of the phonemes that are currently input and the stored second prediction result.

[0318] For the process of reasoning with a streaming network structure, specific examples can be found in the foregoing Figure 11 and Figure 12 The foregoing description of the reasoning process of the partial description, it should be noted that, Figure 11 and Figure 12 The reasoning process is only some examples of the present application, the execution order of the steps and the specific values do not limit the application in any way.

[0319] According to the voice recognition method of the above-mentioned embodiment of the application, the phonemes to be recognized output by the acoustic model are sent into the first multi-task neural network model of the streaming network structure in a cycle, so that the prediction result of the phonemes to be recognized not only refers to the previous phonemes, but also refers to the subsequent phonemes, thereby improving the prediction accuracy.

[0320] In a possible implementation, the first multi-task neural network model is a non-streaming network structure, and the terminal device outputs the first prediction result by using the first multi-task neural network model, which can include that the terminal device outputs the first prediction result by using the first multi-task neural network model based on the relationship between the total length of the phonemes to be recognized and the phoneme length threshold.

[0321] In a possible implementation, the terminal device outputs the first prediction result by using the first multi-task neural network model based on the relationship between the total length of the phonemes to be recognized and the phoneme length threshold, which can include that:

[0322] If the total length of the phonemes to be recognized is less than the phoneme length threshold, the first multi-task neural network model is used to output the first prediction result according to all the phonemes to be recognized.

[0323] In a possible implementation, the terminal device outputs the first prediction result by using the first multi-task neural network model based on the relationship between the total length of the phonemes to be recognized and the phoneme length threshold, which can further include that:

[0324] If the total length of the phonemes to be recognized is not less than the phoneme length threshold, before all the phonemes to be recognized are input into the first multi-task neural network model: if the length of the currently input phoneme is less than the phoneme length threshold, the terminal device continues to input the next phoneme; if the length of the currently input phoneme is not less than the phoneme length threshold, the terminal device obtains the second prediction result of the first phoneme of the currently input phoneme according to the characters and punctuation of the currently input phoneme and stores the second prediction result, and the terminal device continues to input the phonemes other than the first phoneme in the currently input phoneme and the next phoneme in the phonemes to be recognized into the first multi-task neural network model.

[0325] In a possible implementation, the first prediction result is output based on a relationship between a total length of the phoneme to be recognized and a phoneme length threshold by using the first multi-task neural network model, and the method can further include the following steps:

[0326] If the total length of the phoneme to be recognized is not less than the phoneme length threshold, when all the phonemes to be recognized are input into the first multi-task neural network model, a second prediction result of the phoneme currently input is obtained according to the character and the punctuation of the phoneme currently input.

[0327] If there is no stored second prediction result, the second prediction result of the phoneme currently input is taken as the first prediction result of the phoneme to be recognized.

[0328] If there is a stored second prediction result, the first prediction result of the phoneme to be recognized is obtained according to the second prediction result of the phoneme currently input and the stored second prediction result.

[0329] As to the process of inference by using the non-streaming network structure, specific examples can be referred to the description of the process of inference by using the non-streaming network structure in the foregoing Figure 13 It should be noted that, Figure 13 The process of inference is only some examples of the present application, and the execution order of the steps and the specific values do not limit the present application in any way.

[0330] According to the speech recognition method of the foregoing embodiments of the present application, the non-streaming network structure is used, and the phoneme whose result has been predicted does not need to be input into the network model again. Compared with the streaming network structure, the non-streaming network structure does not need to cache the historical result that has been predicted, reduces the occupied memory space, can further reduce the size of the neural network model, and is easy to deploy on the edge side. Moreover, since no splicing, segmentation or other operations are needed on the historical result and the phoneme currently input in the calculation process, the inference speed can be accelerated, and in long speech recognition, the effect of real-time output is remarkable.

[0331] The present application also provides a neural network model training method, which includes the following steps:

[0332] Constructing a training sample, the training sample including a sample sentence including characters, and the training sample further including phonemes corresponding to the characters in the sample sentence and punctuation;

[0333] Training the second multi-task neural network model by using the training sample to obtain a first multi-task neural network model; wherein the second multi-task neural network model and the first multi-task neural network model can both output a first prediction result and display at least part of the first prediction result, and the first prediction result includes a character prediction result and a punctuation prediction result, and the characters and the punctuation of the phoneme are predicted simultaneously.

[0334] The neural network model training method of the embodiments of the present application trains a neural network model for simultaneously predicting characters and punctuation corresponding to phonemes, and trains the neural network model by constructing a training sample set, to obtain a trained neural network model. In the training process, no word segmentation processing is required. The converted phonemes (vectors) of the speech to be recognized are used as the input of the trained neural network model, and forward reasoning is performed, so that the characters and punctuation corresponding to the phonemes can be simultaneously output. Moreover, the neural network model has a small size and can be deployed on an end side.

[0335] To further solve the technical problem that the prediction results cannot be simultaneously output in the scenario where the lengths of the input sequence and the output sequence are different in the related art, in a possible implementation, constructing a training sample can include:

[0336] The characters in the sample sentence are phonetized to obtain phonemes corresponding to the characters according to a phonetic dictionary, and the phonemes corresponding to the characters, the characters and the punctuation are aligned.

[0337] In the process of constructing the training sample set, the length of the characters in the sample sentence is aligned with the length of the phonemes after phonetization and the length of the punctuation. After the neural network model is trained by using the training sample set constructed by the embodiments of the present application, the neural network model can simultaneously perform phoneme-to-character conversion and punctuation prediction, so that the predicted characters and punctuation results can be simultaneously output.

[0338] In a possible implementation, the phonemes corresponding to multi-syllable words in the aligned Chinese are any one of the multiple phonemes corresponding to the multi-syllable words; the aligned English characters include alignment characters, and the length of the aligned English characters is the same as the length of the phonemes corresponding to the English characters; for the characters without punctuation before alignment, the aligned punctuation is blank. Specifically, the alignment processing of the phonemes corresponding to the characters, the characters and the punctuation can include:

[0339] For multi-syllable words in Chinese, any one of the multiple phonemes corresponding to the multi-syllable words is selected as the phoneme corresponding to the multi-syllable word;

[0340] For English characters, the length of the phonemes corresponding to the characters is aligned by adding alignment characters in the characters;

[0341] If there is no punctuation after the characters, the punctuation corresponding to the characters is set as blank, so that the length of the punctuation is aligned with the length of the characters.

[0342] For English characters, the position where the alignment character is added in the character can be located at both sides of the character to be aligned, such as before or after. That is, the character and the phoneme can be left-aligned or right-aligned. The right alignment can be that the alignment character is added to the left side of the character to be aligned, and the left alignment can be that the alignment character is added to the right side of the character to be aligned. For the form of the alignment character and the adding manner, please refer to the above Figure 7 The introduction of the part is not repeated here. In addition, the three steps of the above alignment processing can be performed respectively or simultaneously, and the present application does not limit this.

[0343] In a possible implementation, training the second multi-task neural network model by using the training sample to obtain the first multi-task neural network model can include:

[0344] inputting the training sample into the second multi-task neural network model to determine a character probability matrix and a punctuation probability matrix corresponding to the training sample;

[0345] calculating a character cross-entropy loss and a punctuation cross-entropy loss according to the character probability matrix and the punctuation probability matrix, respectively;

[0346] calculating a weighted cross-entropy loss according to the character cross-entropy loss, a first weight corresponding to the character cross-entropy loss, the punctuation cross-entropy loss, and a second weight corresponding to the punctuation cross-entropy loss;

[0347] adjusting parameters of the second multi-task neural network model according to the weighted cross-entropy loss to obtain the trained first multi-task neural network model.

[0348] For the specific introduction of the training process, please refer to the content of the above Figure 8 part, which is not repeated here.

[0349] The training method of the multi-task neural network model of the present application can realize the training of the tasks of character prediction and punctuation prediction at the same time. In addition, since the training sample set constructed includes multiple languages, the training method of the multi-task neural network model of the present application can also realize the training of the task of recognizing (predicting) multiple languages. The multi-task neural network model trained by the training method of the multi-task neural network model according to the embodiment of the present application can simultaneously predict multiple languages and punctuation, and the multi-task neural network model is smaller in size compared with a traditional acoustic model, and can be deployed on an end side.

[0350] The embodiment of the present application also provides a speech recognition device, which can be applied to a terminal device as shown in Figure 1 or Figure 3 . Figure 14 shows a block diagram of a speech recognition device according to an embodiment of the present application, as shown inFigure 14 The speech recognition apparatus can include:

[0351] The input module 1400 is configured to input phonemes to be recognized into a first multi-task neural network model, wherein the first multi-task neural network model is obtained by training a second multi-task neural network model using training samples, and the training samples include sample sentences, and the sample sentences include characters, and the training samples further include phonemes corresponding to the characters in the sample sentences and punctuation marks.

[0352] The inference module 1401 is configured to output a first prediction result using the first multi-task neural network model, wherein the first prediction result includes character prediction results and punctuation mark prediction results corresponding to the phonemes to be recognized.

[0353] The display module 1402 is configured to display at least part of the first prediction result on a display screen of the terminal device according to the first prediction result.

[0354] The speech recognition apparatus of the embodiments of the present application can simultaneously predict and output characters and punctuation marks corresponding to phonemes by constructing a neural network model for simultaneously predicting and outputting characters and punctuation marks corresponding to phonemes, and constructing a training sample set to train the neural network model, to obtain a trained neural network model. In the training process, no word segmentation processing is required. The converted phonemes (vectors) of the speech to be recognized are used as the input of the trained neural network model, and forward inference is performed. The characters and punctuation marks corresponding to the phonemes can be simultaneously output. Moreover, the neural network model has a small size and can be deployed on an end side.

[0355] In a possible implementation, the length of the character in the sample sentence is the same as the length of the phoneme and the length of the punctuation mark.

[0356] In the process of constructing the training sample set, the length of the character in the sample sentence is aligned with the length of the phoneme after annotation and the length of the punctuation mark. After the neural network model is trained using the training sample set constructed by the embodiments of the present application, the neural network model can simultaneously perform phoneme-to-character conversion and punctuation mark prediction, so that the predicted character and punctuation mark results can be simultaneously output.

[0357] In a possible implementation, the first multi-task neural network model is a streaming network structure, and the input module 1400 can include a first input unit configured to cyclically input the phonemes to be recognized into the first multi-task neural network model. The inference module 1401 includes a first inference unit configured to output the first prediction result based on the length of the currently input phonemes to be recognized using the first multi-task neural network model. The prediction result of the phonemes to be recognized not only refers to the previous phonemes, but also refers to the subsequent phonemes, thereby improving the prediction accuracy.

[0358] In a possible implementation, the first input unit is further configured to: before all the to-be-recognized phonemes are input into the first multi-task neural network model, if the length of the currently input phoneme is less than the receptive field, the terminal device continues to input the next phoneme.

[0359] In a possible implementation, the first inference unit is further configured to: before all the to-be-recognized phonemes are input into the first multi-task neural network model, if the length of the currently input phoneme is not less than the receptive field, the first inference unit obtains a second prediction result of a first phoneme of the currently input phoneme according to the character and the punctuation of the currently input phoneme, and stores the second prediction result; and the terminal device continues to input the feature vector of the first phoneme, the phonemes other than the first phoneme in the currently input phoneme, and the next phoneme in the to-be-recognized phonemes into the first multi-task neural network model.

[0360] In a possible implementation, the first inference unit is further configured to: when all the to-be-recognized phonemes are input into the first multi-task neural network model, obtain a second prediction result of the currently input phoneme according to the character and the punctuation of the currently input phoneme.

[0361] If there is no stored second prediction result, the second prediction result of the currently input phoneme is taken as the first prediction result of the to-be-recognized phoneme.

[0362] If there is a stored second prediction result, the first prediction result of the to-be-recognized phoneme is obtained according to the second prediction result of the currently input phoneme and the stored second prediction result.

[0363] According to the speech recognition apparatus in the above-mentioned embodiments of the present application, the to-be-recognized phonemes output by the acoustic model are cyclically input into the first multi-task neural network model in a streaming network structure, so that the prediction result of the to-be-recognized phoneme not only refers to the previous phoneme, but also refers to the subsequent phoneme, thereby improving the accuracy of prediction.

[0364] In a possible implementation, the first multi-task neural network model is a non-streaming network structure, and the inference module 1401 includes:

[0365] The second inference unit is configured to output the first prediction result based on the relationship between the total length of the to-be-recognized phonemes and the phoneme length threshold value by using the first multi-task neural network model.

[0366] In a possible implementation, the second inference unit is further configured to: if the total length of the to-be-recognized phonemes is less than the phoneme length threshold value, output the first prediction result according to all the to-be-recognized phonemes by using the first multi-task neural network model.

[0367] In a possible implementation, the second inference unit is further configured to:

[0368] If the total length of the phonemes to be recognized is not less than the phoneme length threshold, before completing inputting all the phonemes to be recognized into the first multi-task neural network model: if the length of the currently input phoneme is less than the phoneme length threshold, the next phoneme is continuously input; if the length of the currently input phoneme is not less than the phoneme length threshold, the second prediction result of the first phoneme of the currently input phoneme is obtained according to the character and punctuation of the currently input phoneme and stored, and the terminal device continues to input the phonemes other than the first phoneme in the currently input phoneme and the next phoneme in the phonemes to be recognized into the first multi-task neural network model.

[0369] In a possible implementation, the second inference unit is further configured to:

[0370] If the total length of the phonemes to be recognized is not less than the phoneme length threshold, when completing inputting all the phonemes to be recognized into the first multi-task neural network model, the second prediction result of the currently input phoneme is obtained according to the character and punctuation of the currently input phoneme;

[0371] If there is no stored second prediction result, the second prediction result of the currently input phoneme is taken as the first prediction result of the phonemes to be recognized;

[0372] If there is the stored second prediction result, the first prediction result of the phonemes to be recognized is obtained according to the second prediction result of the currently input phoneme and the stored second prediction result.

[0373] According to the speech recognition apparatus in the above embodiments of the present application, a non-stream network structure is adopted, and there is no need to re-input the phonemes whose results have been predicted into the network model. Compared with a stream network structure, the non-stream network structure does not need to cache the historical results that have been predicted, reduces the occupied memory space, can further reduce the size of the neural network model, and is easy to deploy on the terminal side. Moreover, since there is no need to perform splicing, segmentation and other operations on the historical results and the currently input phonemes in the calculation process, the inference speed can be accelerated, and in long speech recognition, the effect of real-time output is remarkable.

[0374] The embodiments of the present application further provide a neural network model training apparatus. Figure 15 A block diagram of the neural network model training apparatus according to an embodiment of the present application is shown in FIG. 1. Figure 15 As shown in FIG. 1, the neural network model training apparatus can include:

[0375] The constructing module 1500 is configured to construct a training sample, the training sample comprising: a sample sentence, the sample sentence comprising characters, and the training sample further comprising: phonemes corresponding to the characters in the sample sentence, and punctuation marks;

[0376] The training module 1501 is configured to train the second multi-task neural network model to obtain a first multi-task neural network model by using the training sample; wherein the second multi-task neural network model and the first multi-task neural network model can output a first prediction result, and display at least part of the first prediction result, the first prediction result comprising character prediction results and punctuation mark prediction results, and the phonemes, the characters and the punctuation marks are predicted simultaneously.

[0377] The neural network training apparatus of the embodiments of the present application trains the neural network model by constructing a neural network model capable of simultaneously performing phoneme-to-character conversion and punctuation mark prediction, and constructing a training sample set to train the neural network model, to obtain a trained neural network model. In the training process, no word segmentation processing is required. The converted phonemes (vectors) of the to-be-recognized speech are used as the input of the trained neural network model, and forward reasoning is performed, so that the phonemes corresponding to the characters and the punctuation marks can be output simultaneously. Moreover, the neural network model has a small size, and can be deployed on an end side. In a possible implementation manner, the constructing module 1500 comprises:

[0378] The alignment unit is configured to perform phonetic transcription on the characters in the sample sentence according to a phonetic transcription dictionary to obtain the phonemes corresponding to the characters, and perform alignment processing on the phonemes corresponding to the characters, the characters and the punctuation marks. The length of the characters in the sample sentence is the same as the length of the phonemes and the length of the punctuation marks.

[0379] In a possible implementation manner, the phonemes corresponding to a multi-sound character in the aligned Chinese are any one of the multiple phonemes corresponding to the multi-sound character; the aligned English character comprises an alignment character, and the length of the aligned English character is the same as the length of the phonemes corresponding to the English character; and for a character without punctuation marks before alignment, the punctuation mark after alignment is blank. Specifically, the alignment unit is further configured to:

[0380] For a multi-sound character in Chinese, any one of the multiple phonemes corresponding to the multi-sound character is selected as the phoneme corresponding to the multi-sound character;

[0381] For an English character, an alignment character is added in the character to align the length of the alignment character with the length of the phonemes corresponding to the character;

[0382] If there is no punctuation mark after the character, the punctuation mark corresponding to the character is set as blank, so that the length of the punctuation mark is aligned with the length of the character.

[0383] By aligning the length of the characters in the sample sentence, the length of the phonemes after the annotation, and the length of the punctuation in the process of constructing the training sample set, the neural network model trained by the training sample set constructed by the embodiment of the present application can simultaneously perform phoneme-to-character conversion and punctuation prediction, so as to simultaneously output the predicted character and punctuation results.

[0384] In a possible implementation, the training module 1501 includes:

[0385] A determination unit configured to input a training sample into the second multi-task neural network model, and determine a character probability matrix and a punctuation probability matrix corresponding to the training sample;

[0386] A first calculation unit configured to calculate a character cross-entropy loss and a punctuation cross-entropy loss according to the character probability matrix and the punctuation probability matrix, respectively;

[0387] A second calculation unit configured to calculate a weighted cross-entropy loss according to the character cross-entropy loss, a first weight corresponding to the character cross-entropy loss, the punctuation cross-entropy loss, and a second weight corresponding to the punctuation cross-entropy loss;

[0388] An adjustment unit configured to adjust parameters of the second multi-task neural network model according to the weighted cross-entropy loss, to obtain a trained first multi-task neural network model.

[0389] The training apparatus of the multi-task neural network model of the present application can train the tasks of character prediction and punctuation prediction simultaneously. In addition, since the training sample set constructed includes multiple languages, the training method of the multi-task neural network model of the present application can also train the task of recognizing (predicting) multiple languages. The multi-task neural network model trained by the training apparatus of the multi-task neural network model according to the embodiment of the present application can simultaneously predict multiple languages and punctuation, and the multi-task neural network model is smaller in size than the traditional acoustic model and can be deployed on the terminal side.

[0390] The embodiment of the present application provides a speech recognition apparatus, including a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.

[0391] The embodiment of the present application provides a neural network model training apparatus, including a processor and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions.

[0392] The embodiment of the present application provides a nonvolatile computer readable storage medium, which stores computer program instructions, and the computer program instructions are executed by a processor to realize the method.

[0393] The embodiment of the present application provides a computer program product, which comprises computer readable codes or a nonvolatile computer readable storage medium carrying the computer readable codes, and when the computer readable codes are run in a processor of an electronic device, the processor in the electronic device executes the method.

[0394] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any appropriate combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital video disc (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch cards or punched tape, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se.

[0395] Computer readable program instructions or code described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0396] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0397] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0398] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0399] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0400] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses, systems, methods and computer program products according to various embodiments of the present application. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical functions (s). In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by hardware, firmware, computer software, or their combinations.

[0401] It should also be noted that each of the blocks of the block diagrams and / or flowchart illustrations, and combinations thereof, can be implemented by hardware, firmware, computer software, or their combinations. Such hardware, firmware, computer software, or their combinations can be implemented within one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, micro-controllers, microprocessors, other electronic units designed to perform the functions described herein, or a combination thereof.

[0402] Although the present application is described in conjunction with the preferred embodiments thereof, a variety of modifications and alterations are possible constructed by those skilled in the art upon reading and understanding the present specification and the annexed drawings. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite articles "a" or "an" do not exclude a plurality. A single processor or other unit can fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0403] Having described various embodiments of the application, it is to be understood that the above description is meant not to limit and not to encompass all of the possible embodiments. Many modifications and variations of this application can be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. It is intended that the scope of the application be defined by the scope of the patent and by the claims as allowed by the patent office, which can include adaptations based on the description, equivalents, and / or substitutions of elements individually or collectively to the entire disclosure.

Claims

1. A speech recognition method, characterized in that, The method includes: The terminal device inputs the phonemes to be identified into the first multi-task neural network model; The first prediction result is output using the first multi-task neural network model. The first prediction result includes the character prediction result and punctuation prediction result corresponding to the phoneme to be identified. The terminal device displays at least a portion of the first prediction result on its display screen based on the first prediction result. The first multi-task neural network model is obtained by training the second multi-task neural network model using training samples. The training samples include: sample sentences, which include characters. The training samples also include: phonemes and punctuation marks corresponding to the characters in the sample sentences. The length of the characters in the sample sentences is the same as the length of the phonemes and the length of the punctuation marks.

2. The method according to claim 1, characterized in that, The first multi-task neural network model is a streaming network structure. The terminal device inputs the phoneme to be identified into a first multi-task neural network model and outputs a first prediction result using the first multi-task neural network model, including: the terminal device cyclically feeds the phoneme to be identified into the first multi-task neural network model and outputs the first prediction result based on the length of the currently input phoneme to be identified using the first multi-task neural network model.

3. The method according to claim 2, characterized in that, The terminal device cyclically feeds the phonemes to be identified into a first multi-task neural network model, and uses the first multi-task neural network model to output the first prediction result based on the length of the currently input phonemes to be identified, including: Before inputting all the phonemes to be identified into the first multi-task neural network model, if the length of the currently input phoneme is less than the receptive field, the terminal device continues to input the next phoneme. Before inputting all the phonemes to be identified into the first multi-task neural network model, if the length of the currently input phoneme is not less than the receptive field, the terminal device obtains the second prediction result of the first phoneme of the currently input phoneme based on the characters and punctuation of the currently input phoneme, and stores the second prediction result; the terminal device continues to input the feature vector of the first phoneme, the phonemes other than the first phoneme in the currently input phonemes, and the next phoneme in the phonemes to be identified into the first multi-task neural network model.

4. The method according to claim 2 or 3, characterized in that, The terminal device cyclically feeds the phonemes to be identified into a first multi-task neural network model, and uses the first multi-task neural network model to output the first prediction result based on the length of the currently input phonemes to be identified, further comprising: When all the phonemes to be identified are input into the first multi-task neural network model, the terminal device obtains the second prediction result of the current input phoneme based on the characters and punctuation of the current input phoneme. If there is no stored second prediction result, the terminal device will use the second prediction result of the currently input phoneme as the first prediction result of the phoneme to be identified. If a stored second prediction result exists, then the first prediction result of the phoneme to be identified is obtained based on the second prediction result of the currently input phoneme and the stored second prediction result.

5. The method according to claim 1, characterized in that, The first multi-task neural network model is a non-streaming network structure. The first prediction result is output using the first multi-task neural network model, including: The first multi-task neural network model is used to output the first prediction result based on the relationship between the total length of the phonemes to be identified and the phoneme length threshold.

6. The method according to claim 5, characterized in that, The first multi-task neural network model, based on the relationship between the total length of the phonemes to be identified and the phoneme length threshold, outputs the first prediction result, including: If the total length of the phonemes to be identified is less than the phoneme length threshold, the first multi-task neural network model is used to output the first prediction result based on all the phonemes to be identified.

7. The method according to claim 5 or 6, characterized in that, The first multi-task neural network model, based on the relationship between the total length of the phonemes to be identified and the phoneme length threshold, outputs the first prediction result, including: If the total length of the phonemes to be identified is not less than the phoneme length threshold, before inputting all the phonemes to be identified into the first multi-task neural network model: if the length of the currently input phoneme is less than the phoneme length threshold, the terminal device continues to input the next phoneme; if the length of the currently input phoneme is not less than the phoneme length threshold, the terminal device obtains the second prediction result of the first phoneme of the currently input phoneme based on the character and punctuation of the currently input phoneme and stores the second prediction result. The terminal device then inputs the phonemes other than the first phoneme in the currently input phonemes and the next phoneme in the phonemes to be identified into the first multi-task neural network model.

8. The method according to claim 7, characterized in that, The first multi-task neural network model, based on the relationship between the total length of the phonemes to be identified and the phoneme length threshold, outputs the first prediction result, and further includes: If the total length of the phonemes to be identified is not less than the phoneme length threshold, when all the phonemes to be identified are input into the first multi-task neural network model, the second prediction result of the current input phoneme is obtained based on the characters and punctuation of the current input phoneme. If there is no stored second prediction result, the second prediction result of the currently input phoneme will be used as the first prediction result of the phoneme to be identified. If a stored second prediction result exists, then the first prediction result of the phoneme to be identified is obtained based on the second prediction result of the currently input phoneme and the stored second prediction result.

9. A method for training a neural network model, characterized in that, The method includes: Construct training samples, which include: sample sentences, which contain characters, and the training samples also include: phonemes and punctuation marks corresponding to the characters in the sample sentences; The training samples are used to train the second multi-task neural network model to obtain the first multi-task neural network model; wherein, both the second multi-task neural network model and the first multi-task neural network model can output a first prediction result and display at least a part of the first prediction result, the first prediction result including character prediction result and punctuation prediction result; The construction of training samples includes: The phonemes corresponding to the characters in the sample sentence are obtained by phonetic transcription of the characters according to the phonetic dictionary, and the phonemes corresponding to the characters are aligned with the characters and punctuation marks. The length of the characters in the sample sentence is the same as the length of the phonemes and the length of the punctuation marks.

10. The method according to claim 9, characterized in that, The phoneme corresponding to a polyphonic character in the aligned Chinese is any one of the multiple phonemes corresponding to the polyphonic character. The aligned English characters include the alignment character, and the length of the aligned English characters is the same as the length of the phonemes corresponding to the English characters; For characters that did not have punctuation before alignment, the aligned punctuation will be blank.

11. The method according to claim 9, characterized in that, The first multi-task neural network model is obtained by training the second multi-task neural network model using the training samples, including: The training samples are input into the second multi-task neural network model to determine the character probability matrix and punctuation probability matrix corresponding to the training samples. Calculate the character cross-entropy loss and the punctuation cross-entropy loss based on the character probability matrix and the punctuation probability matrix, respectively. Calculate the weighted cross-entropy loss based on the character cross-entropy loss, the first weight corresponding to the character cross-entropy loss, the punctuation cross-entropy loss, and the second weight corresponding to the punctuation cross-entropy loss. The parameters of the second multi-task neural network model are adjusted according to the weighted cross-entropy loss to obtain the trained first multi-task neural network model.

12. A voice recognition device, characterized in that, The device includes: The input module is used to input the phonemes to be identified into the first multi-task neural network model; The inference module is used to output a first prediction result using the first multi-task neural network model. The first prediction result includes the character prediction result and punctuation prediction result corresponding to the phoneme to be identified. The display module is used to display at least a portion of the first prediction result on the display screen of the terminal device based on the first prediction result. The first multi-task neural network model is obtained by training the second multi-task neural network model using training samples. The training samples include: sample sentences, which include characters. The training samples also include: phonemes and punctuation marks corresponding to the characters in the sample sentences. The length of the characters in the sample sentences is the same as the length of the phonemes and the length of the punctuation marks.

13. The apparatus according to claim 12, characterized in that, The first multi-task neural network model is a streaming network structure. The input module includes: a first input unit, used to cyclically feed the phonemes to be identified into a first multi-task neural network model; The reasoning module includes: The first inference unit is used to output the first prediction result based on the length of the phoneme to be identified in the current input using the first multi-task neural network model.

14. The apparatus according to claim 13, characterized in that, The first input unit is also configured to: before completing the input of all the phonemes to be recognized into the first multi-task neural network model, if the length of the currently input phoneme is less than the receptive field, the terminal device continues to input the next phoneme; Before inputting all the phonemes to be identified into the first multi-task neural network model, if the length of the currently input phoneme is not less than the receptive field, the first inference unit is used to obtain the second prediction result of the first phoneme of the currently input phoneme based on the characters and punctuation of the currently input phoneme, and store the second prediction result; the first input unit is also used to continue to input the feature vector of the first phoneme, the phonemes other than the first phoneme in the currently input phonemes, and the next phoneme in the phonemes to be identified into the first multi-task neural network model.

15. The apparatus according to claim 13 or 14, characterized in that, The first inference unit is also used for: When all the phonemes to be identified are input into the first multi-task neural network model, the second prediction result of the current input phoneme is obtained based on the characters and punctuation of the current input phoneme. If there is no stored second prediction result, then the second prediction result of the currently input phoneme will be used as the first prediction result of the phoneme to be identified. If a stored second prediction result exists, then the second prediction result of the phoneme to be identified is obtained based on the second prediction result of the currently input phoneme and the stored second prediction result.

16. The apparatus according to claim 12, characterized in that, The first multi-task neural network model is a non-streaming network structure. The reasoning module includes: The second inference unit is used to output the first prediction result based on the relationship between the total length of the phonemes to be identified and the phoneme length threshold using the first multi-task neural network model.

17. The apparatus according to claim 16, characterized in that, The second inference unit is further configured to, if the total length of the phonemes to be identified is less than the phoneme length threshold, use the first multi-task neural network model to output the first prediction result based on all the phonemes to be identified.

18. The apparatus according to claim 16 or 17, characterized in that, The second inference unit is also used for: If the total length of the phonemes to be identified is not less than the phoneme length threshold, before inputting all the phonemes to be identified into the first multi-task neural network model: if the length of the currently input phoneme is less than the phoneme length threshold, then continue to input the next phoneme; if the length of the currently input phoneme is not less than the phoneme length threshold, then based on the character and punctuation of the currently input phoneme, obtain the second prediction result of the first phoneme of the currently input phoneme and store the second prediction result, and continue to input the phonemes other than the first phoneme in the currently input phonemes and the next phoneme in the phonemes to be identified into the first multi-task neural network model.

19. The apparatus according to claim 18, characterized in that, The second inference unit is also used for: If the total length of the phonemes to be identified is not less than the phoneme length threshold, when all the phonemes to be identified are input into the first multi-task neural network model, the second prediction result of the current input phoneme is obtained based on the characters and punctuation of the current input phoneme. If there is no stored second prediction result, the second prediction result of the currently input phoneme will be used as the first prediction result of the phoneme to be identified. If a stored second prediction result exists, then the first prediction result of the phoneme to be identified is obtained based on the second prediction result of the currently input phoneme and the stored second prediction result.

20. A neural network model training device, characterized in that, The device includes: A construction module is used to construct training samples, wherein the training samples include: sample sentences, wherein the sample sentences include characters, and the training samples also include: phonemes and punctuation marks corresponding to the characters in the sample sentences; The training module is used to train the second multi-task neural network model using the training samples to obtain the first multi-task neural network model; wherein both the second multi-task neural network model and the first multi-task neural network model are capable of outputting a first prediction result and displaying at least a portion of the first prediction result, the first prediction result including character prediction result and punctuation prediction result; The building module includes: The alignment unit is used to annotate the characters in the sample sentence according to the phonetic dictionary to obtain the phonemes corresponding to the characters, and to align the phonemes corresponding to the characters with the characters and punctuation marks. The length of the characters in the sample sentence is the same as the length of the phonemes and the length of the punctuation marks.

21. The apparatus according to claim 20, characterized in that, The phoneme corresponding to a polyphonic character in the aligned Chinese is any one of the multiple phonemes corresponding to the polyphonic character. The aligned English characters include the alignment character, and the length of the aligned English characters is the same as the length of the phonemes corresponding to the English characters; For characters that did not have punctuation before alignment, the aligned punctuation will be blank.

22. The apparatus according to claim 20, characterized in that, The training module includes: A determining unit is used to input training samples into a second multi-task neural network model and determine the character probability matrix and punctuation probability matrix corresponding to the training samples. The first calculation unit is used to calculate the character cross-entropy loss and the punctuation cross-entropy loss respectively based on the character probability matrix and the punctuation probability matrix. The second calculation unit is used to calculate the weighted cross-entropy loss based on the character cross-entropy loss, the first weight corresponding to the character cross-entropy loss, the punctuation cross-entropy loss, and the second weight corresponding to the punctuation cross-entropy loss. The adjustment unit is used to adjust the parameters of the second multi-task neural network model according to the weighted cross-entropy loss to obtain the trained first multi-task neural network model.

23. A voice recognition device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1-8 when executing the instructions.

24. A neural network model training device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 9-11 when executing the instructions.

25. A non-volatile computer-readable storage medium storing computer program instructions thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-8, or the method described in any one of claims 9-11.

Images