Chinese speech recognition correction method and device, and electronic equipment

Abstract

The application provides a Chinese speech recognition error correction method and device and electronic equipment, which are suitable for the field of speech recognition technology in artificial intelligence, can correct Chinese speech recognition, and the method comprises the following steps: obtaining speech data, processing the speech data by using a Chinese speech recognition error correction model, and obtaining a corrected Chinese speech recognition result. The Chinese speech recognition error correction model is a model based on a neural network, contains two sub-models of an acoustic model and a first text error correction model, and optionally, can also contain a language model. In the embodiment of the application, a position expansion weighting is used to form a first mixed batch input which fuses pinyin information and Chinese character information, so that the Chinese speech recognition error correction model in the embodiment of the application can fully utilize the information of the speech data, and the error correction effect is good.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech recognition, and particularly relates to a Chinese speech recognition error correction method and device and electronic equipment. BACKGROUND

[0002] The human-computer interaction technology based on speech recognition (ASR) is a very important technology in the terminal artificial intelligence (AI) field, and is widely applied in various electronic devices (such as mobile phones, tablet computers, desktop computers, etc.) to improve the human-computer interaction efficiency between users and electronic devices. For example, an intelligent voice assistant can recognize user speech and then understand user intent to automatically perform corresponding operations; for another example, a voice input method can recognize user speech and directly transcribe it into text, saving the complex process of typing input. The efficiency of human-computer interaction based on speech recognition depends largely on the effect of speech recognition.

[0003] As a branch of the technical field of speech recognition, Chinese speech recognition usually faces the following technical problems. On the one hand, one pronunciation in Chinese can correspond to multiple Chinese characters, making Chinese speech recognition prone to homophone errors; on the other hand, pronunciation habit problems caused by dialects make Chinese speech recognition prone to confusing sound errors. Therefore, Chinese speech recognition needs to be corrected to obtain more accurate recognition results. SUMMARY

[0004] The present application provides a Chinese speech recognition error correction method, device and electronic equipment, which can realize speech recognition and error correction for Chinese.

[0005] In a first aspect, the present application provides a Chinese speech recognition error correction method, which comprises the following steps: first, obtaining pinyin information and first Chinese character information of speech data. The speech data can be user speech data collected in real time by a receiving module of an electronic device, or a piece of speech data stored in the electronic device, etc. The pinyin information and the first Chinese character information are obtained based on the speech data, wherein the pinyin information can be obtained based on the speech data first, and then the first Chinese character information can be obtained based on the pinyin information, or the first Chinese character information can be obtained based on the speech data first, and then the pinyin information can be obtained based on the Chinese character information, or the pinyin information and the first Chinese character information can be obtained based on the speech data simultaneously. Second, fusing the pinyin information and the first Chinese character information to obtain mixed information. The mixed information obtained after fusion contains information in the pinyin information and information in the first Chinese character information, and the pinyin information and the first Chinese character information in the mixed information can be the same or different in weight. Next, applying a text error correction model to process the mixed information to obtain second Chinese character information. The text error correction model can be a model based on a neural network, which is obtained by training using speech data for model training in advance, so that the trained text error correction model has the function of processing speech data and outputting error correction results. Since the mixed information fuses the pinyin and Chinese character information, the text error correction model can utilize the pinyin and Chinese character information for error correction, thereby obtaining a better error correction effect. The second Chinese character information obtained contains corrected speech recognition results. Finally, outputting the corrected speech recognition results, thereby completing Chinese speech recognition error correction.

[0006] In a first implementation manner of the first aspect, the pinyin information contains pinyin probabilities, and the first Chinese character information contains Chinese character probabilities. That is, the probability values can be used to represent the possibility of different pinyins in the pinyin information and the possibility of different Chinese characters in the first Chinese character information, so that the probability values can be larger at the pinyin or Chinese character that the speech data is more likely to correspond to, and smaller at the pinyin or Chinese character that the speech data is less likely to correspond to, thereby realizing the quantitative representation of the pinyin information and the first Chinese character information. The probability values can be decimals between 0 and 1. The step of fusing the pinyin information and the first Chinese character information to obtain the mixed information specifically comprises: weighting and fusing the pinyin probabilities in the pinyin information and the Chinese character probabilities in the first Chinese character information to obtain the mixed information containing a plurality of sub-mixed information. Thus, the mixed information can contain the pinyin information and the first Chinese character information simultaneously, and the text error correction model can utilize the information in the pinyin and Chinese character for error correction, thereby obtaining a better error correction effect.

[0007] In a second implementation form of the first aspect of the present application, before the Pinyin probability in the Pinyin information and the Chinese character probability in the first Chinese character information are fused by weighting, the method further comprises: determining the positions of Chinese characters whose Chinese character probability is less than a threshold value in the Chinese character probability in the first Chinese character information, and fusing by weighting according to the positions. The threshold value can be a preset threshold value, for example, 0.9. If the Chinese character probability of one or more Chinese characters in the first Chinese character information is less than the threshold value, it can be considered that the one or more Chinese characters corresponding to the Chinese character probability can have errors, so that different weighting strategies can be used for targeted weighting fusion according to the position or positions, to achieve targeted probability adjustment by weighting, and to obtain better error correction effect.

[0008] In a third implementation form of the first aspect of the present application, the step of fusing the Pinyin probability in the Pinyin information and the Chinese character probability in the first Chinese character information by weighting specifically comprises: obtaining a plurality of position expansion regions based on the positions according to a preset rule; wherein the preset rule comprises a plurality of left offsets and a plurality of right offsets; and the position expansion region covers the positions, a left offset number of positions located on the left side of the positions, and a right offset number of positions located on the right side of the positions. Thus, the position expansion region covers one or more positions around the positions in addition to the positions. Since the Chinese character probability at the positions is less than the threshold value, the Chinese characters at the positions can have errors, so the Chinese characters at the positions around the positions can also have errors. Thus, by using the position expansion process, the position expansion region with wider coverage is obtained, so that the Chinese characters that can have errors are as far as possible included in the position expansion region, and the Pinyin probability with greater weight is introduced into the Chinese character probability at the positions in the position expansion region, to use the Pinyin information to assist in correcting the Chinese characters, so as to obtain better error correction effect.

[0009] In a fourth implementation form of the first aspect of the present application, the step of fusing the Pinyin probability in the Pinyin information and the Chinese character probability in the first Chinese character information by weighting specifically further comprises: replacing the Chinese character probability of the Chinese characters in the first Chinese character information located in the position expansion region with Pinyin mixed weighted probability, and replacing the Chinese character probability of the Chinese characters in the first Chinese character information located outside the position expansion region with Chinese character mixed weighted probability, to complete the weighting fusion. The Pinyin mixed weighted probability and the Chinese character mixed weighted probability mix and weight the Pinyin probability and the Chinese character probability with different weights, so that the Pinyin information is more introduced to assist in correcting the Chinese characters that can have errors in the position expansion region, and the Pinyin information is only introduced a little to assist in correcting the Chinese characters located outside the position expansion region, so that targeted Chinese character correction is achieved, and better error correction effect is obtained.

[0010] In the fifth implementation manner of the first aspect, the method further has the following characteristics: the pinyin mixed weighted probability is obtained by weighting and adding the pinyin probability in the pinyin information by the first weight and the Chinese character probability in the first Chinese character information by the second weight; the Chinese character mixed weighted probability is obtained by weighting and adding the Chinese character probability in the first Chinese character information by the first weight and the pinyin probability in the pinyin information by the second weight; and the first weight is greater than the second weight. Thus, the pinyin information with a greater proportion is introduced into the Chinese character probability with a greater possibility of error in the position expansion area to assist the text correction model in error correction, and the pinyin information with a smaller proportion is introduced into the Chinese character probability outside the position expansion area to assist error correction, so as to introduce the pinyin information with different weights for different Chinese characters to correct errors, and further improve the error correction effect of the text correction model.

[0011] In the sixth implementation manner of the first aspect, the method further has the following characteristics: the pinyin information is a pinyin probability tensor, the pinyin probability tensor includes a pinyin probability matrix composed of a pinyin probability vector; the first Chinese character information is a Chinese character probability tensor, the Chinese character probability tensor includes a Chinese character probability matrix composed of a Chinese character probability vector; and the mixed information is a mixed tensor, the mixed tensor includes a pinyin Chinese character mixed probability matrix composed of a pinyin Chinese character mixed probability vector. That is, in one possible implementation manner of the present application, a Chinese pinyin or a Chinese character can be represented by a probability vector, a sentence including multiple pinyins or Chinese characters can be further represented by multiple vectors, the sentence can be further represented by a matrix composed of multiple vectors, multiple sentences that can be simultaneously input into the text correction model in a batch of data can be represented by multiple matrices, and the mixed information including multiple sentences can be represented by a tensor including multiple matrices.

[0012] In a seventh implementation form of the first aspect as per the first implementation form of the first aspect of the application, the method further comprises the following features. The pinyin probability vector and the Chinese character probability vector are based on a vocabulary, and the vocabulary comprises a plurality of pinyins and a plurality of Chinese characters. In the pinyin probability vector, a value corresponding to a pinyin in the vocabulary is non-zero, and a value corresponding to a Chinese character in the vocabulary is zero. In the Chinese character probability vector, a value corresponding to a pinyin in the vocabulary is zero, and a value corresponding to a Chinese character in the vocabulary is non-zero. In the pinyin-Chinese character hybrid probability vector, values corresponding to the pinyin and the Chinese character in the vocabulary are non-zero. That is, the pinyin probability vector and the Chinese character probability vector represent the pinyin and Chinese character information of the speech data based on the same vocabulary. The pinyin probability vector represents the pinyin information to which the speech data can correspond by using a non-zero probability value in the pinyin field of the vocabulary. The Chinese character probability vector represents the Chinese character information to which the speech data can correspond by using a non-zero probability value in the Chinese character field of the vocabulary. The pinyin-Chinese character hybrid probability vector corresponds to the weighted fusion of the pinyin and Chinese character information by using non-zero values in the pinyin field and the Chinese character field of the vocabulary. Thus, the pinyin, the Chinese character, and the pinyin-Chinese character hybrid information obtained by the weighted fusion of the pinyin and Chinese character information are represented in a quantitative manner.

[0013] In an eighth implementation form of the first aspect as per the first implementation form of the first aspect of the application, the step of fusing the pinyin information and the first Chinese character information to obtain hybrid information comprises: performing weighted fusion on the pinyin probability vector in the pinyin probability tensor and the Chinese character probability vector in the Chinese character probability tensor to obtain a hybrid tensor comprising a plurality of pinyin-Chinese character hybrid probability matrices.

[0014] In a ninth implementation form of the first aspect as per the eighth implementation form of the first aspect of the application, before the weighted fusion of the pinyin probability vector and the Chinese character probability vector, the method further comprises: determining, in the Chinese character probability vector in the Chinese character probability tensor, a position of a Chinese character probability vector in which a maximum Chinese character probability is less than a threshold value, and performing the weighted fusion according to the position. The maximum Chinese character probability is a maximum probability value in the Chinese character probability vector. Since the Chinese character probability vector comprises a plurality of Chinese characters and a plurality of Chinese character probabilities corresponding to the plurality of Chinese characters, in this implementation form, whether the judgment result of the Chinese character corresponding to the Chinese character probability vector is likely to be incorrect is determined by examining the maximum Chinese character probability in the Chinese character probability vector. Thus, more pinyin information is introduced in the Chinese character probability vector in which the judgment result is likely to be incorrect to assist the text error correction model in performing error correction, thereby obtaining a better error correction effect.

[0015] In the tenth implementation manner of the first aspect, the step of obtaining the pinyin information and the first Chinese character information of the speech data specifically comprises: applying the text correction model to process the pinyin information to obtain the first Chinese character information. In the existing manner of obtaining Chinese characters from pinyin, a language model is usually used. In this implementation manner, the text correction model is used to realize the function of the language model, so that the language model in the entire Chinese speech recognition correction model becomes an optional sub-model, and the language model can be omitted by reusing the text correction model, thereby achieving the beneficial effects of reducing the model parameter quantity and the model size.

[0016] In the eleventh implementation manner of the first aspect, the step of obtaining the pinyin information and the first Chinese character information of the speech data specifically further comprises: applying an acoustic model to process the speech data to obtain the pinyin information, wherein the acoustic model can be a neural network model. The acoustic model can complete the process of converting speech to pinyin by extracting speech features in the speech data.

[0017] In the second aspect, the present application provides an electronic device for executing the method in the first aspect and the implementation manners of the first aspect to recognize and correct Chinese speech.

[0018] In the third aspect, the present application provides a computer readable storage medium storing computer instructions for executing the method in the first aspect and the implementation manners of the first aspect.

[0019] In the fourth aspect, the present application provides a chip device for executing the computer instructions in the third aspect. BRIEF DESCRIPTION OF DRAWINGS

[0020] Figure 1A is a functional and structural schematic diagram of a Chinese speech recognition model provided by an embodiment of the present application;

[0021] Figure 1B is a functional and structural schematic diagram of a Chinese speech recognition correction model provided by an embodiment of the present application;

[0022] Figure 2A is a reasoning process schematic diagram of a Chinese speech recognition correction model provided by an embodiment of the present application;

[0023] Figure 2B is another reasoning process schematic diagram of a Chinese speech recognition correction model provided by an embodiment of the present application;

[0024] Figure 3A is a word table field schematic diagram provided by an embodiment of the present application;

[0025] Figure 3Bis another word table field schematic diagram provided by an embodiment of the present application;

[0026] Figure 3C is another word table field schematic diagram provided by an embodiment of the present application;

[0027] Figure 4 is another inference process schematic diagram of a Chinese speech recognition error correction model provided by an embodiment of the present application;

[0028] Figure 5 is a position expansion weighting process flowchart provided by an embodiment of the present application;

[0029] Figure 6 is a position expansion schematic diagram provided by an embodiment of the present application;

[0030] Figure 7 is a weighting schematic diagram provided by an embodiment of the present application;

[0031] Figure 8 is an integration, search and decoding process flowchart provided by an embodiment of the present application;

[0032] Figure 9 is a beam search schematic diagram provided by an embodiment of the present application;

[0033] Figure 10 is a text error correction model structure schematic diagram provided by an embodiment of the present application;

[0034] Figure 11 is a relationship schematic diagram of a text error correction model training stage and an inference stage provided by an embodiment of the present application;

[0035] Figure 12 is a text error correction model training stage schematic diagram provided by an embodiment of the present application;

[0036] Figure 13 is a text error correction model training stage flowchart provided by an embodiment of the present application;

[0037] Figure 14 is a text error correction model training stage flowchart provided by an embodiment of the present application;

[0038] Figure 15 is an application schematic diagram of a Chinese speech recognition error correction model in a smart voice assistant of a terminal device provided by an embodiment of the present application;

[0039] Figure 16 is an application schematic diagram of a Chinese speech recognition error correction model in a voice input method of a terminal device provided by an embodiment of the present application;

[0040] Figure 17is a schematic diagram of application of a Chinese speech recognition error correction model in a speech-to-text function of a terminal device according to an embodiment of the present application;

[0041] Figure 18A is a structural schematic diagram of an electronic device according to an embodiment of the present application;

[0042] Figure 18B is a software structure block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0043] In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular sequences of steps, techniques, etc., in order to provide a thorough understanding of the present embodiments. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, and circuits are omitted so as not to obscure the description of the present application with unnecessary detail.

[0044] In order to facilitate understanding of the embodiments of the present application, the relevant knowledge in the art is briefly introduced first.

[0045] At present, a speech recognition system usually uses an acoustic model and a language model to represent statistical features of speech. The acoustic model is used to establish a relationship between a speech signal and a phoneme or a phonetic unit, for example, for Chinese, the acoustic model can convert a speech signal into a pinyin. The language model is used to build a relationship between words or characters in a language, for example, for Chinese, the language model can convert a pinyin output by the acoustic model into a piece of text. The acoustic model and the language model can be obtained through training and / or learning, for example, by using labeled data to train a deep model. The combined use of the acoustic model and the language model can realize the conversion of a speech signal into a piece of text, thereby realizing speech recognition.

[0046] There are a large number of homophonic characters in Chinese, and one pinyin often corresponds to multiple Chinese characters; there are also a large number of confusing sounds in Chinese, for example, confusion of front and back nasal sounds due to dialect speaking habits. These characteristics make it easy to produce errors in Chinese speech recognition. For example, a homophonic character error of recognizing "key" as "if", and a confusing sound error of recognizing "today" as "Jing Tian". Therefore, Chinese speech recognition needs to be corrected to obtain more accurate recognition results.

[0047] Therefore, the application provides a Chinese speech recognition error correction model and method based on deep learning, which can realize Chinese language recognition error correction with high accuracy.

[0048] Figure 1A An exemplary function structure diagram of a Chinese speech recognition model provided by the embodiments of the application is shown. Figure 1A As shown, the Chinese speech recognition model can include two sub-models, i.e., an acoustic model and a language model. Speech data is processed by the acoustic model and the language model in the Chinese speech recognition model in sequence, and is converted into a speech recognition result for output. As described above, the acoustic model and the language model can be obtained by training. For example, in the model and method provided by the embodiments of the application, the language model can be a deep learning model based on a neural network architecture. The language model that has completed learning after being trained using labeled data can have the functions described above, i.e., the function of converting the pinyin output by the acoustic model into text. Optionally, the Chinese speech recognition model can further include other sub-models.

[0049] Figure 1B An exemplary function structure diagram of a Chinese speech recognition error correction model provided by the embodiments of the application is shown. Figure 1B As shown, the Chinese speech recognition error correction model can include an acoustic model, a language model, and a first text error correction model. The first text error correction model can correct the output results of the acoustic model and / or the language model, and thus output a corrected speech recognition result. In an optional manner provided by the embodiments, the first text error correction model can take the output results of the acoustic model and the language model as input data. In another optional manner provided by the embodiments, the first text error correction model also has the function of the language model, and can replace the language model to realize the function of converting pinyin into text. Thus, the Chinese speech recognition error correction model can not include the language model, and the first text error correction model can directly take the output result of the acoustic model as input data. Optionally, the Chinese speech recognition error correction model can further include other sub-models.

[0050] It should be clear that, in the embodiments and the accompanying drawings provided by the application, when the output result of a previous model is described as input data of a subsequent model, it does not mean that the output result of the previous model is not processed before being input into the subsequent model. For example, Figure 1B the output result of the acoustic model in the Chinese speech recognition model can be pre-processed, such as being converted into a form suitable for input or being integrated with some other information, before being input into the first text error correction model as input data. Figure 1BBefore the mid-acoustic model output result and the language model output result are input into the first text correction model, data fusion processing of the two output results can be first performed, and the fused result is taken as input data of the first text correction model. Similarly, the output result of the first text correction model can also be subjected to certain post-processing before becoming the final output of the Chinese speech recognition correction model, that is, the corrected speech recognition result.

[0051] The Chinese speech recognition correction model provided in the embodiments of the present application can be a deep learning-based correction model, which can be implemented based on a neural network architecture. Therefore, the functions of the model provided in the embodiments of the present application are obtained by training using data. Specifically, first, an initial model composed of a plurality of neural network layers is built, which can have random initial parameters; then, using labeled data as training data, the model parameters (such as the weight coefficients and bias of neurons) are iteratively updated through a loop process of back propagation, gradient descent, etc. according to a preset loss function until the loss value of the loss function is less than a preset threshold to end the loop, and the model training is completed; finally, after the last loop and parameter update are completed, the trained model is obtained.

[0052] The trained model has the function of reasoning for data similar to the type of training data. For example, if a speech recognition correction model is trained using Mandarin speech data, the model has the function of speech recognition correction for Mandarin. Specifically, inputting a piece of Mandarin speech data, the model can better implement recognition and correction of the speech data; if a piece of Cantonese speech data is input, the model can not be able to well implement recognition and correction. For another example, a speech recognition correction model trained with daily conversation chat speech cannot well implement recognition and correction of news broadcast speech. It should be clear that the model and method provided in the embodiments of the present application do not limit the type of training data used, and those skilled in the art can train the required model according to the type of speech data to be corrected.

[0053] Next, the reasoning phase of the Chinese speech recognition correction model provided in the embodiments of the present application is first introduced, that is, the process of applying the model to recognize and correct unlabeled speech data.

[0054] Figure 2A is a schematic diagram of a reasoning process of a Chinese speech recognition correction model provided in the embodiments of the present application. Figure 2A The reasoning process shown can be Figure 1B The reasoning process of the Chinese speech recognition correction model shown comprises an acoustic model, a language model, and a first text correction model.

[0055] Figure 2A The reasoning process shown includes:

[0056] The first step is to input the first speech data into the acoustic model. After the acoustic model extracts and processes the acoustic features of the first speech data, it outputs the first tensor.

[0057] The first speech data can be an unlabeled segment of speech data to be recognized. "Unlabeled" means the model cannot obtain information about the correct speech recognition result corresponding to the first speech data. Processing the first speech data with a model aims to identify the possible speech recognition results corresponding to it. The various models applied during the inference phase (e.g., ...) Figure 2A The acoustic model, language model, and first text correction model mentioned above are all trained models that have completed parameter learning. These models have reasoning capabilities and can take unlabeled data as input to output results that the model deems likely to be correct. The processing of the first speech data by the aforementioned acoustic model may include the extraction, recognition, and / or classification of acoustic features (such as Melp features).

[0058] Therefore, in this step, the acoustic model takes the unlabeled first speech data as input, and the first output tensor can be the pinyin data that the acoustic model believes may correctly correspond to the first speech data. For example, if the first speech data is a speech data containing the content "The weather is nice", the acoustic model can output a result similar to "tian1 qi4 bu2 cuo4", where the numbers after the pinyin are used to represent the tone of the Chinese characters, with 0-4 representing the neutral tone, first tone, second tone, third tone, and fourth tone, respectively.

[0059] In this embodiment, the term "tensor n" is used to refer to the output data of each model, but this does not constitute a limitation on this embodiment. The tensor mentioned in this embodiment can refer to a vector, matrix, or tensor, and can contain various types of information such as character information, numerical information, and sorting information.

[0060] For example, if the first voice data is a voice data with content "wei", the acoustic model considers that the voice data has a probability of 0.95 to be "wei2", a probability of 0.03 to be "wei3", and a probability of 0.02 to be "wei4". The acoustic model can output a 3-dimensional vector containing the probability information of the pinyin, similar to [(wei2, 0.95), (wei3, 0.03), (wei4, 0.02)]. If "wei" is regarded as a sentence containing one word, the acoustic model can use a 1*3 matrix [[(wei2, 0.95), (wei3, 0.03), (wei4, 0.02)]] to represent the recognition result of the sentence. If the sentence belongs to a batch of input data containing one voice sample, the acoustic model can use a 1*1*3 tensor [[[(wei2, 0.95), (wei3, 0.03), (wei4, 0.02)]]] to represent the recognition result of the batch of input data.

[0061] The above description of the data output by the acoustic model is only one possible implementation provided by the embodiments of the present application, and is not limited. The data output by the acoustic model is within the scope covered by the embodiments of the present application as long as it contains pinyin information.

[0062] In the second step, the first tensor is input into the language model, and the language model outputs a second tensor after processing.

[0063] As described above, the language model has the function of converting the phoneme information output by the acoustic model into text information. The language model in the Chinese speech recognition error correction model provided by the embodiments of the present application has the function of converting the pinyin information output by the acoustic model into text (for example, Chinese characters) information. Therefore, the second tensor can be an output result containing text information.

[0064] For example, if the first tensor is a vector containing pinyin probability information, similar to [(wei2, 0.95), (wei3, 0.03), (wei4, 0.02)], the language model considers that the text information corresponding to the first tensor has a probability of 0.98 to be "wei", a probability of 0.01 to be "wei", and a probability of 0.01 to be "wei". The language model can output a vector containing Chinese character probability information, similar to [(wei, 0.98), (wei, 0.01), (wei, 0.01)]. Similarly, if the first tensor output by the acoustic model is in the form of a matrix or a tensor, the language model can correspondingly output a first tensor in the form of a matrix or a tensor.

[0065] In the third step, the first tensor and the second tensor are fused to convert into a third tensor.

[0066] The third tensor obtained after information fusion contains information in the first tensor and information in the second tensor. The Chinese speech recognition correction model provided in the embodiments of the present application jointly uses pinyin information and character information, so that a better correction effect can be obtained based on the information in two aspects. Specifically, the information fusion process described above can be, for example, a weighted addition process of the first tensor and the second tensor according to a preset weighting rule and a preset weighting coefficient. For example, the third tensor can be a weighted fusion of 0.9 times the second tensor and 0.1 times the first tensor. For another example, part of the third tensor is a weighted fusion of 0.9 times the second tensor and 0.1 times the first tensor, and another part is a weighted fusion of 0.1 times the second tensor and 0.9 times the first tensor. Those skilled in the art can configure appropriate weighting rules and weighting coefficients according to specific needs, without going beyond the scope of the embodiments of the present application. This specification will provide a feasible information fusion method, i.e., a position expansion weighting method, in the subsequent embodiments. The specific steps of this method will be described in detail in subsequent embodiments, and will not be described here.

[0067] In the fourth step, the third tensor is input into the first text correction model, and the fourth tensor is output after the first text correction model is processed.

[0068] As described above, the first text correction model used in the inference process is a model obtained after training, which has the function of correcting the input data and can output the output result considered more correct by the model. For example, the first text correction model can identify the language model as "Jing Tian Tianji not bad" based on the input information, and output "today's weather is good" after correction.

[0069] Optionally, the fourth tensor can be the output result of the first text correction model containing a plurality of possible correction results and the probability of each result. For example, a piece of content "today's weather is good" is obtained after the first speech data is processed by the previous steps to obtain the third tensor, and the first text correction model outputs the result considered correct by the third tensor. For example, the first character in the first position is "Jin" with a probability of 0.95, and "Jing" with a probability of 0.05, the second character in the first position is "Tian" with a probability of 0.99, and "Tian" with a probability of 0.01, and so on. The first text correction model can output a plurality of possible Chinese characters and corresponding probability judgment results for each position. In the subsequent step, the plurality of probability judgment results can be post-processed, for example, searched according to a preset rule, and a most suitable result is selected as the corrected speech recognition result.

[0070] In the fifth step, the fourth tensor is post-processed to convert into a corrected speech recognition result, which is the final output result of the entire speech recognition correction model.

[0071] Optionally, if the fourth tensor has been a deterministic error correction judgment result, no post-processing can be performed, and the output of the first text error correction model is directly taken as the corrected speech recognition result.

[0072] Figure 2B is another schematic diagram of an inference process of a Chinese speech recognition error correction model provided by the embodiment of the present application. Figure 2B The inference process shown can be Figure 1B The inference process of the Chinese speech recognition error correction model shown contains an acoustic model and a first text error correction model.

[0073] It can be seen that Figure 2B The inference process shown is different from Figure 2A The difference between the inference process shown and the inference process shown in FIG. 1 is that the output result of the acoustic model is no longer processed by using the language model, but is processed by using the first text error correction model. That is, the function of the language model is replaced by the first text error correction model, the reuse of the first text error correction model is realized, and thus the language model can not be contained in the entire Chinese speech recognition error correction model, so that the embodiment can realize the beneficial effects of reducing the model size and reducing the model parameter amount.

[0074] The reuse of the first text error correction model in the inference process shown is because the first text error correction model obtained after training has the function of converting the pinyin information into the character information by using a specific training method. The specific training method will be described in detail in subsequent embodiments. Figure 2B

[0075] Figure 2B The other parts of the inference process shown are the same as those of the inference process shown in FIG. 1, and thus the repetitive description is not performed here. Figure 2A

[0076] The inference process shown in FIG. 2 is different from the inference process shown in FIG. 1 in that Figure 2A , Figure 2B ​​It can be known from the inference process of the Chinese speech recognition error correction model provided in the embodiments of the present application that the functions of the acoustic model and the language model are as follows: the acoustic model is used to extract and process the acoustic features of the first speech data, and represent the first speech data as a first tensor containing pinyin information. The language model is used to process the first tensor output by the acoustic model, and convert it into a second tensor containing Chinese character information. The first text error correction model can have two functions: on the one hand, it has the function of the language model, and can realize the process of converting data containing pinyin information into data containing Chinese character information; on the other hand, it can complete the correction of numerical values, process and adjust the numerical values in the third tensor that fuses pinyin information and Chinese character information, so that the numerical values in the third tensor are closer to the possible correct speech recognition results represented by the numerical values. That is, the first text error correction model processes the third tensor, so that the numerical values at the positions representing the possible correct speech recognition results become larger, and the numerical values at the positions representing the possible incorrect speech recognition results become smaller, so that the numerical values in the fourth tensor output by the first text error correction model are biased towards the more possible correct recognition results considered by the first text error correction model. Finally, the fourth tensor can be post-processed to obtain the corrected speech recognition result, and the entire speech error correction model has the function of error correction.

[0077] It can be known from the above analysis that the first text error correction model has two functions. In combination with the above analysis, it can be seen that: Figure 1A and Figure 1B It can be seen that:

[0078] On the one hand, the first text error correction model is added to a Chinese speech recognition model without the function of error correction or with a weak function of error correction (A), so that a Chinese speech recognition error correction model (B) with the function of error correction or with a stronger function of error correction than the Chinese speech recognition model can be obtained. Figure 1A Figure 1B

[0079] On the other hand, since the first text error correction model also has the function of the language model, the first text error correction model can be used to replace the language model, that is, the language model is no longer used, and the function of the original language model is realized by reusing the first text error correction model. In this way, the language model indispensable to the original Chinese speech recognition model can be selected to be deleted, so that the size of the entire Chinese speech recognition error correction model is reduced, and the parameter amount of the entire model is reduced.

[0080] ​​In an optional implementation provided by the embodiments of the present application, the first to fourth tensors are all three-dimensional tensors. The first tensor and the second tensor have the same shape, different positions of non-zero values, and are one-dimensional in the first dimension. The third tensor and the fourth tensor have the same shape and are at least one-dimensional in the first dimension, and the dimensions of the second and third dimensions are the same as the dimensions of the second and third dimensions of the first and second tensors. For example, assuming that the first and second tensors are tensors with a shape of [1, U, V], the third and fourth tensors are tensors with a shape of [B, U, V], where B is greater than or equal to 1.

[0081] The acoustic model, the language model, and the text correction model are all implemented based on a neural network. As an optional implementation provided by the embodiments of the present application, the acoustic model can be a model based on a transducer structure, the language model can be a model based on a convolutional neural network (CNN) layer, and the first text correction model can be a model based on a bidirectional encoder (BERT) structure of a transducer architecture.

[0082] In an optional implementation provided by the embodiments of the present application, the first to fourth tensors are probability tensors based on a vocabulary. The probability values are used to represent character elements in the vocabulary, that is, the values corresponding to the character elements in the tensors are all decimals greater than 0 and less than 1. The vocabulary is an ordered list containing V elements. Each element contains at least a character element and can also contain a non-character element. Each element in the vocabulary corresponds to a positive integer indicating the position of the element in the list, so that each element in the vocabulary is arranged in a fixed order.

[0083] In an optional implementation provided by the embodiments of the present application, the vocabulary used by the Chinese speech recognition model and the Chinese speech recognition correction model includes at least Chinese characters and Chinese pinyin, and optionally includes foreign language pinyin (such as foreign language pinyin “wai” and “fai” of foreign language “WiFi”) and other character elements. The non-character elements can include, but are not limited to, one or more of the following: zero padding, reserved bit, character bit, and separator.

[0084] Figure 3A FIG. 1 is a schematic diagram of a vocabulary field. As shown in FIG. 1, the vocabulary field includes a character element and a non-character element. Figure 3AAs shown, the vocabulary consists of 6,538 elements in total. The 1st to 5,301st positions of the vocabulary are 5,301 common Chinese characters arranged in dictionary order from "啊" to "酢"; the 5,302nd to 6,538th positions are 1,237 common Chinese pinyin arranged in alphabetical order from "a0" to "zuo4".

[0085] Figure 3B is a schematic diagram of another vocabulary field. As Figure 3B shown, the vocabulary consists of 6,896 elements in total. The 1st to 5,301st positions of the vocabulary are 5,301 common Chinese characters arranged in dictionary order from "啊" to "酢"; the 5,302nd to 6,538th positions are 1,237 common Chinese pinyin arranged in alphabetical order from "a0" to "zuo4"; the 6,539th to 6,896th positions are 358 common foreign language pinyin arranged in alphabetical order from "a" to "zi".

[0086] Figure 3C is a schematic diagram of yet another vocabulary field. As Figure 3C shown, the vocabulary consists of 7,008 elements in total. The 1st position of the vocabulary is 1 padding zero position, denoted as [PAD], used to mark elements that need to be filled with zeros; the 2nd to 100th positions are 99 reserved positions reserved for subsequent addition of new elements; the 101st to 109th positions are 9 character positions, including five punctuation marks [UNK], [CLS], [SEP], [MASK] and "、,。?!", where [UNK] is used to represent out-of-vocabulary words (Out-of-Vocabulary, OOV), [CLS] is the sentence start symbol, used to identify the start of a sentence, [SEP] is the sentence separator, used to separate two sentences, and [MASK] is used to mask characters during model training; the 110th to 5,410th positions are 5,301 common Chinese characters arranged in dictionary order from "啊" to "酢"; the 5,411th and 5,412th positions are separator symbols; the 5,413th to 6,649th positions are 1,237 common Chinese pinyin arranged in alphabetical order from "a0" to "zuo4"; the 6,650th to 7,007th positions are 358 common foreign language pinyin arranged in alphabetical order from "a" to "zi"; the 7,008th position is a separator symbol.

[0087] It should be clear that the above Figure 3A 、 Figure 3B and Figure 3CThe shown word table field and field arrangement order are only three possible embodiments in the scheme, not limited. Those skilled in the art can make corresponding adjustments according to actual application needs, for example, the word table can contain less than or more than 5301 Chinese characters, can contain less than or more than 1237 Chinese pinyin, can contain other types of elements in addition to zero filling, reserved bit, character bit, Chinese character, separator, Chinese pinyin, foreign language pinyin, and can design the arrangement order of corresponding character elements and non-character elements according to actual needs.

[0088] Figure 4 is another schematic diagram of the inference process of the Chinese speech recognition error correction model provided in the embodiment of the present application.

[0089] By Figure 4 It can be seen that the inference process of the Chinese speech recognition error correction model provided in the embodiment is as follows: first, the acoustic model extracts and processes the acoustic features of the first speech data, converts the first speech data into a first pinyin probability tensor for output; then, the first pinyin probability tensor is converted into a first Chinese character probability tensor for output after being processed by the language model or the first text error correction model; then, the first pinyin probability tensor and the first Chinese character probability tensor are converted into a first mixed batch input after information fusion; subsequently, the first mixed batch input is converted into a second Chinese character probability tensor for output after being processed by the first text error correction model; finally, the second Chinese character probability tensor is converted into a corrected speech recognition result for output after post-processing, that is, the final output of the entire speech recognition error correction model.

[0090] In one possible embodiment of the scheme, the above-mentioned first pinyin probability tensor, first Chinese character probability tensor and second Chinese character probability tensor can be probability tensors based on the word table shown in Figure 3A , Figure 3B or Figure 3C .

[0091] Taking the probability tensor based on the word table shown in Figure 3A as an example, the above-mentioned first pinyin probability tensor is a tensor with a shape of 1xUx6538, wherein the first dimension is 1, indicating that the number of samples in the pinyin probability tensor is 1; the second dimension is U, indicating that a piece of first speech data contains U characters, for example, there are 6 Chinese character in "today the weather is good", so its corresponding U=6; the third dimension is the number of elements of the based word table, since the embodiment adopts the word table shown in Figure 3A , therefore the word table contains 6538 elements. The above-mentioned first pinyin probability tensor is denoted as I=[[I 1,1 ,I 1,2 ,…,I 1,U ]], wherein I1=[I 1,1 ,I 1,2 ,…,I1,U ] is the phonetic probability vector corresponding to the Uth character in the phonetic probability matrix I n in the first phonetic probability tensor; the phonetic probability vector corresponding to the Uth character in the phonetic probability matrix I n,U = [a n,U,1 ,a n,U,2 ,…,a n,U,6538 ] for the first phonetic probability tensor, n = 1; the phonetic probability vector in the present scheme has values of 0 in the Chinese character field and values of decimals greater than 0 and less than 1 in the Chinese pinyin field, and the sum of all the decimals is 1, thereby representing probability values; for the phonetic probability vector I n,U in the present embodiment, the conditions to be met can be expressed as: a n,U,1 to a n,U,5301 are 0, a n,U,5302 to a n,U,6538 are decimals greater than 0 and less than 1, and a n,U,5302 +a n,U,5303 +…+a n,U,6538 = 1.

[0092] The first Chinese character probability tensor is a tensor with a shape of 1 × U × 6538, and the first Chinese character probability tensor is denoted as J = [[J 1,1 ,J 1,2 ,…,J 1,U ]], wherein J1 = [J 1,1 ,J 1,2 ,…,J 1,U ] is the Chinese character probability matrix corresponding to the first sample in the first Chinese character probability tensor; the Chinese character probability vector corresponding to the Uth character in the Chinese character probability matrix J n corresponding to the nth sample in the first Chinese character probability tensor is denoted as J n,U = [b n,U,1 ,b n,U,2 ,…,b n,U,6538 ]; for the first Chinese character probability tensor, n = 1; the Chinese character probability vector in the present scheme has values of 0 in the Chinese pinyin field and values of decimals greater than 0 and less than 1 in the Chinese character field, and the sum of all the decimals is 1, thereby representing probability values; for the Chinese character probability vector J n,U in the present embodiment, the conditions to be met can be expressed as: b n,U,5302 to b n,U,6538 are 0, b n,U,1 to b n,U,5301 are decimals greater than 0 and less than 1, and b n,U,1 +b n,U,2 +…+b n,U,5301 = 1.

[0093] The probability tensor of the second Chinese character mentioned above is a tensor of shape n×U×6538, denoted as K=[[K 1,1 ,K 1,2 ,…,K 1,U ],[K 2,1 ,K 2,2 ,…,K 2,U ],…,[K n,1 ,K n,2 ,…,K n,U ]], where K n =[K n,1 ,K n,2 ,…,K n,U [ ] is the probability matrix of the Chinese character corresponding to the nth sample in the second Chinese character probability tensor; K is the probability matrix of the Chinese character corresponding to the nth sample in the second Chinese character probability tensor. n The probability vector of the Chinese character corresponding to the U-th character in the array is denoted as K. n,U =[c n,U,1 ,c n,U,2 ,c n,U,3 ,…,c n,U,6538 For the second Chinese character probability tensor, n≥1; according to the conditions that the Chinese character probability vector in this scheme should satisfy, then for the Chinese character probability vector K in this embodiment... n,U The condition that must be met can be expressed as: c n,U,5302 To c n,U,6538 c is 0 n,U,1 To c n,U,5301 A decimal greater than 0 and less than 1, and c n,U,1 +c n,U,2 +…+c n,U,5301 =1.

[0094] Compared to the first Chinese character probability tensor, the value of the second Chinese character probability tensor has been corrected, making its probability value at the Chinese character position corresponding to the correct speech recognition result larger and its probability value at the Chinese character position corresponding to the incorrect speech recognition result smaller. This makes the results of subsequent integration, search and decoding processes more accurate, thus enabling the text error correction model to have error correction function.

[0095] exist Figure 4 In the inference process of the speech recognition error correction model shown, a first speech data segment with specific semantic information is processed by the acoustic model and outputs a first pinyin probability tensor. This first pinyin probability tensor, with pinyin probability values ​​of varying magnitudes, reflects the acoustic model's judgment result on the semantic information that the first speech data may possess, based on... Figure 3A Taking the probability tensor of the vocabulary as an example, a first speech data of the phrase "The weather is nice today" is input into the acoustic model. After processing by the acoustic model, the output is the first pinyin probability tensor I = [[I1,1 , I 1,2 , I 1,3 , I 1,4 , I 1,5 , I 1,6 , where I1 = [I 1,1 , I 1,2 , I 1,3 , I 1,4 , I 1,5 , I 1,6 is the pinyin probability matrix corresponding to the sample "Today's weather is nice", and I 1,1 to I 1,6 respectively correspond to the pinyin probability vectors of the 6 characters from "今" to "错". Suppose the judgment result of the acoustic model on the pinyin probability vector corresponding to the first speech data corresponding to "今" is: the probability of "jin1" is 0.95, the probability of "jing1" is 0.02, the probability of "jin4" is 0.008, etc., and in Figure 3A the pinyin characters "jin1", "jing1" and "jin4" in the shown vocabulary table correspond to the 5729th, 5732nd and 5731st elements respectively. Then, in the pinyin probability vector corresponding to "今" in the pinyin probability matrix of the sample "Today's weather is nice" in the first pinyin probability tensor output by the acoustic model, there are a 1,1,5729 = 0.95, a 1,1,5732 = 0.02, a 1,1,5731 = 0.008, and a 1,1,1 to a 1,1,5301 are all 0, and a 1,1,5302 + a 1,1,5303 +... + a 1,1,6538 = 1. For the convenience of recording and intuitive representation, without limitation, the pinyin probability vector corresponding to "今" in the pinyin probability matrix I1 of the first pinyin probability tensor corresponding to the first speech data of "Today's weather is nice" can be recorded as I 1,1 = [(jin1, 0.95), (jing1, 0.02), (jin4, 0.008),...]. It should be clear that in this recording method, the order of the elements does not represent the true position of the elements in the vector, but only for the purpose of convenient recording and intuitive representation, the elements with larger probabilities are placed before the elements with smaller probabilities.

[0096]

[0096] Next, taking the first pinyin probability tensor output by the acoustic model as the input, after being processed by the language model or the first text error correction model, the first Chinese character probability tensor is output. The first Chinese character probability tensor reflects the judgment result of the language model or the first text error correction model on the correspondence between Chinese pinyin and Chinese characters based on the information provided by the first pinyin probability tensor and combined with the reasoning ability of the model itself. Similarly, based onFigure 3A For example, the probability tensor of the word table shown, there is a piece of first voice data "today the weather is good", as described above, the phonetic probability vector corresponding to "jin" in the phonetic probability matrix in the first phonetic probability tensor output after the first voice data is processed by the acoustic model is I 1,1 =[(jin1,0.95),(jing1,0.02),(jin4,0.008),…], assuming I 1,1 After being processed by the language model or the first text correction model, the language model or the first text correction model obtains a judgment result based on the probability values in the first phonetic probability tensor, combines the context semantics of the whole piece of first voice data, and uses the reasoning ability of the model itself, which is that there is a 0.975 probability of "jin", a 0.015 probability of "jin", a 0.003 probability of "jin", and so on, and in Figure 3A "jin", "jin" and "jin" in the word table shown correspond to the 1814th, 1822nd and 1830th elements respectively, then the "jin" corresponding to the Chinese character probability vector in the Chinese character probability matrix corresponding to the sample "today the weather is good" in the first Chinese character probability tensor output by the language model or the first text correction model has b 1,1,814 =0.975, b 1,1,1822 =0.015, b 1,1,1830 =0.003, and b 1,1,1 +b 1,1,2 +…+b 1,1,5301 =1, b 1,1,5302 to b 1,1,6538 are all 0. Similarly, for the convenience of recording and intuitive representation, but not limitation, the "jin" corresponding to the Chinese character probability vector in the Chinese character probability matrix J1 in the first Chinese character probability tensor corresponding to the piece of first voice data "today the weather is good" can be recorded as J 1,1 =[(jin,0.975),(jin,0.015),(jin,0.003),…], it should be clear that the order of elements in this recording method does not represent the real position of the element, but only for the purpose of convenient recording and intuitive representation, the element with larger probability is placed before the element with smaller probability.

[0097] Figure 5 An exemplary flowchart of an information fusion process provided by the embodiment of the application is shown, that is, the flowchart of position expansion weighting, which can include steps 501-503:

[0098] 501, find the position of the Chinese character probability vector J n in the Chinese character probability matrix J n,U,max in the first Chinese character probability tensor J, whose maximum probability P n,U is less than the threshold value T. Wherein, the maximum probability P n,U,max represents the Chinese character probability matrix Jn The probability vector J of the Chinese character corresponding to the U-th character in the middle. n,U The maximum probability value in.

[0099] 502. Expand the location of one or more found locations based on multiple preset rules. Specifically, expand the location with the highest probability P. n,U,max The probability vector J of Chinese characters less than the threshold T n,U Centered on the target, the position is expanded according to several preset left and right offsets to obtain several position expansion methods.

[0100] 503. Replace the Chinese character probability vectors located within the position expansion region with a mixed weighted vector of pinyin, and replace the Chinese character probability vectors located outside the position expansion region with a mixed weighted vector of Chinese characters. Specifically, the aforementioned mixed weighted vector of pinyin can be the sum of λ times the pinyin probability vector and (1-λ) times the Chinese character probability vector; the aforementioned mixed weighted vector of Chinese characters can be the sum of (1-λ) times the pinyin probability vector and λ times the Chinese character probability vector, where λ∈(0.5,1).

[0101] 504. Obtain the first mixed batch input.

[0102] Next, combined Figure 6 , Figure 7 The schematic diagram illustrates, through examples, a detailed implementation of the above-described position expansion weighting method provided in this application.

[0103] Figure 6 An exemplary schematic diagram illustrating a method for determining a location expansion region according to an embodiment of this application is shown. For example... Figure 6 As shown, assuming the vocabulary contains V characters, taking a speech data segment with the content "That woman is called Bella" as an example, then U=7. After the speech data with the content "That woman is called Bella" is processed by the acoustic model, it outputs a first pinyin probability tensor with a shape of 1×7×V. The first pinyin probability tensor contains a pinyin probability matrix 607 with a shape of 7×V, which contains 7 V-dimensional pinyin probability vectors. After the above first pinyin probability tensor is processed by the language model or the first text correction model, it outputs a first Chinese character probability tensor with a shape of 1×7×V. The first Chinese character probability tensor contains a Chinese character probability matrix 608 with a shape of 7×V, which contains 7 V-dimensional Chinese character probability vectors. Using the above pinyin probability matrix 607 and Chinese character probability matrix 608 as input, the position expansion related steps are performed:

[0104] 1. Find the maximum probability P in the Chinese character probability matrix 608. n,U,max The probability vector J of Chinese characters less than the threshold T n,U Among them, the maximum probability P n,U,maxDenote the maximum probability value in the Chinese character probability vector corresponding to the U-th character in the Chinese character probability matrix 608. Taking Figure 6 as an example, if T = 0.9 is set, then since the maximum probability P n,6,max = 0.49509 in the Chinese character probability vector [(bei, 0.49509), …] corresponding to the 6th character 'bei' is less than the threshold 0.9, the Chinese character probability vector 601 in the Chinese character probability matrix 608 with the maximum probability less than the threshold is found to be J n,6 = [(bei, 0.49509), …]. <​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​​The 4 position expansion region diagrams in the embodiment shown 609 schematically show the 4 position expansion regions described above, for example, to express that the sample in which the first position expansion region 603 is located is "that woman just bei4 pulled", it should be clear that this expression is only to facilitate the differentiation of the different processing methods of the Chinese character probability vectors within and outside the position expansion region in the subsequent weighting processing according to the position expansion region, and is not the actual processing method of the position expansion region.

[0107] Figure 6 The embodiment shown belongs to the case where there is a Chinese character probability vector with a maximum probability less than the threshold value in the Chinese character probability matrix 608. If there is no Chinese character vector with a maximum probability less than the threshold value in the first Chinese character probability matrix, correspondingly, there is no offset center for position expansion, so it is impossible to perform position expansion according to the left offset and the right offset, therefore, in this case, all Chinese character vectors are directly regarded as being located outside the position expansion region.

[0108] In another possible embodiment of the present scheme, there are two or more Chinese character probability vectors with a maximum probability P n,U,max less than the threshold value T in the Chinese character probability matrix 608, which can correspond to Figure 5 steps 501 and 502 in the above embodiment, at this time, multiple Chinese character probability vectors with a maximum probability less than the threshold value are simultaneously taken as offset centers, and position expansion is performed according to multiple combinations of the preset left offset and right offset to obtain multiple position expansion regions. For example, taking a character segment with U=10 as an example, if the Chinese character probability vectors corresponding to the 2nd character (J n,2 ) and the 9th character (J n,9 ) satisfy the condition that the maximum probability is less than the threshold value, and the left offset left_offset=[-1, 0] and the right offset right_offset=[0, 1], then (1) when left_offset=-1 and right_offset=0 with J n,2 as the offset center, and left_offset=-1 and right_offset=0 with J n,9 as the offset center, the first position expansion region is obtained, that is, J n,2 is expanded by-1 characters to the left and 0 characters to the right with J n,9 as the offset center, and J n,1 is expanded by-1 characters to the left and 0 characters to the right with J n,2 as the offset center, so that the position expansion region contains J n,8 , J n,9 , J n,2left_offset = -1, right_offset = 0, and J n,9 left_offset = 0, right_offset = 1, and J n,2 left_offset = 0, right_offset = 1, and J n,1 left_offset = 0, right_offset = 1, and J n,2 left_offset = 0, right_offset = 1, and J n,9 left_offset = 0, right_offset = 1, and J n,10 left_offset = 0, right_offset = 1, and J n,2 left_offset = -1, right_offset = 0, and J n,9 left_offset = -1, right_offset = 0, and J n,2 left_offset = -1, right_offset = 0, and J n,9 left_offset = -1, right_offset = 0, and J n,2 left_offset = -1, right_offset = 0, and J n,3 left_offset = -1, right_offset = 0, and J n,8 left_offset = -1, right_offset = 0, and J n,9 left_offset = 0, right_offset = 1, and J n,2 left_offset = 0, right_offset = 1, and J n,9 left_offset = 0, right_offset = 1, and J n,2 left_offset = 0, right_offset = 1, and J n,9 left_offset = 0, right_offset = 1, and J n,2 left_offset = 0, right_offset = 1, and J n,3 left_offset = 0, right_offset = 1, and J n,9 left_offset = 0, right_offset = 1, and J n,10 .

[0109] Figure 7 are the schematic diagrams of the weighting according to the position expansion region. As shown in Figure 7 , the position expansion process according to the examples in Figure 6 , when the left offset vector left_offset = [0, 0, -1, -1] and the right offset vector right_offset = [0, 1, 0, 1] are set, four position expansion regions are generated, which are (J n,6 ), (J n,6 , J n,7 ), (Jn,5 ,J n,6 ) and (J n,5 ,J n,6 ,J n,7 ), based on the above four position expansion regions, a weighted correlation step is performed:

[0110] One: for the above four position expansion regions, the Chinese character probability vector located in the position expansion region is replaced by the pinyin mixed weighted vector, and the Chinese character vector located outside the position expansion region is replaced by the Chinese character mixed weighted vector; the above pinyin mixed weighted vector is the sum of the pinyin probability vector multiplied by λ times and the Chinese character probability vector multiplied by (1-λ) times, and the above Chinese character mixed weighted vector is the sum of the pinyin probability vector multiplied by (1-λ) times and the Chinese character probability vector multiplied by λ times, wherein λ∈(0.5,1), which is expressed by formula, the pinyin mixed weighted vector is λ·I n,U +(1-λ)·J n,U , and the Chinese character mixed weighted vector is (1-λ)·I n,U’ +λ·J n,U’ , wherein the subscript contains U indicates the vector in the position expansion region, and the subscript contains U' indicates the vector outside the position expansion region. Similarly, λ∈(0.5,1). As shown in Figure 7 , if λ=0.9 is taken, then: (1) for the first position expansion region, i.e. (J n,6 ), the Chinese character probability vector of J n,6 is replaced by the pinyin mixed weighted vector 0.9·I n,U +0.1·J n,U , and the Chinese character probability vectors of J n,1 to J n,5 and J n,7 are replaced by the Chinese character mixed weighted vector 0.1·I n,U’ +0.9·J n,U’ , where U=6 and U'=1,2,3,4,5,7; (2) for the second position expansion region, i.e. (J n,6 ,J n,7 ), the Chinese character probability vectors of J n,6 and J n,7 are respectively replaced by the pinyin mixed weighted vector 0.9·I n,U +0.1·J n,U , and the Chinese character probability vectors of J n,1 to J n,5 are replaced by the Chinese character mixed weighted vector 0.1·I n,U’ +0.9·J n,U’ , where U=6,7 and U'=1,2,3,4,5; (3) for the third position expansion region, i.e. (J n,5 ,J n,6 ), the Chinese character probability vectors of J n,5, J n,6 are replaced by the pinyin mixed weighted vector 0.9·I n,U +0.1·J n,U , J n,1 to J n,4 , and J n,7 are replaced by the Chinese character mixed weighted vector 0.1·I n,U’ +0.9·J n,U’ , where U=5, 6 and U’=1, 2, 3, 4, 7;(4) for the fourth position expansion region, i.e. (J n,5 , J n,6 , J n,7 ), the Chinese character probability vectors of J n,5 , J n,6 and J n,7 are respectively replaced by the pinyin mixed weighted vector 0.9·I n,U +0.1·J n,U , the Chinese character vectors of J n,1 to J n,4 are replaced by the Chinese character mixed weighted vector 0.1·I n,U’ +0.9·J n,U’ , where U=5, 6, 7 and U’=1, 2, 3, 4. It should be clear that, in the pinyin mixed weighted vector and the Chinese character mixed weighted vector shown in Figure 7 , the order of the elements in the vector does not represent the real position of the element in the vector, but only for the purpose of convenient recording and intuitive representation, the element with larger probability is placed before the element with smaller probability.

[0111] Two: the first mixed batch input is composed of a plurality of mixed weighted matrices generated under a plurality of position expansion regions. As shown in Figure 7 , under four position expansion regions, four mixed weighted matrices are generated, so that the four mixed weighted matrices constitute the first mixed batch input, Figure 7 , the mixed weighted matrix shown in the example in the first mixed batch input is a matrix with a shape of 7xV, and the first mixed batch input is a tensor with a shape of 4x7xV, where V is the number of elements in the adopted vocabulary, i.e. the dimension of the vocabulary.

[0112] Take the embodiments shown in Figure 6 , Figure 7 as an example to illustrate the beneficial effects of the position expansion weighting processing. Before the position expansion processing, the actual content is the voice data of "that woman is called Bella", due to the problems such as non-standard pronunciation, limited recognition ability of the acoustic model, etc., there may be recognition deviation of Chinese pinyin in the pinyin probability matrix 607 in the first pinyin probability tensor output by the acoustic model, for example, Figure 6In the example, the pronunciation of the 5th actual "jiao4" is identified by the acoustic model as the pinyin probability vector [(jiu4, 0.74515),...], and it can be seen that the acoustic model considers the pronunciation of the actual "jiao4" as "jiu4" with the maximum probability of 0.74515, resulting in a judgment deviation. Next, the language model or the first text correction model, based on the first pinyin probability tensor with the judgment deviation, combines the semantic understanding ability of the model itself to output the judgment of the Chinese characters corresponding to the pinyin of the model, that is, to output the first Chinese character probability tensor. Due to the deviation in the first pinyin probability tensor, there may be a recognition deviation in the Chinese character probability matrix 608 in the output first Chinese character probability tensor, for example, Figure 6 In the example, the 5th and 6th actual Chinese characters "jiao" and "bei" are identified by the language model or the first text correction model as the Chinese character probability vector [(jiao, 0.98870),...] and [(bei, 0.49509),...]. Then, in step 501, the position expansion finds the Chinese character probability vector with the maximum probability less than the threshold value in the Chinese character probability matrix 608 as the offset center. In step 502, the position expansion region is generated based on the offset center and the set left and right offset amounts, for example, Figure 6 In the example, the maximum probability of the 6th Chinese character probability vector [(bei, 0.49509),...] is 0.49509, which is less than the set threshold value 0.9, so [(bei, 0.49509),...] is taken as the offset center. Finally, in the mixed weighting of steps 504 and 505, the mixed probability matrix is generated according to the set left and right offset amounts with [(bei, 0.49509),...] as the offset center, and the first mixed batch input is formed.

[0113] If position expansion is not adopted, that is, the processing of step 502 is not performed, but instead the Chinese character probability vector with the maximum probability less than the set threshold found in step 501 is directly replaced with the pinyin mixing matrix, there will be a situation where only [(被, 0.49509), …] is replaced with the pinyin mixing weighted vector, and [(就, 0.98870), …] with the same recognition deviation in the Chinese character probability matrix will not be replaced with the pinyin mixing weighted vector. Furthermore, in the subsequent process, it is impossible to introduce pinyin probability information at the position of [(就, 0.98870), …] with recognition deviation, so that the first text error correction model cannot provide pinyin probability information during the error correction process of [(就, 0.98870), …], assisting the text error correction model to correct errors. As a result, the first text error correction model may consider that there may be no deviation in the judgment of the model for [(就, 0.98870), …] based on the relatively high maximum probability of 0.98870 in [(就, 0.98870), …], thus possibly missing the error correction of [(就, 0.98870), …]. If position expansion is adopted, that is, taking the Chinese character probability vector with the maximum probability less than the set threshold as the offset center, performing left and right offsets to generate more types of position expansion regions, reducing the impact caused by the possible incompleteness or inaccuracy of the judgment result based only on the maximum probability less than the set threshold, and increasing the possibility that the position expansion region actually covers the Chinese character probability vector with actual recognition deviation, as Figure 6 shown in the example in

[0114] If the position expansion weighting is not used, and the Chinese character probability vectors located in the position expansion region are replaced by the pinyin probability vectors, the Chinese character probability vectors located outside the position expansion region are not processed, that is, no longer weighted, and only the Chinese character probability matrix in the first Chinese character probability tensor is converted into a mixed probability matrix composed of pinyin probability vectors and Chinese character probability vectors, which is equivalent to the case where the weighting parameter λ is 1. This will cause the Chinese character probability information of the characters located in the position expansion region to be lost, and the pinyin probability information cannot be fused into the characters located outside the position expansion region to a certain extent, and thus is not conducive to the improvement of the correction effect. If the position expansion weighting is used, the probability vectors of each character position can contain both pinyin probability information and Chinese character probability information, thereby providing more possible valuable information for the first text correction model and being conducive to the improvement of the correction effect of the model.

[0115] Figure 8 An exemplary flowchart of a post-processing process provided by the embodiments of the application is shown, which includes steps 801-803:

[0116] 801. The Chinese character probability vectors of the sample corresponding Chinese character probability matrix in the second Chinese character probability tensor at the same character position are integrated together.

[0117] 802. The characters in the integrated Chinese character probability vectors are searched to find the candidate characters meeting the conditions.

[0118] 803. The candidate character with the maximum probability sum is taken as the final result, the word table is searched for decoding, and the corrected speech recognition result is output.

[0119] In a possible embodiment of the present scheme, if the content shown in Figure 6 , Figure 7 is the speech data of “that woman is called Bella”, the four position expansion regions generated by the four kinds of left and right offsets are taken as examples, and then the integration process of step 801 is as follows: as shown in Figure 7 The first mixed batch input composed of the four mixed weighting matrices is a tensor with a shape of 4x7xV, and after the first mixed batch input is processed by the first text correction model, the first text correction model outputs a second Chinese character probability tensor with a shape of 4x7xV, denoted as K = [[K 1,1 ,K, 1,2 ,…,K 1,7 ],[K 2,1 ,K 2,2 ,…,K 2,7 ],…,[K 4,1 ,K 4,2 ,…,K 4,7 ]]. The above integration process is to integrate the Chinese character probability vectors K 1,U, K 2,U , K 3,U and K 4,U are concatenated; for the integrated Chinese character probability vectors above, assuming for U = 1, K 1,1 = [(na, 0.95), (nέ, 0.02),...], K 2,1 = [(na, 0.93), (nέ, 0.03),...], K 3,1 = [(na, 0.98), (nέ, 0.001),...], K 4,1 = [(na, 0.99), (nέ, 0.002),...], then the integration forms K c,1 = [(na, 0.95), (nέ, 0.02),..., (na, 0.93), (nέ, 0.03),..., (na, 0.98), (nέ, 0.001),..., (na, 0.99), (nέ, 0.002),...], similarly, for U = 2, 3,..., 7, the integrated vectors K c,2 to K c,7 The search described in step 802 can employ search algorithms such as Exhaustive Search, Greedy Search and Beam Search, etc., in the Chinese character probability values in the integrated vectors corresponding to each character, the character sequence corresponding to the sample is searched; it should be clear that the search algorithms exemplified here are only several optional implementation manners of the search process in the present scheme, and are not limiting, and those skilled in the art can also select other suitable search algorithms according to specific application requirements. The decoding process described in step 803 is to find the decoding result corresponding to the number in the character sequence by querying the word table according to the character sequence determined by the search, assuming that the word table used is as shown in Figure 3A , then the decoding result is the Chinese character sequence.

[0120] Figure 9 An exemplary beam search process schematic diagram employed in a possible embodiment provided by the present application is shown, the beam search algorithm includes a parameter beam size, the beam size specifies the number of sequences selected from each candidate sequence, Figure 9The beam search setting beam width is 2, and the selected sequence in each candidate sequence is indicated by bold in the figure. As shown in the figure, the integration vector of the first character is [(jin, 0.6), (jing, 0.4), …], and since the beam width is 2, when the search is performed to the first character, “jin” and “jing” are the two Chinese characters with the largest probability in the integration vector, and therefore “jin” and “jing” are selected as the two candidate sequences of the first step. The integration vector of the second character is [(tian, 0.8), (tian, 0.1), …, (guo, 0.92), (tian, 0.05), …], and the sum of the sequence probabilities of “today”, “jintian”, …, “guojin”, “jingtian” and the like is calculated respectively. Since the beam width is 2, when the search is performed to the second character, “today” and “guojin” are the two sequences with the largest sum of sequence probabilities, and therefore “today” and “guojin” are selected as the two candidate sequences of the second step. The integration vector of the third character is [(tian, 0.65), (tian, 0.3), …, (tian, 0.34), (qu, 0.2), …], and the sum of the sequence probabilities of “today tian”, “today tian”, …, “guo tian”, “jing qu” and the like is calculated respectively. Since the beam width is 2, when the search is performed to the third character, “today tian” and “today tian” are the two sequences with the largest sum of sequence probabilities, and therefore “today tian” and “today tian” are selected as the two candidate sequences of the third step. The beam search process for the fourth, fifth and subsequent characters is similar, and the sequence with the largest sum of probabilities is finally output as the search result.

[0121] The exhaustive search is equivalent to the beam search with the beam width being the dimension of the integration vector, that is, all possible results of each character are searched, and all possible sequences are selected as the candidate sequence before searching the next character. The greedy search is equivalent to the beam search with the beam width being 1, that is, for each character, only the sequence with the largest probability is selected as the candidate sequence. Therefore, the exhaustive search can find the globally optimal sequence, and the greedy search and the beam search may not find the globally optimal sequence. Compared with the above three search algorithms, the exhaustive search has the largest computational cost and the longest time consumption, the greedy search has the smallest computational cost and the shortest time consumption, and the beam search has a computational cost and time consumption between the exhaustive search and the greedy search, which depends on the selection of the hyperparameter beam width. In practical applications, those skilled in the art can select a suitable search algorithm according to actual needs, including but not limited to the above three search algorithms.

[0122] Figure 10 A possible embodiment of the structure of the Chinese text correction model is shown in the figure. Figure 10As shown, the text correction model can include two parts of an embedding layer 1002 and a neural network model 1004. In a possible implementation of the present scheme, the input 1001 and the output 1005 can be three-dimensional tensors, the embedding layer 1002 and the neural network model 1004 can be implemented based on a neural network structure, the function implemented by the embedding layer 1002 is different according to the type of the neural network model 1004 used, the embedding layer 1002 at least has the function of token embedding, the embedding layer output 1003 can be a matrix or a tensor after at least tokenization processing, and the neural network model 1004 can be a convolutional neural network (CNN), a recurrent neural network (RNN), a long short-term memory (LSTM) network, a bidirectional long short-term memory (BiLSTM) network, a bidirectional encoder (BERT) network based on a transformer architecture, or a neural network with other structures. It should be clear that the above examples are only a list of possible implementations, not a limitation, and those skilled in the art can select a neural network with a suitable structure to implement the function of the neural network model 1004 in the text correction model according to the actual application requirements.

[0123] In a possible implementation of the present scheme, the neural network model 1004 can be a Bidirectional Encoder Representations from Transformers (BERT) network (hereinafter referred to as “BERT”) based on a transformer architecture, at this time, the input 1001 and the output 1005 are tensors with a shape of BxUxV, where B represents the number of samples in a batch input, U represents the maximum number of characters in a batch of samples in the batch input, and V represents the number of elements in the vocabulary, i.e., the dimension of the vocabulary; the embedding layer 1002 is a neural network layer with token embedding and position embedding functions, and the embedding layer output 1003 is a tensor with a shape of BxUxD after tokenization and position embedding processing, where D represents the dimension of the vector representing each character in the input sample after embedding layer processing; the token embedding process is as follows: using a token embedding neural network layer with a VxD parameter matrix, a V-dimensional probability vector representing a character in the input sample is converted into a D-dimensional vector; the token embedding process can be understood as a process of changing the data dimension by matrix multiplication, i.e., a character represented by a V-dimensional probability vector is multiplied by a VxD matrix and converted into a character represented by a D-dimensional vector; the position embedding process is as follows: using a position embedding neural network with a LxD parameter matrix, the position information of the characters in the input sample, i.e., the position of the characters in the sentence, is converted into a D-dimensional vector; the position embedding process can be understood as a lookup table process, where L represents the maximum number of position information that the lookup table can represent; characters at the same position in different sample sentences have the same position embedding vector regardless of whether they are the same character or not, for example, the first character “today” in the sentence “today weather how” and the first character “in” in the sentence “eat what at noon” have the same position embedding vector; after completing the tokenization and position embedding processing of the input 1001, the embedding layer 1002 adds the D-dimensional vector obtained by the token embedding process and the D-dimensional vector obtained by the position embedding process, so that the input 1001 with a shape of BxUxV is converted into a tensor with a shape of BxUxD, and the embedding layer output 1003 is obtained; when the BERT used is a pre-trained model, D=768 and L=512, it should be clear that the values of D and L are only examples of possible implementations and are not limiting, and those skilled in the art can use a BERT pre-trained model with specific parameter values or train a BERT model with custom parameter configurations according to actual application requirements.

[0124] In a possible implementation of the present solution, the above-mentioned several models based on the neural network structure can be implemented based on general hardware devices through software programming, or can be implemented based on specific hardware devices with corresponding neural network structures. The former is referred to as a software implementation manner, and the latter is referred to as a hardware implementation manner. In the software implementation manner, the neural network programming framework that can be used includes but is not limited to TensorFlow, PyTorch, Keras, Caffe, and the like. In the hardware implementation manner, the hardware devices that can be used include but are not limited to a field programmable gate array (FPGA).

[0125] Figure 11 An exemplary relationship diagram between a training phase and an inference phase of a text correction model provided by an embodiment of the present application is shown. As shown in the diagram, the training phase is used to obtain a first text correction model, and the inference phase is used to apply the first text correction model. Figure 11

[0126] In a possible implementation of the present solution, in the training phase, the second text correction model with randomly initialized parameters is trained by using the training data with real labels, i.e., the second speech data with corresponding correct text content, based on the configured learning rate and loss function, and through gradient descent and back propagation, the parameters are constantly updated, so that the model constantly learns the data distribution of the training data. When the error calculated based on the configured loss function is less than the configured threshold, the training is completed, the model parameters stop updating, and the trained text correction model, i.e., the first text correction model, is obtained. When the model used is a neural network-based model, the above-mentioned parameters can include the weight and bias values of each neuron in the neural network. Specifically, as shown in the diagram, Figure 11 ​As shown, the training phase used in this scheme can include two phases: Training Phase 1 and Training Phase 2. Both phases require the aforementioned gradient descent, backpropagation, and loss function error calculation steps, resulting in two training iterations of the text correction model. After Training Phase 1, a third text correction model is obtained. This third model then undergoes another round of parameter updates in Training Phase 2, ultimately yielding the first text correction model. During the inference phase, the parameters of the first text correction model are no longer updated, and therefore the aforementioned gradient descent, backpropagation, and loss function error calculation steps are not performed. Instead, the trained first text correction model with fixed parameters is used to perform forward computation on the unlabeled first speech data, outputting the model's judgment result on the text content of the first speech data. The specific process of the inference phase is as described above. Figure 1B Figure 2 Figure 4 and Figure 5 As shown, it will not be explained again here.

[0127] Figure 12 An exemplary diagram illustrating the training phase of a text correction model provided in an embodiment of this application is shown. Figure 12 As shown, before entering the first training phase, training data is prepared. Labeled second speech data is processed by an acoustic model to output a second pinyin probability tensor. The aforementioned labeled second speech data refers to speech data whose corresponding correct text content is known. Therefore, during training, the loss function error can be calculated based on the known correct text content, and the model parameters are updated through gradient descent, backpropagation, and other processes. In an optional implementation of this scheme, if a model based on... Figure 3A Given the probability matrix of the word list, the pinyin probability vector in the pinyin probability matrix of the second pinyin probability tensor has a value of 0 in the Chinese character field and a decimal value in the (0,1) interval in the Chinese pinyin field.

[0128] like Figure 12As shown, at the beginning of the training phase 1201, the parameters of the second text correction model and the Chinese character probability tensor at the output of the second text correction model are initialized. During the first training phase, the second pinyin probability tensor is used as input, and the second text correction model is propagated forward to calculate the loss function error between the Chinese character probability tensor output by the second text correction model and the correct text content corresponding to the second speech data. If the loss function error is greater than a set threshold, gradient descent and backpropagation are performed to update the parameters of the second text correction model and the Chinese character probability tensor at the output of the second text correction model. This process is repeated until the loss function error is less than the set threshold, completing the first training phase. After the first training phase is completed 1202, that is, after the last model parameter update of the first training phase, the third text correction model and the third Chinese character probability tensor at the output of the third text correction model are obtained. In an optional implementation of this scheme, if a model based on... Figure 3A Given the probability matrix of the vocabulary, the Chinese character probability vector in the Chinese character probability matrix of the third Chinese character probability tensor has a value in the range (0,1) as a decimal and a value of 0 in the Chinese character field. In an optional implementation of this scheme, batch input processed by batch normalization can be used as the input of the text correction model during the training phase. At this time, the value of the second pinyin probability tensor has been normalized and can be a tensor of shape B×U×V, where B represents the number of data items in the same batch of input to the second text correction model for forward propagation. Optionally, B can be set to 128. This batch of data may contain second speech data with different corresponding sentence lengths. The number of characters in the longest sentence can be taken as U, and the matrix corresponding to samples with a length less than this can be filled with 0 at the missing character position. V is the number of elements in the vocabulary, i.e., the dimension of the vocabulary. When using the batch input method described above, the model parameters are input with a batch of data and then forward propagated. The loss function error is calculated uniformly, followed by gradient descent and backpropagation to update the model parameters. Using the batch input method for model training can make the training more stable, effectively avoid gradient vanishing and gradient exploding, and improve the fault tolerance of parameter initialization.

[0129] Before the second training stage starts 1203, firstly, the second pinyin probability tensor and the third Chinese character probability tensor obtained after the first training stage are mixed and weighted to form a second mixed batch input, and the third text correction model and the third Chinese character probability tensor obtained after the first training stage are used as the initialized model; during the second training stage, the second mixed batch input is input, the third text correction model is forward propagated, the loss function error between the Chinese character probability tensor output by the third text correction model and the correct text content corresponding to the second speech data is calculated, if the loss function error is greater than the set threshold, gradient descent and back propagation are performed, the parameters of the third text correction model and the third Chinese character probability tensor at the output end of the third text correction model are updated, the above process is repeatedly performed until the loss function error is less than the set threshold, and the second training stage is completed; after the completion of the second training stage 1204, that is, after the last model parameter update of the second training stage, the first text correction model and the fourth Chinese character probability tensor at the output end of the first text correction model are obtained.

[0130] The mixed weighting process before the second training stage is similar to the mixed weighting process in the inference stage. Specifically, the mixed weighting process is as follows: the character probability vector at the character position in the Chinese character probability matrix in the third Chinese character probability tensor that fails to have the maximum probability at the correct Chinese character is replaced by the pinyin probability vector based on the pinyin probability matrix in the second pinyin probability tensor and the Chinese character probability vector of the Chinese character probability matrix in the third Chinese character probability tensor to form a pinyin mixed weighting vector; the character probability vector at the character position in the Chinese character probability matrix in the third Chinese character probability tensor that has the maximum probability at the correct Chinese character is replaced by the Chinese character mixed weighting vector formed by the pinyin probability vector based on the pinyin probability matrix in the second pinyin probability tensor and the Chinese character probability vector of the Chinese character probability matrix in the third Chinese character probability tensor; the pinyin mixed weighting vector is the sum of the pinyin probability vector of the pinyin probability matrix in the second pinyin probability tensor multiplied by λ and the Chinese character probability vector of the Chinese character probability matrix in the third Chinese character probability tensor multiplied by (1-λ); and the Chinese character mixed weighting vector is the sum of the pinyin probability vector of the pinyin probability matrix in the second pinyin probability tensor multiplied by (1-λ) and the Chinese character probability vector of the Chinese character probability matrix in the third Chinese character probability tensor multiplied by λ, where λ ∈ (0.5, 1). The difference between the mixed weighting process before the second training stage and the position expansion weighting process in the inference stage is that the Chinese character probability vector at which position is replaced by the pinyin mixed weighting vector and the Chinese character probability vector at which position is replaced by the Chinese character mixed weighting vector are determined in different ways. Specifically, the correct character corresponding to the input first speech data in the inference stage is unknown, so the Chinese character probability vector at which position is replaced by the pinyin mixed weighting vector and the Chinese character probability vector at which position is replaced by the Chinese character mixed weighting vector are determined according to the position expansion region generated by position expansion, while the second training stage uses the second speech data with labels, and the correct character corresponding to the second speech data is known in advance, so the maximum probability of the Chinese character probability vector at which position in the Chinese character probability matrix in the third Chinese character probability tensor is directly and accurately obtained, and the character at which position is determined to be incorrect, and the character at which position is determined to be correct, so that the pinyin probability vector with a higher weight is mixed to form the pinyin mixed weighting vector at the position at which the character corresponding to the maximum probability of the Chinese character probability vector is determined to be incorrect, and the Chinese character probability vector with a higher weight is mixed to form the Chinese character mixed weighting vector at the position at which the character corresponding to the maximum probability of the Chinese character probability vector is determined to be correct. The mixed weighting process can introduce pinyin information into the Chinese character probability tensor, so that in the case where the Chinese character probability judgment result is biased, the relatively correct result can still be inferred according to the information provided by the pinyin probability, so that the text correction model fully utilizes the pinyin and Chinese character information and has better correction ability than the existing text correction model.

[0131] Figure 13 An exemplary flowchart of a first stage of a text correction model training phase is shown. As shown in Figure 13 , the first stage of the text correction model training phase includes steps 1301-1305:

[0132] 1301, parameter initialization: the text correction model and the Chinese character probability tensor at the output end of the text correction model are parameterized.

[0133] 1302, forward propagation: the pinyin probability tensor output by the acoustic model is input into the text correction model, and the calculation result is propagated forward after calculation.

[0134] 1303, error calculation: calculate the total error between the Chinese character probability output by the text correction model and the real character content corresponding to the second speech data.

[0135] 1304, judgment: judge whether the total error is less than the set threshold, if yes, execute steps 1305 and 1306, otherwise, go to step 1307 to complete the first stage of training.

[0136] 1305, model parameter update: if the total error is greater than the set threshold, the error is returned to the text correction model, and the text correction model parameters are updated according to the error.

[0137] 1306, calculate new error: calculate the total error between the Chinese character probability output by the text correction model after parameter update and the real character content corresponding to the speech data. Then go to step 1304 again to judge whether the total error is less than the set threshold.

[0138] 1307, complete the first stage of training: if the total error is less than or equal to the set threshold, the first stage of training is completed, and the text correction model and the Chinese character probability tensor after the completion of the first stage of training are obtained.

[0139] In a possible implementation of the present scheme, the total error in step 1303 above can be calculated using cross entropy loss function, and the calculation formula is: Loss=(1 / N)*Σ itoN [-logp(u i )], where N represents the number of characters in the sample sentence, u i represents the i-th character of the real character, and p(u i) represents the probability value of the i-th character corresponding to the true character in the Chinese character probability tensor, and log represents the logarithmic function with base 2. Taking a segment of speech data with the corresponding true characters being "天气不错" as an example, if in the Chinese character probability matrix corresponding to this sample output by the text error correction model, the probabilities corresponding to these four characters are [(天, 0.25), (气, 0.125), (不, 0.5), (错, 0.25)], then the formula for calculating the loss value using the cross-entropy loss function is: Loss = (1 / 4) * [-log0.25 - log0.125 - log0.5 - log0.25] = 2.

[0140] In a possible implementation manner of this solution, for the "transmitting the error back into the text error correction model and updating the parameters of the text error correction model according to the error" in step 1305 above, it can be implemented by gradient descent and backpropagation. That is, calculate the gradient of the loss function with respect to the model parameters, multiply the gradient by the set learning rate to calculate the size by which the parameters need to be updated, and subtract this size that needs to be updated from the original parameters to obtain the updated model parameters. The gradient descent method adopted can include global gradient descent, stochastic gradient descent, and batch gradient descent according to the different number of samples participating in the calculation each time. Among them, global gradient descent calculates the error using all samples, stochastic gradient descent randomly selects one sample each time to calculate the error, and batch gradient descent calculates the error using a batch of sample data each time.

[0141] Figure 14 Exemplarily shows a flowchart of the second stage of the training of the text error correction model provided by an embodiment of the present application. As Figure 14 shown, the second stage of the training of the text error correction model includes steps 1401 - 1405:

[0142] 140l. Parameter initialization: Initialize the model using the text error correction model and the Chinese character probability tensor obtained after the first stage of training. ​​​​​​​​​​1405、Model parameter update: if the total error is greater than the set threshold, the error is fed back into the text correction model, and the text correction model parameters are updated according to the error.

[0147] 1406、Calculate new error: calculate the total error between the Chinese character probability output by the text correction model after parameter update and the real character content corresponding to the speech data. Then go to step 1404 again to judge whether the total error is less than the set threshold.

[0148] 1407、Complete training phase two: if the total error is less than or equal to the set threshold, the training phase two is completed, and the text correction model and Chinese character probability tensor after the completion of the training phase two are obtained.

[0149] The threshold in step 1304 and step 1404 can be the same or different, and the present scheme does not limit this.

[0150] In a possible implementation manner of the present scheme, the total error in step 1403 can be calculated by using a cross-entropy loss function; and the "error is fed back into the text correction model, and the text correction model parameters are updated according to the error" in step 1405 can be realized by using gradient descent and back propagation. The specific processes of the above cross-entropy loss function, gradient descent and back propagation are as described above, and will not be repeated here.

[0151] In a possible implementation manner of the present scheme, the text correction model is trained by using hundreds of millions of speech data as training data, and the effect of the text correction model of the present scheme is verified on test data containing more than 10,000 speech data, and the obtained experimental effect is shown in Table 1:

[0152] Table 1: Experimental results of text correction effect of text correction model

[0153] Input Word accuracy Sentence accuracy Pinyin probability tensor 94.92％ 71.72％ Hanzi probability tensor 95.06％ 72.86％ Pinyin and Hanzi mixed probability tensor 95.74％ 78.41％ Pinyin and Hanzi mixed weighted probability tensor 95.90％ 78.62％

[0154] Wherein, the word accuracy is 1 minus the word error rate (Word Error Rate, WER); and the sentence accuracy is the proportion of the number of correctly recognized sentences to the total number of sentences. The above word error rate is the total number of words that need to be replaced, deleted or inserted, divided by the percentage of the total number of words in the correct result, and the calculation formula is: 100% x (S+D+I) / N, wherein S represents the number of substituted words, D represents the number of deleted words, I represents the number of inserted words, and N represents the total number of words.

[0155] The experiment of this embodiment compares the word accuracy and sentence accuracy under the input data of four forms of the first text correction model: the pinyin probability tensor, the Hanzi probability tensor, the mixed pinyin and Hanzi probability tensor, and the mixed weighted pinyin and Hanzi probability tensor. As shown in Table 1, the optimal effect is obtained by using the mixed weighted pinyin and Hanzi probability tensor as the input, followed by the mixed pinyin and Hanzi probability tensor, the Hanzi probability tensor, and the pinyin probability tensor. The mixed weighted pinyin and Hanzi probability tensor is the aforementioned first mixed batch input and / or the second mixed batch input, and the mixed probability vector obtained when the weighted parameter λ is 1. Therefore, it can be proved that the mixed use of pinyin and Hanzi probability information can obtain better correction effect than the use of only pinyin probability information or only Hanzi probability information, and the mixed weighted use of pinyin and Hanzi probability information can further improve the correction effect, so that the method of constructing a mixed batch input by position expansion and mixed weighting proposed in the present solution has the beneficial effect of improving the correction ability of the text correction model.

[0156] Figure 15 、 Figure 16 and Figure 17 are schematic diagrams of three application scenarios of the Chinese speech recognition correction model of the present solution.

[0157] Figure 15 is an application of the Chinese speech recognition correction model in the intelligent voice assistant of a terminal device. As shown in Figure 15 (a), the intelligent voice assistant of the terminal device is started, the intelligent voice assistant software is started and runs, and the microphone enters the listening state; as shown in Figure 15 (b), the user speaks a content of “please play me an English song”, the microphone receives the voice signal, and inputs the speech recognition correction model in the intelligent voice assistant. Before the speech recognition correction model completes the correction reasoning process, the display interface of the terminal device can display the content containing the misjudgment character, for example, “English” is recognized as “because” due to factors such as non-standard pronunciation of the user; as shown in Figure 15 (c), the speech recognition correction model in the intelligent voice assistant completes the correction reasoning process, successfully corrects the previous misjudgment “because” to “English”, and based on the semantic understanding of the correct sentence, makes a corresponding response, that is, plays an English song for the user. The intelligent voice assistant equipped with the speech recognition correction model with correction function can make the speech recognition result more accurate, provide more accurate text information for the subsequent semantic understanding of the intelligent voice assistant, and make more accurate responses, so that the intelligent voice assistant is more intelligent.

[0158] Figure 16 is an application of the Chinese speech recognition correction model in the voice input method of a terminal device. As shown inFigure 16 (a) shows that the terminal device receives a short message, and a pinyin keyboard input method interface is displayed by default under the input box; Figure 16 (b) shows that the user clicks "voice input" to switch to a voice input method with error correction function, the microphone is started to receive voice signals, and a voice recognition error correction model in the voice input method is input. Before the voice recognition error correction model completes the error correction reasoning process, the display interface of the terminal device can display the content containing misjudged characters. For example, due to factors such as non-standard pronunciation of the user, "good" in "sorry" is recognized as "roast"; as shown in Figure 16 (c) shows that the voice recognition error correction model in the voice input method completes the error correction reasoning process, and successfully corrects the previously recognized "roast" to "good"; as shown in Figure 16 (d) shows that the user clicks "send" to complete the sending of the short message. The voice input method equipped with the voice recognition error correction model with error correction function can reduce the user's manual correction and change of the automatic recognition result, so that the user can still complete relatively accurate text input in inconvenient typing input situations, such as Figure 16 as shown in the driving state, and improve the user experience.

[0159] Figure 17 The application of the Chinese voice recognition error correction model in the voice-to-text function of the terminal device. As shown in Figure 17 (a) shows that the terminal device receives a voice message, and the user starts the voice-to-text function by long pressing the icon of the voice message; as shown in Figure 17 (b) shows that before the voice recognition error correction model in the voice-to-text function completes the error correction reasoning process, the display interface of the terminal device can display the content containing misjudged characters. For example, due to factors such as non-standard pronunciation of the sender of the received voice message or limited vocabulary in the Chinese voice recognition model vocabulary, "Wi-Fi" is recognized as "skewed hair"; as shown in Figure 17 (c) shows that the voice recognition error correction model in the voice-to-text function completes the error correction reasoning process, and successfully corrects the previously recognized "skewed hair" to "Wi-Fi". The voice-to-text function equipped with the voice recognition error correction model with error correction function can recognize the received voice information with higher accuracy, thereby eliminating the process of playing the voice information, and facilitating the user to understand the message content and reply to the message in time in inconvenient voice information playing and listening situations.

[0160] It should be understood that when used in the specification and the appended claims of the present application, the term "comprising" indicates the presence of the described features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0161] It should also be understood that, in the description of the application and the appended claims, the term "and / or" is used to mean one or more of the associated listed items, as well as the sum of all possible combinations of the associated listed items. It should also be understood that, in the description of the application and the appended claims, the term "if" is used to mean "when" or "upon" or "in response to a determination" or "in response to a detection," as appropriate, depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon a determination" or "in response to a determination" or "upon detecting [a described condition or event]" or "in response to detecting [a described condition or event]," as appropriate, depending on the context.

[0162] As used in the description of the application and the appended claims, the terms "if can be interpreted to mean "when," or "upon," or "in response to a determination," or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "upon a determination" or "in response to a determination" or "upon detecting [a described condition or event]" or "in response to detecting [a described condition or event]," as appropriate, depending on the context.

[0163] In addition, the description in the specification of the application and the appended claims, the terms "first," "second," "third," etc. are used merely as labels for convenience and are not intended to signify relative importance. It should also be understood that, although the terms "first," "second," etc. are used in the text to describe various elements, these elements should not be limited by these terms. These terms are merely used to distinguish one element from another. For example, a first table can be named a second table, and similarly, a second table can be named a first table without departing from the scope of the various described embodiments. The first table and the second table are both tables, but they are not the same table.

[0164] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrase "in one embodiment" or "in some embodiments" in various places in the specification are not necessarily all referring to the same embodiment, although it can. The terms "comprises," "comprising," "including," and "having," as well as variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Also, the terms "first," "second," "third," etc. are used merely as labels, and are not intended to signify relative importance or significance. Also, the terms "a number of" is intended to mean two or more.

[0165] The voice recognition method provided by the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, notebook computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), and the like. The embodiments of the present application do not limit the specific type of electronic device.

[0166] For example, the electronic device can be a station (STATION, ST) in a WLAN, and can be a cellular phone, a cordless phone, a Session Initiation Protocol (SIP) phone, a Personal Digital Assistant (PDA) device, a handheld device with wireless communication function, a computing device, or other processing device connected to a wireless modem, a vehicle-mounted device, a vehicle networking terminal, a computer, a laptop computer, a handheld communication device, a handheld computing device, a satellite radio device, a wireless modem card, a television set top box (STB), a customer premise equipment (CPE), and / or other devices for communicating over a wireless system, and a next-generation communication system, for example, an electronic device in a 5G network or an electronic device in a future evolved Public Land Mobile Network (PLMN) network, and the like.

[0167] By way of example and not limitation, when the electronic device is a wearable device, the wearable device can also be a general term for smart design and development of daily wear using wearable technology, such as glasses, gloves, watches, clothing, and shoes, etc. The wearable device is a portable device that can be directly worn on the body or integrated into the user's clothes or accessories. The wearable device is not only a hardware device, but also a powerful function realized through software support and data interaction, cloud interaction. The general wearable smart device includes a full function, large size, and can realize complete or partial functions without relying on a smart phone, such as a smart watch or smart glasses, etc., and only focuses on a certain application function, and needs to cooperate with other devices such as a smart phone, such as various smart wristbands, smart jewelry, and the like for monitoring vital signs.

[0168] Figure 18A A structural schematic diagram of an electronic device is shown.

[0169] The electronic device 100 can include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headset interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. The sensor module 180 can include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0170] It can be understood that the structure shown in the embodiments of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 can include more or fewer components than shown, or combine certain components, or split certain components, or different arrangement of components. The components shown can be implemented in hardware, software, or a combination of software and hardware.

[0171] In addition, for the description of each component in the electronic device 100, please refer to the related description in the patent application with the publication number CN110519451A and the invention name of "a power-off control method and device for electronic device", the description in the specification from

[0054] to

[0104] will not be repeated here.

[0172] In addition, the mobile communication module 150 and the audio module 170 are supplemented as follows:

[0173] In the embodiments of the present application, the mobile communication module 150 can also be used for information interaction with other electronic devices, that is, it can send voice-related data to other electronic devices, or the mobile communication module 150 can be used to receive voice recognition and / or error correction requests, and encapsulate the received voice recognition and / or error correction requests into messages in a specified format.

[0174] In addition, the electronic device 100 can realize audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headset interface 170D, and the application processor, etc. For example, music playing, recording, etc. When voice recognition and / or error correction is performed, the microphone 170C can be used to pick up the user's voice.

[0175] It should be understood that in actual applications, the electronic device 100 can include more or fewer components than those shown in the figure, and the embodiments of the present application are not limited. The electronic device 100 shown in the figure is only an example, and the electronic device 100 can have more or fewer components than those shown in the figure, can combine two or more components, or can have a different component configuration. The various components shown in the figure can be implemented in hardware, software, or a combination of hardware and software, including one or more signal processing and / or application-specific integrated circuits. Figure 18A

[0176] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a microkernel architecture, a microservice architecture, or a cloud architecture. The embodiments of the present application take the Android system with a layered architecture as an example to exemplarily illustrate the software structure of the electronic device.

[0177] Figure 18B is a software structure block diagram of the electronic device of the embodiments of the present application.

[0178] For the description of the software system of the electronic device, reference can be made to the related description in paragraphs

[0107] -

[0128] of the specification of the patent application with the publication number CN110519451A and the invention name of Shutdown Control Method and Device of Electronic Equipment, which will not be repeated here.

[0179] The working flow of the software and hardware of the electronic device 100 will be exemplarily described below in the scenario of real-time speech recognition and / or error correction of the electronic device 100.

[0180] When the microphone 170C picks up the speech data of the user, a corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the speech data into a raw input event, which is stored in the kernel layer. The application framework layer obtains the raw input event from the kernel layer and performs speech recognition and / or error correction on the speech data by calling the resource manager in the application framework layer.

[0181] It can be understood that the software structure of the electronic device exemplified by the embodiments of the present application is only used for exemplification and does not constitute a specific limitation on the electronic device.

[0182] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0183] ​The embodiment of the present application further provides a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the above-mentioned various method embodiments.

[0184] The embodiment of the present application further provides a computer program product. When the computer program product is run on an electronic device, the electronic device is caused to perform the steps in the above-mentioned various method embodiments.

[0185] The embodiment of the present application further provides a chip system, which comprises a processor coupled with a memory. The processor executes a computer program stored in the memory to implement the steps in the above-mentioned various method embodiments.

[0186] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the embodiment of the present application can also implement all or part of the processes in the above-mentioned method embodiments by a computer program to instruct related hardware to complete, and the computer program can be stored in a computer readable storage medium. The computer program is executed by a processor to implement the steps in the above-mentioned various method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files or some intermediate forms. The computer readable storage medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0187] In the above-mentioned embodiments, the description of each embodiment has its own focus. The parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0188] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0189] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected to achieve the purpose of the embodiment scheme according to actual needs.

[0190] The above embodiments are only used to illustrate the technical solutions of the present application, not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can still be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

[0191] Finally, it should be noted that the above is only a specific embodiment of the present application, but the protection scope of the present application is not limited to this, any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A Chinese speech recognition error correction method, characterized in that, The method comprises: obtaining pinyin information and first Chinese character information of voice data; the pinyin information comprises pinyin probabilities, and the first Chinese character information comprises Chinese character probabilities; performing weighted fusion on the pinyin probabilities in the pinyin information and the Chinese character probabilities in the first Chinese character information to obtain mixed information comprising a plurality of sub-mixed information; applying a text correction model to process the mixed information to obtain second Chinese character information; outputting a corrected speech recognition result; wherein the second Chinese character information comprises the corrected speech recognition result; and the text correction model is a neural network model.

2. The method of claim 1, wherein, Before the weighted fusion on the pinyin probabilities in the pinyin information and the Chinese character probabilities in the first Chinese character information, the method further comprises: determining, in the Chinese character probabilities in the first Chinese character information, a position of a Chinese character whose probability is less than a threshold value, and performing the weighted fusion according to the position.

3. The method of claim 2, wherein, The weighted fusion on the pinyin probabilities in the pinyin information and the Chinese character probabilities in the first Chinese character information specifically comprises: obtaining a plurality of position expansion regions based on the position according to a preset rule; wherein the preset rule comprises a plurality of left offsets and a plurality of right offsets; and the position expansion region covers the position, the left offset positions on the left side of the position, and the right offset positions on the right side of the position.

4. The method of claim 3, wherein, The weighted fusion on the pinyin probabilities in the pinyin information and the Chinese character probabilities in the first Chinese character information further specifically comprises: replacing the Chinese character probabilities in the first Chinese character information within the position expansion region with pinyin mixed weighted probabilities, and replacing the Chinese character probabilities in the first Chinese character information outside the position expansion region with Chinese character mixed weighted probabilities, to complete the weighted fusion.

5. The method of claim 4, wherein: the pinyin mixed weighted probability is obtained by weighted addition of the pinyin probability in the pinyin information multiplied by a first weight and the Chinese character probability in the first Chinese character information multiplied by a second weight; the Chinese character mixed weighted probability is obtained by weighted addition of the Chinese character probability in the first Chinese character information multiplied by the first weight and the pinyin probability in the pinyin information multiplied by the second weight; and the first weight is greater than the second weight.

6. The method of any one of claims 1-5, wherein: the pinyin information is a pinyin probability tensor comprising a pinyin probability matrix composed of pinyin probability vectors; the first Chinese character information is a Chinese character probability tensor comprising a Chinese character probability matrix composed of Chinese character probability vectors; the mixed information is a mixed tensor comprising a pinyin Chinese character mixed probability matrix composed of pinyin Chinese character mixed probability vectors.

7. The method of claim 6, wherein, the pinyin probability vector and the Chinese character probability vector are word table-based probability vectors; and the word table comprises a plurality of pinyins and a plurality of Chinese characters; a value corresponding to the pinyin in the word table in the pinyin probability vector is non-zero, and a value corresponding to the Chinese character in the word table is zero. The value corresponding to the pinyin in the vocabulary in the Chinese character probability vector is zero, and the value corresponding to the Chinese character in the vocabulary is nonzero; The values corresponding to the pinyin and Chinese character in the vocabulary in the pinyin-Chinese character mixed probability vector are nonzero.

8. The method of claim 6, wherein, The pinyin probability in the pinyin information and the Chinese character probability in the first Chinese character information are weighted and fused to obtain mixed information containing multiple sub-mixed information, specifically including: The pinyin probability vector in the pinyin probability tensor and the Chinese character probability vector in the Chinese character probability tensor are weighted and fused to obtain the mixed tensor containing multiple pinyin-Chinese character mixed probability matrices.

9. The method of claim 8, wherein, Before the pinyin probability vector in the pinyin probability tensor and the Chinese character probability vector in the Chinese character probability tensor are weighted and fused, the method further includes: In the Chinese character probability vector in the Chinese character probability tensor, the Chinese character probability vector whose maximum Chinese character probability is less than a threshold value is determined in the position of the Chinese character probability matrix, and the weighted fusion is performed according to the position; wherein the maximum Chinese character probability is the maximum probability value in the Chinese character probability vector.

10. The method of claim 1, wherein, The pinyin information and the first Chinese character information of the speech data are obtained, specifically including: The pinyin information is processed by applying the text error correction model to obtain the first Chinese character information.

11. The method of claim 10, wherein, The pinyin information and the first Chinese character information of the speech data are obtained, specifically further including: The pinyin information is obtained by processing the speech data by applying an acoustic model; The acoustic model is a neural network model.

12. An electronic device, comprising: A computer program product for performing the method of any one of claims 1-11 to recognize and correct Chinese speech.

13. A computer-readable storage medium, characterized in that, Computer instructions for performing the method of any one of claims 1-11 are stored.

14. A chip device, characterized by Computer instructions for performing the method of claim 13.

Images