Speech recognition model training, testing, speech recognition method and device

Abstract

The present disclosure provides a method and device for training a speech recognition model, which relates to the field of artificial intelligence technology, specifically to technical fields such as speech recognition, deep learning, and large models, and can be applied to scenarios such as content generation in artificial intelligence. The specific implementation scheme is as follows: obtaining a speech sample set, the speech sample set including at least one speech sample, the speech sample including: an audio feature sequence and an initial word unit sequence; obtaining an initial speech recognition model, the speech recognition model being used to characterize the correspondence between the audio feature sequence and the predicted word unit sequence; replacing the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language, to obtain a training sample set, wherein the predicted word units are predicted word units in the predicted word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model; training the speech recognition model based on the training sample set to obtain a trained speech recognition model.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, specifically to technical fields such as speech recognition, deep learning, and large models, and can be applied to scenarios such as artificial intelligence content generation. In particular, it relates to a speech recognition model training method and device, a speech recognition model testing method and device, a speech recognition method and device, an electronic device, a computer-readable storage medium, and a computer program product. Background Art

[0002] Speech recognition technology has become increasingly mature, especially for widely used languages ​​like English and Chinese, where it can now rival the performance of professional human speech recognition. However, existing models have limited speech recognition capabilities for some languages ​​due to objective factors such as the limited number of training samples. Summary of the Invention

[0003] The present disclosure provides a speech recognition model training method and device, a speech recognition model testing method and device, a speech recognition method and device, an electronic device, a computer-readable storage medium, and a computer program product.

[0004] According to a first aspect, a speech recognition model training method is provided, the method comprising: obtaining a speech sample set, the speech sample set comprising at least one speech sample, the speech sample comprising: an audio feature sequence and an initial word unit sequence; obtaining an initial speech recognition model, the speech recognition model being used to characterize the correspondence between the audio feature sequence and the predicted word unit sequence; replacing the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language to obtain a training sample set, the predicted word units being predicted word units in the predicted word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model; training the speech recognition model based on the training sample set to obtain a trained speech recognition model.

[0005] According to the second aspect, a speech recognition model testing method is provided, the method comprising: obtaining a test set and a trained speech recognition model, the test set comprising at least one test sample, the test sample comprising: an audio feature sequence and a test word unit sequence; the trained speech recognition model is trained using the speech recognition model training method as described in any implementation of the first aspect, the speech recognition model comprising: an encoder, a decoder and a keyword module; selecting a test sample from the test set; inputting the audio feature sequence in the test sample into the encoder to obtain audio intermediate features; inputting the audio intermediate features and the current word unit sequence into the decoder to obtain a predicted word unit output by the speech recognition model; in response to the predicted word unit being a non-terminal symbol, updating the current word unit sequence based on the predicted word unit, continuing to input the audio intermediate features and the current word unit sequence into the decoder until the predicted word unit output by the speech recognition model is a terminator, thereby obtaining all predicted word units corresponding to the test sample; based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample, detecting whether the speech recognition model has passed the test.

[0006] According to the third aspect, a speech recognition method is provided, which includes: obtaining a speech to be recognized; processing the speech to be recognized to obtain audio feature data; inputting the audio feature data into a speech recognition model to obtain a predicted word unit sequence of the speech to be recognized, wherein the speech recognition model is a trained speech recognition model obtained by using the speech recognition model training method described in any implementation method of the first aspect; and obtaining text data of the speech to be recognized based on the predicted word unit sequence.

[0007] According to a fourth aspect, a speech recognition model training device is provided, which includes: a sample acquisition unit, configured to acquire a speech sample set, the speech sample set including at least one speech sample, the speech sample including: an audio feature sequence and an initial word unit sequence; a model acquisition unit, configured to acquire an initial speech recognition model, the speech recognition model being used to characterize the correspondence between the audio feature sequence and the prediction word unit sequence; a replacement unit, configured to replace the language word units in the initial word unit sequence in the speech sample set with prediction word units representing the language, to obtain a training sample set, the prediction word units being prediction word units in the prediction word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model; and a training unit, configured to train the speech recognition model based on the training sample set, to obtain a trained speech recognition model.

[0008] According to the fifth aspect, a speech recognition model testing device is provided, which includes: an information acquisition unit, configured to acquire a test set and a trained speech recognition model, the test set including at least one test sample, the test sample including: an audio feature sequence and a test word unit sequence; the trained speech recognition model is trained using the speech recognition model training device described in any implementation method of the fourth aspect, the speech recognition model including: an encoder, a decoder and a keyword module; a selection unit, configured to select a test sample from the test set; an input unit, configured to input the audio feature sequence in the test sample into the encoder to obtain audio intermediate features; an acquisition unit, configured to input the audio intermediate features and the current word unit sequence into the decoder to obtain a predicted word unit output by the speech recognition model; an update unit, configured to update the current word unit sequence based on the predicted word unit in response to the predicted word unit being a non-terminal symbol, and continue to input the audio intermediate features and the current word unit sequence into the decoder until the predicted word unit output by the speech recognition model is a terminator, thereby obtaining all predicted word units corresponding to the test sample; a testing unit, configured to detect whether the speech recognition model has passed the test based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample.

[0009] According to the sixth aspect, a speech recognition device is provided, which includes: a speech acquisition unit, configured to acquire speech to be recognized; a processing unit, configured to process the speech to be recognized and obtain audio feature data; a recognition unit, configured to input the audio feature data into a speech recognition model to obtain a predicted word unit sequence of the speech to be recognized, and the speech recognition model is a trained speech recognition model obtained by the speech recognition model training device described in any implementation method of the fourth aspect; and a conversion unit, configured to obtain text data of the speech to be recognized based on the predicted word unit sequence.

[0010] According to the seventh aspect, an electronic device is provided, which includes: at least one processor; and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described in any implementation of the first aspect, the second aspect, or the third aspect.

[0011] According to an eighth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, where the computer instructions are used to cause a computer to execute the method described in any one of the implementations of the first aspect, the second aspect, or the third aspect.

[0012] According to a ninth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the method described in any one of the implementations of the first aspect, the second aspect, or the third aspect.

[0013] The embodiments of the present disclosure provide a speech recognition model training method and device. First, a speech sample set is obtained. The speech sample set includes at least one speech sample. The speech sample includes: an audio feature sequence and an initial word unit sequence. Second, an initial speech recognition model is obtained. The speech recognition model is used to characterize the correspondence between the audio feature sequence and the predicted word unit sequence. Third, a predicted word unit representing the language is used to replace the language word unit in the initial word unit sequence in the speech sample set to obtain a training sample set. The predicted word unit is a predicted word unit in the predicted word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model. Finally, based on the training sample set, the speech recognition model is trained to obtain a trained speech recognition model. Therefore, before training the speech recognition model, replacing the language word unit in the initial word unit sequence in the speech sample set with the predicted word unit representing the language can enable the model to predict the audio feature sequence in its familiar word unit environment, thereby improving the model convergence speed and the model training effect.

[0014] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0016] Figure 1 is a flowchart of an embodiment of a speech recognition model training method according to the present disclosure;

[0017] Figure 2 It is a structural diagram of the keyword module training process disclosed in the present invention;

[0018] Figure 3 is a flow chart of an embodiment of a method for testing a speech recognition model according to the present disclosure;

[0019] Figure 4 This is a structural diagram of the test process of the speech recognition model disclosed in the present invention;

[0020] Figure 5 is a flow chart of an embodiment of a speech recognition method according to the present disclosure;

[0021] Figure 6 is a structural diagram of an embodiment of a speech recognition model training device according to the present disclosure;

[0022] Figure 7 is a structural diagram of an embodiment of a speech recognition model testing device according to the present disclosure;

[0023] Figure 8 is a structural diagram of an embodiment of a speech recognition device according to the present disclosure;

[0024] Figure 9 It is a block diagram of an electronic device used to implement the speech recognition model training method, speech recognition model testing method, or speech recognition method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0025] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] Traditional speech recognition technology primarily relies on acoustic and language models to achieve speech-to-text conversion. Acoustic models, such as the Hidden Markov Model and Gaussian Mixture Model, are used to build relationships between speech and text. Language models, such as the N-gram model, are used to build relationships between words in text. Due to the limitations of hand-crafted speech features and non-end-to-end recognition solutions, traditional speech recognition technology cannot match human performance.

[0027] Deep learning-based speech recognition technology can automatically extract useful feature representations from raw speech data and use complex network model structures to learn the complex features and structures in speech signals. This enables speech recognition technology to achieve higher accuracy and robustness when processing non-standard voices, accents, dialects, and speech at varying speeds.

[0028] With the widespread use of models like Transformer in speech recognition and the rapid development of computer hardware, speech recognition technology has made a qualitative leap. For widely used languages ​​like English and Chinese, speech recognition technology is now comparable to that of professionals. However, for less widely used languages, speech recognition models remain relatively weak.

[0029] Based on this, this disclosure proposes a speech recognition model training method. Figure 1 A process 100 according to an embodiment of a speech recognition model training method of the present disclosure is shown. The speech recognition model training method includes the following steps:

[0030] Step 101: Acquire a speech sample set.

[0031] In this embodiment, the speech sample set is a collection of speech samples of the language to be recognized. For example, the language to be recognized is a language with few samples. By using a speech recognition model to recognize the speech of this type of language and obtain the text of this type of language, the dissemination, learning and research of this type of language can be improved.

[0032] In this embodiment, the execution subject on which the speech recognition model training method is executed can obtain the speech sample set in various ways. For example, the execution subject can obtain the speech sample set stored in the database server through a wired connection or a wireless connection. For another example, a user can obtain the speech sample set collected by the terminal by communicating with the terminal.

[0033] Here, the speech sample set may include at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence; wherein the audio feature sequence is a sequence obtained after feature extraction of the audio, such as Mel features, Fbank (FilterBank) features, etc.

[0034] In this embodiment, the initial word unit sequence, also called the token sequence, is obtained by dividing the text corresponding to the audio feature sequence into language units such as words, punctuation marks, numbers or pure letters, and the initial unit sequence contains word units that represent the language of the text corresponding to the audio feature sequence.

[0035] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of the voice sample sets involved are carried out after authorization and in compliance with relevant laws and regulations.

[0036] Step 102: Obtain an initial speech recognition model.

[0037] In this embodiment, a speech recognition model is used to characterize the correspondence between an audio feature sequence and a predicted word unit sequence. Specifically, the speech recognition model can be a Whisper model, which is a standard transformer model. The Whisper model takes fixed-length (30 seconds) audio as input, first extracts the audio's Fbank features, and feeds the Whisper model's encoder to generate intermediate features of the audio. The Whisper model's decoder then decodes the intermediate features and outputs the predicted word unit sequence predicted by the model.

[0038] Optionally, the speech recognition model can also be an MMS (Massively Multilingual Speech) model. The MMS model can recognize more than 4,000 spoken languages. Compared with the Whisper model, the MMS model has a larger number of model parameters, more pre-training data, and more powerful model functions. However, the MMS model has a huge number of parameters and the deployment cost is too high.

[0039] Step 103: Use the predicted word units representing the language to replace the language word units in the initial word unit sequence in the speech sample set to obtain a training sample set.

[0040] In this embodiment, the predicted word unit is a predicted word unit in a predicted word unit sequence obtained by inputting a speech sample selected from a speech sample set into a speech recognition model.

[0041] In this embodiment, a speech sample is selected from the speech sample, and the audio feature sequence of the speech sample is input into the speech recognition model, so that a predicted word unit sequence output by the speech recognition model can be obtained, wherein one predicted word unit in the predicted word unit sequence is a word unit representing a language. The predicted word unit representing the language in the predicted word unit sequence replaces the language word unit in the initial word unit sequence in the speech sample set, so that the speech recognition model can be more familiar with the training sample set.

[0042] In this embodiment, the language word units in the initial word unit sequence are replaced by the predicted word units representing the language to obtain a new word unit sequence, and a training sample set including the audio feature sequence and the new word unit sequence can be obtained.

[0043] In this embodiment, the training sample includes: an audio feature sequence X = (x1, x2, ... x k ) and its corresponding token sequence Label=(l1,l2,…l m ). For language recognition tasks with few samples, Label records the text information of the language corresponding to the current audio. The speech recognition model takes the audio sequence X as input and outputs the predicted token sequence Output = (o1, o2, ...o m During training, the cross-entropy loss function is typically used to measure the difference between the label and the output, ensuring that the output is as consistent as possible with the label. During prediction, the output of the speech recognition model can be used to obtain the text in the language predicted by the model.

[0044] Speech recognition models can be divided into non-autoregressive models and non-autoregressive models based on how the output is formed. The non-autoregressive model uses the audio sequence X as the output and decodes the output in parallel; the autoregressive model uses the audio sequence X as the output and decodes the output in serial. The non-autoregressive model uses parallel decoding, which is fast, but it tends to ignore the contextual relationship between audio sequences, resulting in poor model performance; the autoregressive model uses serial decoding, which is slow, but it can construct the contextual relationship between audio sequences, resulting in better model performance. Since there are fewer samples of this type of language and the training difficulty is slightly greater, in order to ensure the effectiveness of the speech recognition model, the present disclosure adopts the decoding method of the autoregressive model.

[0045] In this embodiment, the predicted word unit sequence output by the speech recognition model is an array, such as [223, 2124, 2323, 45], where each number represents a different word, which is the Output above. To improve the training effect of the speech recognition model, during the training process of the speech recognition model, the Output can be obtained all at once (non-autoregressive), while when the speech recognition model is tested, the predicted word units in the Output can be obtained one by one (autoregressive).

[0046] Step 104: train the speech recognition model based on the training sample set to obtain a trained speech recognition model.

[0047] In this embodiment, the execution entity may select training samples from the training sample set obtained in step 103 and perform the training steps from step A to step B to complete one iterative training of the speech recognition model. The method and number of training samples selected from the training sample set are not limited in this application, and the number of iterative trainings of the speech recognition model is also not limited.

[0048] Step A: Calculate the network loss value of the speech recognition model based on the training samples input into the speech recognition model.

[0049] In this embodiment, during each iterative training of the speech recognition model, training samples are selected from the training sample set, and the selected training samples are input into the speech recognition model. Based on the predicted word unit sequence output by the speech recognition model, the loss value of the speech recognition model is calculated.

[0050] In this embodiment, the loss function for calculating the loss value of the speech recognition model can use a cross-entropy loss function. The cross-entropy loss function measures the difference between two different probability distributions for the same random variable. In machine learning, this is represented as the difference between the true probability distribution and the predicted probability distribution. The smaller the value of the cross-entropy loss function, the better the prediction performance of the speech recognition model.

[0051] Optionally, the loss function of the speech recognition model can also adopt the mean square error function, which is the expectation of the square of the difference between the predicted word unit sequence of the speech recognition model and the true value (the new word unit sequence in the training sample). During the iterative training process of the speech recognition model, the gradient descent algorithm can be used to minimize the loss function of the speech recognition model, thereby iteratively optimizing the network parameters of the speech recognition model.

[0052] The original meaning of a gradient is a vector representing the point at which the directional derivative of a loss function reaches its maximum value along that direction. This means that the loss function changes most rapidly and at the highest rate of change along that direction at that point. In deep learning, the primary task of neural networks is to find the optimal network parameters (weights and biases) during learning. These optimal network parameters are also the parameters that minimize the loss function.

[0053] Step B: Based on the loss value of the speech recognition model, the speech recognition model is trained to obtain a trained speech recognition model.

[0054] In this embodiment, the trained speech recognition model is a trained speech recognition model obtained by inputting the selected speech samples into the speech recognition model for iterative training and adjusting the parameters of the speech recognition model.

[0055] In this embodiment, the loss value of the speech recognition model can be used to detect whether the speech recognition model meets the training completion conditions. After the speech recognition model meets the training completion conditions, a trained speech recognition model is obtained.

[0056] In this embodiment, the training completion condition includes: the loss value of the speech recognition model is less than a first loss value threshold. The first loss threshold can be determined based on specific training requirements, for example, the first loss threshold is 0.01.

[0057] Optionally, in this embodiment, in response to the speech recognition model not meeting the training completion conditions, the relevant parameters in the speech recognition model are adjusted so that the loss value of the speech recognition model converges, and based on the adjusted speech recognition model, the above-mentioned training steps A-B are continued.

[0058] In this embodiment, when the speech recognition model does not meet the training completion conditions, adjusting the relevant parameters of the speech recognition model helps to help the loss value of the speech recognition model converge.

[0059] In this embodiment, for the Whisper model that recognizes languages ​​with few samples, since the token sequence (word unit sequence) received by the Whisper model must contain the token (word unit) of the language, it determines which type of language the model performs speech recognition tasks. However, since the original Whisper model has not been trained on data of this type of language, its token sequence lacks the token of this type of language. For this reason, the speech recognition model was trained many times, and attempts were made to replace the token of this type of language with tokens of other languages ​​such as Chinese and English; or to assign a new language token to this type of language, but the effect of the model after training did not meet expectations. When the Whisper model was used to perform a language category test on the audio of this type of language, it was found that the Whisper model would predict most of the audio of this type of language as another type of language (predicted word units representing speech), and replacing the token of this type of language with tokens of another type of language could achieve good training results.

[0060] In this embodiment, the above-mentioned loss value based on the speech recognition model is used to train the speech recognition model to obtain the trained speech recognition model, including: in response to the loss value of the speech recognition model being less than the loss threshold, determining that the speech recognition model meets the training completion condition, and using the feature recognition subnetwork and classification subnetwork in the speech recognition model that meets the training completion condition as the speech recognition model.

[0061] The speech recognition model training method provided by the embodiments of the present disclosure includes the following steps: first, obtaining a speech sample set, the speech sample set including at least one speech sample, the speech sample including: an audio feature sequence and an initial word unit sequence; second, obtaining an initial speech recognition model, the speech recognition model being used to characterize the correspondence between the audio feature sequence and the predicted word unit sequence; third, replacing the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language, to obtain a training sample set, wherein the predicted word units are predicted word units in the predicted word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model; finally, training the speech recognition model based on the training sample set to obtain a trained speech recognition model. Thus, before training the speech recognition model, replacing the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language can enable the model to predict the audio feature sequence in its familiar word unit environment, thereby improving the model convergence speed and the model training effect.

[0062] In some optional implementations of the present disclosure, the above-mentioned speech recognition model includes: an initial recognition sub-model, and a keyword module connected to the initial recognition sub-model; based on the training sample set, the speech recognition model is trained to obtain a trained speech recognition model including: based on the training sample set, the initial recognition sub-model is trained to obtain a trained recognition sub-model; based on the training sample set and the trained recognition sub-model, the keyword module is trained to obtain a trained keyword module; the trained recognition sub-model and the trained keyword module are used as the trained speech recognition model.

[0063] like Figure 2 As shown, the speech recognition model includes: a recognition sub-model and a keyword module. The training samples in the training sample set are input into the recognition sub-model, and the recognition sub-model is trained multiple times to obtain a trained recognition sub-model.

[0064] The method for training a speech recognition model provided by this optional implementation method first trains the initial recognition sub-model to obtain a trained recognition sub-model; then, based on the training sample set and the trained recognition sub-model, the keyword model is trained to obtain a trained keyword module, and the initial recognition sub-model and the keyword module are trained step by step, thereby improving the reliability of the trained speech recognition model.

[0065] In some optional implementations of the present disclosure, the above-mentioned training of the keyword module based on the training sample set and the trained recognition sub-model to obtain the trained keyword module includes: fixing the parameters of the trained recognition sub-model; inputting the training samples in the training sample set into the trained recognition sub-model to obtain the first probability distribution output by the trained recognition sub-model and the second probability distribution output by the keyword module; calculating the loss value of the initial word unit in the training sample based on the first probability distribution and the second probability distribution; and obtaining the trained keyword module based on the loss value.

[0066] In this optional implementation, if Figure 2 As shown, the recognition sub-model is the Whisper model. In order to improve the model's ability to recognize specific keywords (bias words), the keyword module of the present disclosure can use the TCPGen (Tree-constrained pointer generator) module. The TCPGen module can help the Whisper model (the trained recognition sub-model at this time) improve its ability to recognize keywords. First, a keyword sequence (biasing list) that needs to be recognized is generated. Then, the TCPGen module generates a keyword prefix tree based on the keyword sequence, and generates a probability distribution P for all tokens by traversing the tree. ptr (y i ) and the generation probability Pgen (y i Specifically, the TCPGen module first traverses the keyword prefix tree to obtain the token sequence corresponding to the keyword, then obtains the embedding corresponding to each token sequence, and finally uses the fully connected layer and softmax function to generate the probability distribution P ptr (y i ) and the generation probability P gen (y i ). Finally, the two will be combined with the token probability distribution P generated by the Whisper model mdl (y i ) (Although the parameters of the Whisper model are fixed, the Whisper model can also perform forward prediction) perform weighted summation to obtain the final token probability distribution P(y i ), specifically, as shown in formula (1):

[0067] P(y i )=P mdl (y i )(1-P gen (y i ))+P ptr (y i )P gen (y i ) (1)

[0068] In this optional implementation, the keyword model is trained on the basis of the completion of the training of the recognition sub-model, which can ensure the training effect of the keyword model. Specifically, the Whisper model needs to adopt the technology of shifting the word unit sequence right by one position during training, but through training, it is found that the technology of shifting the word unit sequence right by one position should not be used when training the TCPGen module alone, otherwise the training loss will oscillate, resulting in the inability to train the model. This is because the TCPGen module does not directly predict the token sequence, but makes corrections based on the prediction results of the original Whisper model. For this reason, after the Whisper model training is completed, the TCPGen module is trained without using the technology of shifting the word unit sequence right by one position, which can improve the training effect of the TCPGen module.

[0069] The method for training a keyword module provided in this embodiment first fixes the parameters of the trained recognition sub-model; then, based on the training sample set, calculates the first probability score output by the trained recognition sub-model and the second probability distribution output by the keyword module; based on the first probability distribution and the second probability distribution, calculates the loss value of the initial word unit in the training sample, and based on the loss value, obtains the trained keyword module, thereby improving the reliability of keyword module training.

[0070] In some optional implementations of the present disclosure, the above-mentioned training of the initial recognition sub-model based on the training sample set to obtain the trained recognition sub-model includes: selecting training samples from the training sample set to obtain selected samples; shifting the new word units in the new word unit sequence in the selected samples right by one position to obtain a training word unit sequence; training the initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected samples to obtain the trained recognition sub-model.

[0071] In this optional implementation, the new word unit sequence is a word unit sequence obtained by replacing the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language, and the new word units are word units in the new word unit sequence.

[0072] In this optional implementation, the recognition sub-model can be a Whisper model. When training the Whisper model, in order to adapt to the characteristics of the Whisper autoregressive model, the word unit sequence is shifted right by one position (shift_tokens_right) during the training process. That is, the model needs to predict the next word in the token sequence. To achieve this, the token sequence input to the model needs to be shifted right by one unit. However, through training, it was found that the word unit sequence shifted right by one position should not be used when training TCPGen alone, otherwise the training loss will oscillate, making the model unable to be trained. This is because the TCPGen module does not directly predict the token sequence, but makes corrections based on the prediction results of the original Whisper model.

[0073] In this optional implementation, the new word unit in the new word unit sequence in the selected sample is shifted right by one position in order to enable the initial recognition sub-model to effectively predict the next Token.

[0074] The method for training the initial recognition sub-model provided in this embodiment selects training samples from a training sample set, shifts the new word units in the new word unit sequence in the selected samples right by one position to obtain a training word unit sequence, and trains the initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected samples, thereby improving the reliability of the trained recognition sub-model and improving the training effect of the speech recognition model.

[0075] In some optional implementations of the present disclosure, the above-mentioned acquisition of the speech sample set includes: acquiring an initial data set including at least one initial speech; preprocessing the initial data set to obtain a processed data set; performing data enhancement on the processed data set to obtain a speech data set; and obtaining a speech sample set based on the speech data set.

[0076] In this optional implementation, the initial speech is speech data of a single language, and the initial speech may be unprocessed speech data. A speech sample set is obtained by processing the initial speech in the initial dataset. For example, if the initial speech is in a rare language, the resulting speech sample set is a sample set of the rare language, and a speech recognition model can be trained using this sample set.

[0077] In this optional implementation, data preprocessing is the process of preparing original data and making it suitable for the machine learning model. Through data preprocessing, the preprocessed processed data set can be suitable for the training of the machine learning model. The above-mentioned preprocessing of the initial data set to obtain the processed data set includes: removing unrecognizable information such as punctuation marks in the initial data.

[0078] In this optional implementation, based on the problem that the initial data set has little sample information in the language, data enhancement is performed on the processed data set, thereby increasing the amount of speech data in the speech data set.

[0079] In this optional implementation, obtaining a speech sample set based on a speech dataset includes: extracting speech features of each speech data in the speech dataset to obtain speech features of each speech data; limiting each speech feature to a fixed length to obtain audio features of each speech feature; determining an initial word unit of each speech data, and obtaining a speech sample set including at least one speech sample. The speech features may be Mel features or Fbank (Filter Banks) features, and speech feature extraction can be performed using established methods, which will not be further described here.

[0080] The method for obtaining a speech sample set provided by this optional implementation first preprocesses the initial data set to obtain a processed data set; then performs data enhancement on the processed data set to obtain a speech data set; and based on the speech data set, obtains a speech sample set, thereby improving the reliability of obtaining the speech sample set.

[0081] In some optional implementations of the present disclosure, the above-mentioned preprocessing of the initial data set to obtain a processed data set includes at least one of the following: sampling all initial data in the initial data set at the same sampling rate; deleting duplicate and unrecognizable initial data in the initial data set.

[0082] Optionally, the preprocessing of the initial data set to obtain the processed data set may further include: processing all initial data in the initial data set so that all the initial data are located in the same audio channel.

[0083] Optionally, preprocessing the initial dataset to obtain a processed dataset may also include generating new initial data using speech synthesis technology, and adding this new initial data to the processed dataset. Since languages ​​with fewer samples have fewer training samples, the dataset size can be increased by supplementing them with synthesized audio of these languages ​​as training samples. Training has shown that excessive synthesized datasets are not suitable, as this can negatively impact training effectiveness, primarily due to difficulties in ensuring the quality of the synthesized data. Therefore, the amount of new initial data can be kept to approximately 30% of the total speech sample set.

[0084] This optional implementation provides a method for preprocessing an initial data set, including sampling the initial data set at the same sampling rate and deleting repeated and unrecognizable initial data in the initial data set, thereby improving the diversity of speech preprocessing.

[0085] In some optional implementations of the present disclosure, the above-mentioned data enhancement of the processed data set to obtain a speech data set includes at least one of the following: modifying the speaking rate of each processed data in the processed data set to obtain an added data set, and adding the added data set to the processed data set; adding reverberation to each processed data in the processed data set to obtain an reverberation data set, and adding an reverberation data set to the processed data set; adding noise to each processed data in the processed data set to obtain a noise data set, and adding a noise data set to the processed data set; modifying the audio spectrum graph of each processed data in the processed data set to obtain a spectrum data set, and adding a spectrum data set to the processed data set.

[0086] In this optional implementation, the speaking speed of the processed data (audio) in the processed data set is modified, the number of training sets is increased, and the data is made more diverse; reverberation is randomly added to the processed data in the processed data set to achieve the purpose of simulating multiple sound scenes; noise is randomly added to the processed data in the processed data set to increase data complexity; Specaugment technology (a technology used for speech recognition enhancement) can be used to modify the audio spectrum of the processed data in the processed data set to improve the quality of audio features.

[0087] The method for data enhancement of processed data provided by this optional implementation includes: modifying the speaking speed of the processed data in the processed data, adding reverberation to the processed data, adding noise to the processed data, and modifying at least one or more of the audio spectrograms of the processed data to improve the multi-sample nature of the speech data set.

[0088] Furthermore, based on the speech recognition model training method provided in the above embodiment, the present disclosure also provides an embodiment of a speech recognition model testing method. The speech recognition model testing method of the present disclosure combines artificial intelligence fields such as speech recognition and deep learning.

[0089] See also Figure 3 , shows a process 300 according to an embodiment of the speech recognition model testing method of the present disclosure. The speech recognition model testing method provided in this embodiment includes the following steps:

[0090] Step 301: Obtain a test set and a trained speech recognition model.

[0091] In this embodiment, the test set includes at least one test sample, and the test sample includes: an audio feature sequence and a test word unit sequence.

[0092] In this embodiment, the test sample is a sample with the same data structure as the training sample, that is, the test sample includes: an audio feature sequence and a new word unit sequence, the new word unit sequence is a sequence in which the predicted word units representing the language are replaced by the language word units in the initial word unit sequence, and the new word unit sequence includes at least one new word unit, but the test sample has not been used as a training sample to train the speech recognition model.

[0093] In this embodiment, the speech recognition model can be the one described above. Figure 1 The speech recognition model trained by the method described in the embodiment is obtained. The specific training process can be found in Figure 3 The relevant description of the embodiment will not be repeated here.

[0094] In this embodiment, the speech recognition model includes: an encoder, a decoder, and a keyword module, wherein the encoder and the decoder constitute a trained recognition sub-model.

[0095] Step 302: Select test samples from the test set.

[0096] In this embodiment, the execution entity may select a test sample from the test set in step 301.

[0097] In this embodiment, the method and number of test samples selected from the test set are not limited in this application, and the number of iterative training of the speech recognition model is not limited.

[0098] Step 303: Input the audio feature sequence in the test sample into the encoder to obtain audio intermediate features.

[0099] In this embodiment, Figure 4 As shown, the audio feature sequence can be the audio Mel feature of the extracted audio, and the audio Mel feature is input into the encoder of the trained recognition sub-model to obtain the encoder to generate the audio intermediate feature; when the decoder predicts for the first time, the generated starting token sequence ( Figure 4(not shown in the figure) is used as the current word unit sequence. As the test time goes by, after multiple predictions, the current word unit sequence is replaced by the current word unit sequence updated based on the predicted word units.

[0100] Step 304: Input the audio intermediate features and the current word unit sequence into the decoder to obtain the predicted word unit output by the speech recognition model.

[0101] In this embodiment, when the decoder makes its first prediction, the intermediate audio features and the start token sequence are fed into the decoder for decoding to obtain a predicted token output by the decoder. It should be noted that among all the predicted word units output by the decoder, one predicted word unit may be a predicted word unit representing a language.

[0102] Step 305: In response to the predicted word unit being a non-terminal symbol, the current word unit sequence is updated based on the predicted word unit, and the audio intermediate features and the current word unit sequence continue to be input into the decoder until the predicted word unit output by the speech recognition model is a terminal symbol, thereby obtaining all predicted word units corresponding to the test sample.

[0103] like Figure 4 As shown, EOT represents the end symbol. When the predicted word unit is not an end symbol, that is, Not EOT, the current word unit sequence is updated based on the decoder's predicted word unit. Specifically, updating the current word unit sequence can be adding the predicted token (predicted word unit) to the end of the current token sequence. It should be noted that when the decoder predicts for the first time, the predicted word unit output by the decoder is added to the end of the starting token sequence as a result of updating the current word unit sequence. This result is used as the current word unit sequence, and then the decoder makes the next prediction. Figure 4 As shown, when the predicted word unit is an end symbol, the prediction of the audio feature sequence ends.

[0104] In this embodiment, after the decoder performs multiple predictions, if the predicted word unit output by the decoder is EOT, the decoder ends the prediction and combines the predicted word units output by the decoder from the first output to before the output of EOT to obtain all predicted word units.

[0105] Step 306 : Based on all the predicted word units corresponding to the test sample and the test word unit sequence in the test sample, it is detected whether the speech recognition model has passed the test.

[0106] In this embodiment, all predicted word units can be arranged together according to the prediction order of the decoder to obtain a predicted word unit sequence, and the predicted word unit sequence can be compared with the test word unit sequence in the test sample; in response to the predicted word unit sequence being the same as the test word unit sequence, it is determined that the speech recognition model has passed the test, and the speech recognition model can be directly applied to the prediction of the language text; in response to the predicted word unit sequence being different from the test word unit sequence, it is determined that the speech recognition model has failed the test, and the speech recognition model cannot be directly applied to the prediction of the language text.

[0107] The speech recognition model testing method provided by the embodiments of the present disclosure obtains a test set and a speech recognition model, selects a test sample from the test set, inputs the audio feature sequence in the test sample into an encoder, and continuously replaces it with an update unit sequence when the speech recognition model does not output a terminator. As a result, the speech recognition model continuously predicts the intermediate audio features of the test sample to obtain a predicted word unit sequence for the intermediate audio features. By using the predicted word unit sequence to detect whether the speech recognition model has passed the test, the reliability of the speech recognition model test is improved.

[0108] In some optional implementations of the present disclosure, detecting whether the speech recognition model has passed the test based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample includes: sorting all predicted word units corresponding to the test sample to obtain a predicted word unit sequence; calculating the word error rate of the test sample based on the predicted word unit sequence and the test word unit sequence in the test sample; and determining that the speech recognition model is qualified in response to the word error rate being less than an error threshold.

[0109] In this optional implementation, the predicted word unit sequence is compared with the test word unit sequence, the number of word units in the predicted word unit sequence that are different from the test word unit sequence is determined, and the ratio of the number of word units to the total number of test word units in the test word unit sequence is calculated to obtain the word error rate.

[0110] In this optional implementation, the error threshold can be determined based on the test requirements. For example, the error threshold is 20%, that is, less than 20% of the predicted word units in the predicted word unit sequence are different from the test word units in the test word unit sequence, and the speech recognition model is determined to be qualified.

[0111] This optional implementation provides a method for detecting whether a speech recognition model has passed the test. By calculating the word error rate of the test sample through the predicted word unit sequence and the test word unit sequence corresponding to the test sample, it provides a reliable implementation method for testing the speech recognition model and improves the reliability of the speech recognition model test.

[0112] Furthermore, based on the speech recognition model training method provided in the above embodiment, the present disclosure also provides an embodiment of a speech recognition method. The speech recognition method of the present disclosure combines artificial intelligence fields such as speech recognition and deep learning.

[0113] See also Figure 5 , shows a process 500 according to an embodiment of the speech recognition method of the present disclosure. The speech recognition method provided in this embodiment includes the following steps:

[0114] Step 501: Acquire speech to be recognized.

[0115] In this embodiment, the execution entity of the speech recognition method can obtain the speech to be recognized in various ways. For example, the execution entity can obtain the speech to be recognized stored in a database server via a wired or wireless connection. For another example, the execution entity can also receive the speech to be recognized collected in real time by a terminal or other device.

[0116] In this embodiment, the speech to be recognized is speech information that needs to be converted from speech to text. The speech to be recognized may be speech in a scarce language, for example, the speech to be recognized is speech in a language with few samples.

[0117] Step 502: Process the speech to be recognized to obtain audio feature data.

[0118] In this embodiment, the execution entity may process the to-be-recognized speech obtained in step 501 to obtain audio feature data.

[0119] In this embodiment, the audio feature data is data obtained after performing feature recognition (e.g., Mel features or Fbank features) on the speech to be recognized. Specifically, step 502 includes: fixing the length of the speech to be recognized (e.g., 30 seconds), extracting Mel features or Fbank features of the speech to be recognized at that length, and obtaining audio feature data of the fixed length.

[0120] In this embodiment, the audio feature data has the same data structure as the audio feature sequence.

[0121] Step 503: Input the audio feature data into the speech recognition model to obtain a predicted word unit sequence of the speech to be recognized.

[0122] In this embodiment, the speech recognition model can be the one described above. Figure 1 The speech recognition model trained by the method described in the embodiment is obtained. The specific training process can be found in Figure 1 The relevant description of the embodiment will not be repeated here.

[0123] Step 504: Obtain text data of the speech to be recognized based on the predicted word unit sequence.

[0124] In this embodiment, restoring the single sequence to text data is a mature technical means, and converting the predicted word unit sequence into text data of the speech to be recognized is also a mature means, which will not be repeated here.

[0125] The speech recognition method provided in this disclosure can be applied in social, learning, and work scenarios for languages ​​with limited sample sizes. First, it can be applied in regions where the language is the primary language, helping people interact with smart devices more conveniently. Second, the model can be used in education to assist students in learning the pronunciation and listening skills of the language. Furthermore, in the business world, it can be used for customer service and market research to meet the needs of users of the language.

[0126] The speech recognition method provided by the embodiments of the present disclosure obtains a speech to be recognized, processes the speech to be recognized, obtains audio feature data, inputs the audio feature data into a speech recognition model generated by a speech recognition model training method, obtains a predicted word unit sequence of the audio feature data, and obtains text data of the speech to be recognized based on the predicted word unit sequence. Thus, the speech recognition model is used to generate speech recognition results, improving the reliability and accuracy of speech recognition.

[0127] Further references Figure 6 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a speech recognition model training device, which is similar to Figure 1 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0128] like Figure 6 As shown, the speech recognition model training device 600 provided in this embodiment includes: a sample acquisition unit 601, a model acquisition unit 602, a replacement unit 603, and a training unit 604. The sample acquisition unit 601 can be configured to acquire a speech sample set, which includes at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence. The model acquisition unit 602 can be configured to acquire an initial speech recognition model, which is used to characterize the correspondence between the audio feature sequence and the predicted word unit sequence. The replacement unit 603 can be configured to replace the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language to obtain a training sample set, where the predicted word units are predicted word units in the predicted word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model. The training unit 604 can be configured to train the speech recognition model based on the training sample set to obtain a trained speech recognition model.

[0129] In this embodiment, the specific processing of the sample acquisition unit 601, the model acquisition unit 602, the replacement unit 603, and the training unit 604 and the technical effects thereof can be referred to in detail. Figure 1 The relevant descriptions of step 101, step 102, step 103, and step 104 in the corresponding embodiment are not repeated here.

[0130] In some optional implementations of this embodiment, the above-mentioned speech recognition model includes: an initial recognition sub-model, and a keyword module connected to the initial recognition sub-model; the above-mentioned training unit 604 is configured to: train the initial recognition sub-model based on the training sample set to obtain a trained recognition sub-model; train the keyword module based on the training sample set and the trained recognition sub-model to obtain a trained keyword module; and use the trained recognition sub-model and the trained keyword module as the trained speech recognition model.

[0131] In some optional implementations of this embodiment, the above-mentioned training unit 604 is further configured to: fix the parameters of the trained recognition sub-model; input the training samples in the training sample set into the trained recognition sub-model to obtain the first probability distribution output by the trained recognition sub-model and the second probability distribution output by the keyword module; based on the first probability distribution and the second probability distribution, calculate the loss value of the initial word unit in the training sample; based on the loss value, obtain the trained keyword module.

[0132] In some optional implementations of this embodiment, the above-mentioned training unit 604 is further configured to: select a training sample from a training sample set to obtain a selected sample; shift the new word unit in the new word unit sequence in the selected sample right by one position to obtain a training word unit sequence; train an initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected sample to obtain a trained recognition sub-model.

[0133] In some optional implementations of this embodiment, the above-mentioned sample acquisition unit 601 is configured to: acquire an initial data set including at least one initial speech; preprocess the initial data set to obtain a processed data set; perform data enhancement on the processed data set to obtain a speech data set; and obtain a speech sample set based on the speech data set.

[0134] In some optional implementations of this embodiment, the sample acquisition unit 601 is further configured to implement at least one of the following: sampling all initial data in the initial data set at the same sampling rate; deleting duplicate and unrecognizable initial data in the initial data set.

[0135] In some optional implementations of this embodiment, the above-mentioned sample acquisition unit 601 is further configured to implement at least one of the following: modifying the speaking rate of each processed data in the processed data set to obtain an added data set, and adding the added data set to the processed data set; adding reverberation to each processed data in the processed data set to obtain an echo data set, and adding an echo data set to the processed data set; adding noise to each processed data in the processed data set to obtain a noise data set, and adding a noise data set to the processed data set; modifying the audio spectrum graph of each processed data in the processed data set to obtain a spectrum data set, and adding a spectrum data set to the processed data set.

[0136] The speech recognition model training device provided by the embodiment of the present disclosure includes: first, a sample acquisition unit 601 acquires a speech sample set, the speech sample set including at least one speech sample, the speech sample including: an audio feature sequence and an initial word unit sequence; second, a model acquisition unit 602 acquires an initial speech recognition model, the speech recognition model is used to characterize the correspondence between the audio feature sequence and the predicted word unit sequence; third, a replacement unit 603 replaces the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language, thereby obtaining a training sample set, wherein the predicted word units are predicted word units in the predicted word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model; finally, a training unit 604 trains the speech recognition model based on the training sample set to obtain a trained speech recognition model. Therefore, before training the speech recognition model, replacing the language word units in the initial word unit sequence in the speech sample set with predicted word units representing the language can enable the model to predict the audio feature sequence in its familiar word unit environment, thereby improving the model convergence speed and the model training effect.

[0137] Further references Figure 7 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a speech recognition model testing device, which is similar to Figure 3 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0138] like Figure 7As shown, the speech recognition model testing device 700 provided in this embodiment includes: an information acquisition unit 701, a selection unit 702, an input unit 703, an acquisition unit 704, an update unit 705, and a test unit 706. The information acquisition unit 701 can be configured to acquire a test set and a trained speech recognition model, wherein the test set includes at least one test sample, and the test sample includes: an audio feature sequence and a test word unit sequence; the trained speech recognition model is trained using the speech recognition model training device, and the speech recognition model includes: an encoder, a decoder, and a keyword module. The selection unit 702 can be configured to select a test sample from the test set. The input unit 703 can be configured to input the audio feature sequence in the test sample into the encoder to obtain an audio intermediate feature. The acquisition unit 704 can be configured to input the audio intermediate feature and the current word unit sequence into the decoder to obtain the predicted word unit output by the speech recognition model. The updating unit 705 can be configured to, in response to the predicted word unit being a non-terminal symbol, update the current word unit sequence based on the predicted word unit, continue inputting the intermediate audio features and the current word unit sequence into the decoder until the predicted word unit output by the speech recognition model is a terminal symbol, and obtain all predicted word units corresponding to the test sample. The testing unit 706 can be configured to detect whether the speech recognition model has passed the test based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample.

[0139] In this embodiment, the specific processing of the information acquisition unit 701, the selection unit 702, the input unit 703, the obtaining unit 704, the updating unit 705, and the testing unit 706 and the technical effects thereof can be referred to respectively. Figure 3 The relevant descriptions of step 301, step 302, step 304, step 305, and step 306 in the corresponding embodiment are not repeated here.

[0140] In some optional implementations of this embodiment, the above-mentioned test unit 706 is further configured to: sort all predicted word units corresponding to the test sample to obtain a predicted word unit sequence; calculate the word error rate of the test sample based on the predicted word unit sequence and the test word unit sequence in the test sample; and determine that the speech recognition model is qualified in response to the word error rate being less than the error threshold.

[0141] Further references Figure 8 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a speech recognition device, which is similar to Figure 5 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0142] like Figure 8As shown, the speech recognition device 800 provided in this embodiment includes: a speech acquisition unit 801, a processing unit 802, a recognition unit 803, and a conversion unit 804. The speech acquisition unit 801 can be configured to acquire the speech to be recognized. The processing unit 802 can be configured to process the speech to be recognized to obtain audio feature data. The recognition unit 803 can be configured to input the audio feature data into a speech recognition model to obtain a predicted word unit sequence of the speech to be recognized. The speech recognition model is the trained speech recognition model obtained by the speech recognition model training device of the above embodiment. The conversion unit 804 can be configured to obtain text data of the speech to be recognized based on the predicted word unit sequence.

[0143] In this embodiment, the speech recognition device 800 includes the speech acquisition unit 801, the processing unit 802, the recognition unit 803, and the conversion unit 804. The specific processing and technical effects thereof can be referred to in the respective embodiments. Figure 5 The relevant descriptions of step 501, step 502, step 503, and step 504 in the corresponding embodiment are not repeated here.

[0144] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0145] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0146] Figure 9 A schematic block diagram of an example electronic device 900 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0147] like Figure 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 into a random access memory (RAM) 903. Various programs and data required for the operation of the device 900 can also be stored in the RAM 903. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0148] Various components in the device 900 are connected to the I / O interface 905, including an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a magnetic disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0149] The computing unit 901 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as a speech recognition model training method or a speech recognition model testing method or a speech recognition method. For example, in some embodiments, the speech recognition model training method or the speech recognition model testing method or the speech recognition method can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the speech recognition model training method or the speech recognition model testing method or the speech recognition method described above can be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the speech recognition model training method or the speech recognition model testing method or the speech recognition method in any other appropriate manner (for example, by means of firmware).

[0150] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0151] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable speech recognition model training device, a speech recognition model testing device, or a speech recognition device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0152] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0154] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0155] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0156] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0157] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A speech recognition model training method, the method comprising: Acquire a speech sample set, wherein the speech sample set includes at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence; Obtaining an initial speech recognition model, the speech recognition model being used to characterize the correspondence between an audio feature sequence and a predicted word unit sequence, the speech recognition model comprising: an initial recognition sub-model and a keyword module connected to the initial recognition sub-model; Replacing language word units in an initial word unit sequence in the speech sample set with predicted word units representing a language to obtain a training sample set, wherein the predicted word units are predicted word units in a predicted word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model; Based on the training sample set, training the initial recognition sub-model to obtain a trained recognition sub-model; Fixing the parameters of the trained recognition sub-model; Inputting the training samples in the training sample set into the trained recognition sub-model to obtain a first probability distribution output by the trained recognition sub-model and a second probability distribution output by the keyword module; Calculating a loss value of an initial word unit in a training sample based on the first probability distribution and the second probability distribution; Based on the loss value, obtaining a trained keyword module; The trained recognition sub-model and the trained keyword module are used as a trained speech recognition model.

2. The method according to claim 1, wherein The initial recognition sub-model is trained based on the training sample set to obtain the trained recognition sub-model, including: Selecting a training sample from the training sample set to obtain a selected sample; Shifting the new word unit in the new word unit sequence in the selected sample right by one position to obtain a training word unit sequence; Based on the audio feature sequence and the training word unit sequence in the selected sample, the initial recognition sub-model is trained to obtain a trained recognition sub-model.

3. The method according to claim 1, wherein The acquiring of the speech sample set comprises: Acquire an initial data set including at least one initial speech; Preprocessing the initial data set to obtain a processed data set; Performing data enhancement on the processed data set to obtain a speech data set; Based on the speech data set, a speech sample set is obtained.

4. The method according to claim 3, wherein: The preprocessing of the initial data set to obtain a processed data set includes at least one of the following: Sampling all the initial data in the initial data set at the same sampling rate; Duplicate and unrecognizable initial data in the initial dataset are deleted.

5. The method according to claim 3, wherein The performing data enhancement on the processed data set to obtain a speech data set includes at least one of the following: modifying the speech rate of each processed data in the processed data set to obtain an added data set, and adding the added data set to the processed data set; adding an echo to each processed data in the processed data set to obtain an echo data set, and adding the echo data set to the processed data set; adding noise to each processed data in the processed data set to obtain a noise data set, and adding the noise data set to the processed data set; The audio spectrogram of each processed data in the processed data set is modified to obtain a spectrum data set, and the spectrum data set is added to the processed data set.

6. A method for testing a speech recognition model, the method comprising: Obtaining a test set and a trained speech recognition model, wherein the test set includes at least one test sample, and the test sample includes: an audio feature sequence and a test word unit sequence; the trained speech recognition model is trained using the speech recognition model training method according to any one of claims 1 to 5, and the speech recognition model includes: an encoder, a decoder, and a keyword module; Selecting a test sample from the test set; Inputting the audio feature sequence in the test sample into the encoder to obtain audio intermediate features; Inputting the audio intermediate features and the current word unit sequence into the decoder to obtain the predicted word unit output by the speech recognition model; In response to the predicted word unit being a non-terminal symbol, updating the current word unit sequence based on the predicted word unit, continuing to input the audio intermediate features and the current word unit sequence into the decoder until the predicted word unit output by the speech recognition model is a terminal symbol, thereby obtaining all predicted word units corresponding to the test sample; Based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample, it is detected whether the speech recognition model has passed the test.

7. The method according to claim 6, wherein: The detecting whether the speech recognition model is qualified based on all predicted word units corresponding to the test sample and a test word unit sequence in the test sample includes: Sorting all predicted word units corresponding to the test sample to obtain a predicted word unit sequence; Calculating a word error rate of the test sample based on the predicted word unit sequence and the test word unit sequence in the test sample; In response to the word error rate being less than an error threshold, it is determined that the speech recognition model is qualified.

8. A speech recognition method, comprising: Get the speech to be recognized; Processing the speech to be recognized to obtain audio feature data; Inputting the audio feature data into a speech recognition model to obtain a predicted word unit sequence of the speech to be recognized, wherein the speech recognition model is a trained speech recognition model obtained by the speech recognition model training method according to any one of claims 1 to 5; Based on the predicted word unit sequence, text data of the speech to be recognized is obtained.

9. A speech recognition model training device, comprising: A sample acquisition unit is configured to acquire a speech sample set, wherein the speech sample set includes at least one speech sample, and the speech sample includes: an audio feature sequence and an initial word unit sequence; a model acquisition unit configured to acquire an initial speech recognition model, the speech recognition model being used to characterize the correspondence between an audio feature sequence and a predicted word unit sequence, the speech recognition model comprising: an initial recognition sub-model and a keyword module connected to the initial recognition sub-model; a replacement unit configured to replace language word units in an initial word unit sequence in the speech sample set with predicted word units representing a language, to obtain a training sample set, wherein the predicted word units are predicted word units in a predicted word unit sequence obtained by inputting speech samples selected from the speech sample set into the speech recognition model; The training unit is configured to train the initial recognition sub-model based on the training sample set to obtain a trained recognition sub-model; fix the parameters of the trained recognition sub-model; input the training samples in the training sample set into the trained recognition sub-model to obtain a first probability distribution output by the trained recognition sub-model and a second probability distribution output by the keyword module; calculate the loss value of the initial word unit in the training sample based on the first probability distribution and the second probability distribution; obtain the trained keyword module based on the loss value; and use the trained recognition sub-model and the trained keyword module as the trained speech recognition model.

10. The device according to claim 9, wherein The training unit is further configured to: select training samples from the training sample set to obtain selected samples; shift the new word units in the new word unit sequence in the selected samples to the right by one position to obtain a training word unit sequence; train the initial recognition sub-model based on the audio feature sequence and the training word unit sequence in the selected samples to obtain a trained recognition sub-model.

11. The device according to claim 9, wherein The sample acquisition unit is configured to: acquire an initial data set including at least one initial speech; Preprocessing the initial data set to obtain a processed data set; performing data enhancement on the processed data set to obtain a speech data set; Based on the speech data set, a speech sample set is obtained.

12. The device according to claim 11, wherein The sample acquisition unit is further configured to implement at least one of the following: Sampling all the initial data in the initial data set at the same sampling rate; Duplicate and unrecognizable initial data in the initial dataset are deleted.

13. The device according to claim 11, wherein The sample acquisition unit is further configured to implement at least one of the following: modifying the speech rate of each processed data in the processed data set to obtain an added data set, and adding the added data set to the processed data set; adding an echo to each processed data in the processed data set to obtain an echo data set, and adding the echo data set to the processed data set; adding noise to each processed data in the processed data set to obtain a noise data set, and adding the noise data set to the processed data set; The audio spectrogram of each processed data in the processed data set is modified to obtain a spectrum data set, and the spectrum data set is added to the processed data set.

14. A speech recognition model testing device, comprising: An information acquisition unit is configured to acquire a test set and a trained speech recognition model, wherein the test set includes at least one test sample, and the test sample includes: an audio feature sequence and a test word unit sequence; the trained speech recognition model is trained using the speech recognition model training device according to any one of claims 9 to 13, and the speech recognition model includes: an encoder, a decoder, and a keyword module; A selection unit, configured to select a test sample from the test set; an input unit configured to input the audio feature sequence in the test sample into the encoder to obtain audio intermediate features; an obtaining unit configured to input the intermediate audio features and the current word unit sequence into the decoder to obtain the predicted word unit output by the speech recognition model; an updating unit configured to, in response to the predicted word unit being a non-terminal symbol, update a current word unit sequence based on the predicted word unit, continue inputting the audio intermediate features and the current word unit sequence into the decoder until the predicted word unit output by the speech recognition model is a terminal symbol, and obtain all predicted word units corresponding to the test sample; The testing unit is configured to detect whether the speech recognition model passes the test based on all predicted word units corresponding to the test sample and the test word unit sequence in the test sample.

15. The device according to claim 14, wherein The test unit is further configured to: sort all predicted word units corresponding to the test sample to obtain a predicted word unit sequence; calculate the word error rate of the test sample based on the predicted word unit sequence and the test word unit sequence in the test sample; and determine that the speech recognition model is qualified in response to the word error rate being less than an error threshold.

16. A speech recognition device, comprising: A speech acquisition unit, configured to acquire speech to be recognized; a processing unit configured to process the speech to be recognized to obtain audio feature data; a recognition unit configured to input the audio feature data into a speech recognition model to obtain a predicted word unit sequence of the speech to be recognized, wherein the speech recognition model is a trained speech recognition model obtained by using the speech recognition model training device according to any one of claims 9 to 13; The conversion unit is configured to obtain text data of the speech to be recognized based on the predicted word unit sequence.

17. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 8.

19. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 8.

Images