A speech recognition method, a training method, device and equipment of an acoustic model

Abstract

The application provides a speech recognition method, a training method and device of an acoustic model, and equipment. The speech recognition method comprises: receiving a speech recognition request, the speech recognition request comprising speech to be recognized in a target language; extracting speech features from the speech to be recognized and inputting the speech features into a first acoustic model to obtain acoustic features, the acoustic features comprising possible characters of the speech to be recognized, the first acoustic model being a model for recognizing possible characters in the target language in speech data, the model being obtained by training a neural network based on first speech samples and second speech samples, the first speech samples comprising a plurality of speech samples artificially labeled and characters in the target language, and the second speech samples comprising a plurality of speech samples labeled by a second acoustic model trained by using the first speech samples and characters in the target language; and determining a speech recognition result of the target language according to the acoustic features and outputting the speech recognition result. The method improves the speech recognition accuracy of the target language.

Description

Technical Field

[0001] This application relates to speech processing technology, and more particularly to a speech recognition method, an acoustic model training method, apparatus, and device. Background Technology

[0002] Speech recognition is a technology that enables machines to convert speech signals into corresponding text or commands through recognition and understanding. Currently, speech recognition technology has made significant progress, achieving good results in some major languages ​​such as Chinese, English, and French.

[0003] Currently, one method of speech recognition involves training a neural network with manually calibrated speech samples and corresponding language characters to obtain an acoustic model, which is then used to perform speech recognition on the speech to be recognized.

[0004] However, in reality, the number of speech samples for some less common languages, such as Thai, is relatively small, which results in lower accuracy of acoustic models and consequently lower accuracy of speech recognition. Summary of the Invention

[0005] This application provides a speech recognition method, an acoustic model training method, an apparatus, and a device to address the problem that the limited number of speech samples for characters in some minority languages ​​results in low recognition accuracy of the acoustic model, thus leading to low speech recognition accuracy.

[0006] In a first aspect, this application provides a speech recognition method, comprising: receiving a speech recognition request, the speech recognition request including speech to be recognized in a target language; extracting features from the speech to be recognized to obtain speech features; inputting the speech features into a first acoustic model to obtain acoustic features, the acoustic features including possible characters of the speech to be recognized, the first acoustic model being a model trained on a neural network based on a first speech sample and a second speech sample for recognizing possible characters of the target language in speech data, the first speech sample including multiple manually labeled speech samples and corresponding characters of the target language, the second speech sample including multiple speech samples labeled by a second acoustic model trained using the first speech samples and corresponding characters of the target language; determining a speech recognition result of the target language speech based on the acoustic features; and outputting the speech recognition result of the target language speech.

[0007] Secondly, this application provides a method for training an acoustic model, comprising: acquiring a first speech sample and a second speech sample, wherein the first speech sample includes multiple manually calibrated speech samples and corresponding characters of a target language, and the second speech sample includes multiple speech samples calibrated by a second acoustic model trained using the first speech sample and corresponding characters of the target language; and training a neural network based on the first speech sample and the second speech sample to obtain a first acoustic model.

[0008] Thirdly, this application provides a speech recognition device, comprising: a receiving module for receiving a speech recognition request, the speech recognition request including speech to be recognized in a target language; a feature extraction module for extracting features from the speech to be recognized to obtain speech features; an input module for inputting the speech features into a first acoustic model to obtain acoustic features, the acoustic features including possible characters of the speech to be recognized, the first acoustic model being a model trained on a neural network based on a first speech sample and a second speech sample for recognizing possible characters of the target language in speech data, the first speech sample including multiple manually calibrated speech samples and corresponding characters of the target language, the second speech sample including multiple speech samples calibrated by a second acoustic model trained using the first speech samples and corresponding characters of the target language; a determining module for determining the speech recognition result of the target language speech based on the acoustic features; and an output module for outputting the speech recognition result of the target language speech.

[0009] Fourthly, this application provides a training apparatus for an acoustic model, comprising: an acquisition module for acquiring a first speech sample and a second speech sample, wherein the first speech sample includes multiple manually calibrated speech samples and corresponding characters of a target language, and the second speech sample includes multiple speech samples calibrated by a second acoustic model trained using the first speech sample and corresponding characters of the target language; and a training module for training a neural network based on the first speech sample and the second speech sample to obtain a first acoustic model.

[0010] Fifthly, this application provides an electronic device, including: a memory, a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the method as described in the first aspect.

[0011] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in the first aspect.

[0012] In a seventh aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0013] The speech recognition method, acoustic model training method, apparatus, and device provided in this application receive a speech recognition request, which includes speech data in a target language to be recognized. The method extracts speech features from the speech data to be recognized, and inputs these speech features into a first acoustic model to obtain acoustic features. The acoustic features include possible characters from the speech data to be recognized. The first acoustic model is a model trained using a neural network based on first and second speech samples to recognize possible characters in the target language from the speech data. The first speech sample includes multiple manually labeled speech samples and corresponding characters from the target language. The second speech sample includes multiple speech samples labeled using a second acoustic model trained on the first speech samples and corresponding characters from the target language. Based on the acoustic features, the method determines the speech recognition result for the target language speech and outputs the speech recognition result. Since the second speech sample includes multiple speech samples labeled by the second acoustic model trained with the first speech sample and the corresponding characters of the target language, it is possible to obtain the characters of the target language corresponding to the unlabeled speech samples, train the model based on the unlabeled speech samples, thereby improving the accuracy of deep learning models in character recognition for low-resource languages ​​such as minority languages, and thus improving the accuracy of speech recognition. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0015] Figure 1 Application scenario diagrams provided for embodiments of this application;

[0016] Figure 2 An interaction diagram of the speech recognition method provided in the embodiments of this application;

[0017] Figure 3 A logic diagram of speech recognition provided in the embodiments of this application;

[0018] Figure 4 This is a schematic diagram of the neural network structure provided in an embodiment of this application;

[0019] Figure 5 The flow of the speech recognition method provided in the embodiments of this application Figure 1 ;

[0020] Figure 6 The flow of the speech recognition method provided in the embodiments of this application Figure 2 ;

[0021] Figure 7 A schematic diagram of the tree structure provided in the embodiments of this application;

[0022] Figure 8 The flowchart of the training method for the acoustic model provided in the embodiments of this application Figure 1 ;

[0023] Figure 9 A logic diagram illustrating the training process of the neural network provided in the embodiments of this application;

[0024] Figure 10 The flowchart of the training method for the acoustic model provided in the embodiments of this application Figure 2 ;

[0025] Figure 11 The flowchart of the training method for the acoustic model provided in the embodiments of this application Figure 3 ;

[0026] Figure 12 A logic diagram illustrating the training process of the first acoustic model provided in this application embodiment;

[0027] Figure 13 This is a schematic diagram of the structure of the speech recognition device provided in the embodiments of this application;

[0028] Figure 14 A schematic diagram of the structure of the training device for the acoustic model provided in the embodiments of this application;

[0029] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0030] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0031] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0032] Figure 1 This is an application scenario diagram provided for embodiments of this application. For example... Figure 1As shown, the application scenario includes terminal device 1 and server 2; wherein, terminal device 1 includes devices such as smartphones, tablets, and personal computers.

[0033] Server 2 can train a speech recognition model based on speech samples and the manually labeled characters corresponding to the speech samples. This speech recognition model can be stored on server 2 or on terminal device 1.

[0034] In one optional implementation, when terminal device 1 receives a voice recognition request, the voice recognition request includes the voice to be recognized. The voice recognition request is sent to server 2. Server 2 calls the voice recognition model stored in server 2 to perform voice recognition on the voice to be recognized, obtains the voice recognition result, and returns it to terminal device.

[0035] In another optional implementation, when terminal device 1 receives a voice recognition request, terminal device 1 calls the voice recognition model stored in terminal device 1 to perform voice recognition on the voice to be recognized, obtains the voice recognition result, and displays the voice recognition result.

[0036] The training process of a speech recognition model requires a large number of labeled speech samples. The labels represent the text content corresponding to the speech samples. The neural network is trained using the speech samples and labels. The parameters of the neural network are adjusted based on the differences between the training results and the labels. The training process is repeated until the training is completed. The resulting speech recognition model can then be used for speech recognition.

[0037] However, there are nearly 7,000 languages ​​in the world, the vast majority of which are less commonly spoken, and fewer than 10 are spoken by more than 100 million people. Currently, speech recognition technology has made significant progress, achieving good results in some major languages ​​such as Chinese, English, and French. However, for less commonly spoken languages, the limited availability of speech data resources, the difficulty in obtaining annotations, and the high cost result in insufficient training data, leading to poor performance. Thai, as a less commonly spoken language, also faces the problem of not being able to obtain large-scale annotated corpora.

[0038] Currently, speech recognition frameworks are mainly divided into two types: one is a hybrid model framework based on Gaussian Mixed Model (GMM) and Deep Neural Networks (DNN), and the other is an end-to-end model approach. While the GMM-DNN hybrid model is more advantageous for speech recognition tasks with small datasets, it suffers from complex training processes and slow decoding speeds. The end-to-end model approach, on the other hand, eliminates the need to build an expert dictionary, avoids the complex alignment operations found in hybrid model frameworks, has a relatively simple training process, and offers fast decoding speeds for connectionist temporal classification (CTC) end-to-end models, making it beneficial for practical applications. Therefore, this patent will adopt the latter—the CTC-based end-to-end model framework.

[0039] For end-to-end speech recognition models, we use deep network models to model speech; however, for deep network models, the amount of data is a key factor determining the model's learning performance. Therefore, since it is difficult to obtain labeled speech data for less commonly spoken languages, we usually cannot obtain large-scale labeled corpora. Using a small amount of labeled data to supervise the model's training may result in the model learning limited "knowledge," leading to unsatisfactory recognition performance and rendering the model unusable.

[0040] To address the aforementioned technical problems, the inventors of this application propose the following technical concept: With the widespread use of the internet, people all over the world use it, and a vast amount of voice data exists online, including voice data in less commonly spoken languages. This data can be obtained through web scraping. If this data can be used to train an acoustic model, the difficulty of speech recognition in less commonly spoken languages ​​can be solved. The main difficulty in speech recognition of less commonly spoken languages ​​lies in the scarcity of voice data. However, in many cases, the scarcity is not of the voice data itself, but rather of the text annotations. Obtaining unlabeled data is often much easier, such as through web scraping. If a large amount of unlabeled data can be used to train a model, the difficulty of speech recognition in less commonly spoken languages ​​can be solved.

[0041] Speech recognition is used in many scenarios, such as in social media apps where users can input a voice message and have it converted into text; in search engines, voice assistants allow users to search by voice input; users can control objects with their voice, such as home appliances; and in navigation systems, voice assistants and voice-based meeting minutes.

[0042] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0043] based on Figure 1 Application scenarios, Figure 2 An interaction diagram illustrating the speech recognition method provided in an embodiment of this application. For example... Figure 2 As shown, the speech recognition method includes:

[0044] S201. The terminal device sends a speech recognition request to the server, which includes the speech to be recognized in the target language.

[0045] S202, The server receives a speech recognition request.

[0046] S203. The server extracts speech features from the speech to be recognized to obtain speech features.

[0047] Figure 3 A logic diagram for speech recognition in the target language provided in the embodiments of this application. (See diagram below.) Figure 3 As shown, the server extracts speech features from the speech to be recognized in the target language to obtain speech features.

[0048] Specifically, speech feature extraction is performed on the speech to be recognized to obtain speech features, including: extracting speech features from the speech to be recognized according to a speech feature extraction algorithm to obtain speech features.

[0049] Optional speech feature extraction algorithms include Mel-frequency cepstral coefficients (MFCC), linear prediction coefficients (LPC), linear prediction cepstral coefficients (LPCC), line spectral frequency (LSF), discrete wavelet transform (DWT), and perceptual linear prediction (PLP), etc.

[0050] Optionally, before step S203, the speech to be recognized needs to be segmented into frames to obtain multiple speech frames, and features need to be extracted for each speech frame to obtain speech frame features. These speech frame features are the feature extraction results of a single speech frame.

[0051] Optionally, before performing frame segmentation on the speech to be recognized, it is necessary to preprocess the speech to be recognized to obtain preprocessed speech, and then extract speech features from the preprocessed speech as speech features.

[0052] Specifically, preprocess the speech to be recognized, including: removing the silence at the head and tail of the speech to be recognized to obtain the target speech. After obtaining the target speech, the target speech can be framed to obtain multiple speech frames; and for each speech frame in the multiple speech frames, extract the frequency-domain features to obtain the speech frame features.

[0053] Taking the speech feature extraction algorithm MFCC as an example, extract the frequency-domain features of the speech frame, including: performing a fast Fourier transform on the speech frame; performing Mel filtering on the speech frame after Fourier transform; performing a logarithm calculation on the speech frame after Mel filtering; determining the feature data of a preset dimension in the speech frame after taking the logarithm as the speech frame feature. Optionally, the preset dimension can be any dimension between 40 and 160, for example, 80 dimensions.

[0054] S204. The server inputs the speech features into the first acoustic model to obtain acoustic features, and the acoustic features include the possible characters of the speech to be recognized. The first acoustic model is a model for recognizing the possible characters of the target language in speech data, which is trained based on a neural network according to the first speech sample and the second speech sample. The first speech sample includes multiple manually calibrated speech samples and the corresponding characters of the target language, and the second speech sample includes multiple speech samples calibrated by the second acoustic model trained with the first speech sample and the corresponding characters of the target language.

[0055] Please continue to refer to Figure 3 , the server inputs the speech features into the first acoustic model to obtain an acoustic recognition result, and this acoustic recognition result can be understood as the probability distribution result of the possible characters of the speech to be recognized.

[0056] In this embodiment, the modeling unit of the first acoustic model is at the character level. That is to say, the input of the first acoustic model is the frame-level speech features, that is, the features of one speech segment after another, and the output is the probability of the frame-level possible characters, that is, the possible characters corresponding to each speech segment and the corresponding character probabilities. This character probability is used to represent the probability that the character corresponding to this speech segment is a possible character. For example, the pinyin corresponding to speech segment A is "e" with the fourth tone, then the possible characters corresponding to it include "è", "è", "ě", and the character probability of "è" is 0.5, the probability of "è" is 0.3, and the probability of "ě" is 0.2. The combination of the probabilities of the possible characters corresponding to all speech segments is the probability distribution result of the possible characters of the speech to be recognized.

[0057] S205. The server determines the speech recognition result of the target language according to the acoustic features.

[0058] Please continue to refer to Figure 3 , the server inputs the acoustic features into the decoder and obtains the speech recognition result of the target language.

[0059] Specifically, the decoder determines the speech recognition result of the target language according to the acoustic features and the language model. More specifically, the decoder forms a possible text sequence based on the possible characters of the speech to be recognized, and judges whether the possible text sequence conforms to the language habit according to the language model, and determines the possible text sequence that best conforms to the language habit in the possible text sequences as the speech recognition result of the target language.

[0060] In some examples, the language model can adopt the N-gram statistical language model. N-gram predicts the probability of the Nth item based on the previous N-1 items, and is used to calculate the probability of a sentence composed of any sequence of words. The probability formula of a sentence is shown in the following formula (1). The statistical language model is based on a large-scale corpus data collected artificially in advance, and uses real human language as the standard to predict the probability of a text sequence appearing in the corpus. The size of this probability can be used as the basis for judging whether a sentence is "legal". For example, for the text following "The weather today is really __", the probability of following "good" is relatively high, while the probability of following "hao" is almost zero.

[0061] P(S) = P(W1, W2,..., Wk) = p(W1)p(W2|W1)...p(Wk|Wk-n, Wk-(n-1),..., Wk-1); (1)

[0062] In formula (1), S represents the sentence composed of characters W1, W2,..., Wk, p(W1) represents the probability that the speech frame output by the first acoustic model is character W1, p(W2|W1) represents the probability that character W2 follows character W1, and p(Wk|Wk-n, Wk-(n-1),..., Wk-1) represents the probability that character Wk follows the possible text sequence (Wk-n, Wk-(n-1),..., Wk-1).

[0063] S206. The server returns the speech recognition result of the target language to the terminal device.

[0064] S207. The terminal device receives the speech recognition result of the target language and displays it.

[0065] In some optional scenarios, when user A receives a speech in a target language in a dialogue service, and the user initiates a speech recognition request through some operations in the graphical user interface of the terminal device, such as converting the speech in the target language into text, the server can perform speech recognition through the methods in steps S203 to S205, and return the speech recognition result to the terminal device. The terminal device displays the converted text, or performs semantic understanding of the text based on natural language processing (NLP), and applies the semantic understanding result, such as the voice control and voice assistant described above.

[0066] This embodiment receives a speech recognition request, which includes speech data in the target language to be recognized. Speech features are extracted from the speech data to be recognized to obtain speech features. These speech features are then input into a first acoustic model to obtain acoustic features. The acoustic features include possible characters from the speech data to be recognized. The first acoustic model is a model trained on a neural network based on first and second speech samples to identify possible characters in the target language from the speech data. The first speech sample includes multiple manually labeled speech samples and corresponding characters from the target language. The second speech sample includes multiple speech samples labeled using the second acoustic model trained on the first speech samples and corresponding characters from the target language. Based on the acoustic features, the speech recognition result for the target language speech is determined and output. Since the second speech sample includes multiple speech samples labeled by the second acoustic model trained with the first speech sample and the corresponding characters of the target language, it is possible to obtain the characters of the target language corresponding to the unlabeled speech samples, train the model based on the unlabeled speech samples, thereby improving the accuracy of deep learning models in character recognition for low-resource languages ​​such as minority languages, and thus improving the accuracy of speech recognition.

[0067] Figure 4 This is a schematic diagram of the neural network structure provided in an embodiment of this application. Figure 4 As shown, the neural network includes a convolutional unit 41, a self-attention unit 42, and a fully connected unit 43 connected in sequence. Each convolutional unit includes a predetermined number of convolutional neural networks, such as a 3-layer convolutional neural network. Each self-attention unit includes a predetermined number of self-attention blocks, such as 10 self-attention blocks. Each fully connected unit includes a predetermined number of fully connected layers, such as 2 fully connected layers. Here, CTC loss represents the loss function of the end-to-end model.

[0068] based on Figure 4 The structure of the neural network, Figure 5 The flow of the speech recognition method provided in the embodiments of this application Figure 1 .like Figure 5 As shown, the speech features are input into the first acoustic model to obtain the acoustic features, including:

[0069] S501. Input the speech features into the convolutional unit to obtain the first speech features. The convolutional unit includes a preset number of convolutional neural networks.

[0070] S502. Input the first speech feature into the self-attention unit to obtain the second speech feature. The self-attention unit includes a preset number of self-attention blocks.

[0071] S503. Input the second speech feature into the fully connected unit to obtain the speech feature. The fully connected unit includes a preset number of fully connected layers.

[0072] In this embodiment, the convolutional unit extracts local features based on speech features, and then the self-attention unit continues to extract features based on the results of the local feature extraction. The self-attention unit can extract key local features from the local features extracted by the convolutional unit, and then the fully connected unit combines the key local features. The classifier determines the speech features based on the combination result. In this way, the large amount of computation and slow computation speed caused by the fully connected unit needing to combine all local features can be avoided.

[0073] Figure 6 The flow of the speech recognition method provided in the embodiments of this application Figure 2 .like Figure 3 As shown, the speech recognition method includes the following steps:

[0074] S601. For each speech frame in multiple speech frames, input the speech features corresponding to the speech frame into the first acoustic model to obtain the possible characters of the speech features corresponding to the speech frame. Each possible character has a character probability, which is used to represent the probability that the character of the speech feature corresponding to the speech frame is a possible character.

[0075] S602. Combine the possible characters of multiple speech frame features according to the time order of the speech frames to obtain the character feature matrix.

[0076] Table 1 shows the character feature matrix provided in the embodiments of this application. As shown in Table 1, the character feature matrix is ​​a two-dimensional matrix of M*N, with its two dimensions being the number of time frames and the possible characters, respectively, and each possible character in each time frame has a corresponding character probability.

[0077] Table 1 Character Feature Matrix

[0078] t1 t2 t3 …… tn …… tN A11 A21 A31 …… An1 …… AN1 A12 A22 A32 …… An2 …… AN2 …… …… …… …… …… …… …… A1m A2m A3m …… Anm …… AN4 …… …… …… …… …… …… …… A1M A2M A3M …… AnM …… ANM

[0079] In this table, each column represents a time frame, and each time frame corresponds to a speech frame. Each row in that column represents the possible characters and their probabilities for that speech frame. That is, Anm = ("possible character nm", Pnm), where Pnm represents the probability of the possible character nm.

[0080] For the decoder, the inputs include the character feature matrix output by the language model and the first acoustic model.

[0081] Based on the character feature matrix in Table 1, this embodiment decodes the acoustic recognition results to obtain the speech recognition results, including S603 to S607:

[0082] S603. For the i-th speech frame, determine the target character. The target character is a possible character selected from all possible characters in the speech frame according to a preset selection rule, where i is a positive integer greater than 0.

[0083] In this embodiment, the preset selection rule can be to select a character with a probability greater than a probability threshold as the target character.

[0084] The preset selection rule can also be to sort all possible characters in descending order of their character probabilities and select the top n possible characters whose sum of character probabilities equals the preset probability.

[0085] For example, if the sum of the probabilities of all possible characters in a time frame is 1, and it includes 10 possible characters, then the probabilities of these 10 possible characters are sorted in descending order, and the sum of the probabilities is taken as the probability threshold, such as the first n possible characters with a probability of 0.9.

[0086] S604. Based on the target character and the generated candidate paths, determine new candidate paths. The generated candidate paths are used to represent the possible text sequences of the first to the (i-1)th speech frames.

[0087] S605. Score new candidate paths.

[0088] S606. If the score of the new candidate path is greater than the preset score, then the new candidate path is determined as the possible text sequence from the 1st to the i-th speech frame.

[0089] S607. Repeat the above process until the last speech frame to obtain the speech recognition result of the target language speech.

[0090] The implementation methods of steps S601 to S607 will be described in detail below with reference to Table 1:

[0091] Please refer to Table 1. This implementation method includes the following steps:

[0092] a1. For the first time frame t1, the initial candidate path (i.e. the generated candidate path) is set to an empty string. The vector in the first time frame of the character feature matrix output by the first acoustic model is taken, which is the probability of all possible characters in the first time frame.

[0093] Candidate paths can be understood as text sequences consisting of several characters.

[0094] a2. For each character among all possible characters in the first time frame, determine whether the character meets the preset selection rules.

[0095] a3. If a character meets the preset selection rules, the character is added to the end of the generated candidate path to obtain a new candidate path.

[0096] a4. If a character does not meet the preset selection rules, then the character will be discarded.

[0097] If a character does not meet the preset selection rules, it will not participate in the formation of a new candidate path.

[0098] a5. Determine the score of the new candidate path, take the top-ranked number of new candidate paths from the score ranking results, determine them as the initial candidate paths for the next time frame, and execute the method steps in steps a1 to a5 for the next time frame until the last time frame.

[0099] Optionally, the preset quantity can be 200.

[0100] Specifically, the score of a new candidate path can be determined based on the language model and the word insertion penalty term. Determining the score of a new candidate path based on the language model and the word insertion penalty term includes: determining the score of the new candidate path based on the probability of the possible characters in the speech frame, the score of the language model for the possible characters in the speech frame, and the path length corresponding to the possible characters in the speech frame.

[0101] The following formula (2) can be used to score the new candidate path:

[0102] Score = pnet(W; X) * plm(W) α *|leng(W)| β (2)

[0103] In Equation (2), net represents the first acoustic model, X represents the speech features of the input acoustic model, W represents the transcribed text, and p represents the probability. The first product term represents the probability of the output W of the first acoustic model when X is input. lm represents the language model, α represents the weight of the language model, and the second product term represents the score given by the language model. leng(W) represents the path length, and β is the weight of the word insertion penalty term. The third product term is the score of the word insertion penalty term. The product of the three terms is the total path score.

[0104] a6. Among all candidate paths in the last time frame's speech frame, the text sequence corresponding to the candidate path with the highest score is determined as the speech recognition result.

[0105] For example, for the first time frame t1, if the initial candidate path is an empty string, this embodiment first filters out the possible characters with a probability greater than the probability threshold from each possible character in A11 to A1M. Assuming that for the first time frame, the filtered possible characters include A11, A12, A13, A14 and A15, then A11, A12, A13, A14 and A15 are scored according to the language model, and the scores of A11, A12, A13, A14 and A15 are sorted in descending order. Then, the top 3 possible characters are selected as the final candidate path of the speech frame of the first time frame. Assuming the candidate paths obtained in the first time frame include A11, A13, and A14, then A11, A13, and A14 are determined as the initial candidate paths for the second time frame t2. The same method steps as for the multiple possible characters in the first time frame are then applied to the multiple possible characters in the second time frame's speech frame to obtain the final candidate paths for the second time frame's speech frame. Assuming that for the second time frame's speech frame, the top three possible characters are determined to be A22, A23, and A28, then the candidate paths formed by the first and second time frames can be represented as a tree structure.

[0106] Figure 7 This is a schematic diagram of a tree structure provided in an embodiment of this application. Figure 7 As shown, each branch in the tree structure is a candidate path. Figure 7 The tree structure in the code will serve as the initial candidate path for the third time frame.

[0107] Figure 8 The flowchart of the training method for the acoustic model provided in the embodiments of this application Figure 1 .like Figure 8 As shown, the training method for this acoustic model includes the following steps:

[0108] S801. Obtain a first speech sample and a second speech sample. The first speech sample includes multiple manually calibrated speech samples and corresponding characters of the target language. The second speech sample includes multiple speech samples calibrated by a second acoustic model trained using the first speech sample and corresponding characters of the target language.

[0109] In this embodiment, the first speech sample can be obtained from the training database. The second speech sample can be obtained from the network through web crawling.

[0110] Each speech sample in the first speech sample corresponds to a manually labeled text, which includes the sentence corresponding to the speech sample and the Chinese character corresponding to each pronunciation in the speech sample.

[0111] Each speech sample in the second speech sample does not have manually labeled text. By using the second acoustic model trained based on the first speech sample, the characters of the target language corresponding to multiple speech samples in the second speech sample can be obtained.

[0112] S802. Train the neural network based on the first speech sample and the second speech sample to obtain the first acoustic model.

[0113] The architecture of the neural network in step S802 can be referenced from [reference needed]. Figure 4 The architecture of the neural network is shown in the figure. In this embodiment, the network parameters of the first acoustic model are the network parameters of the neural network at the end of training.

[0114] Specifically, step S802 includes:

[0115] Step b1: For each speech sample in the first and second speech samples, input the speech sample into the neural network to obtain the training results for character recognition of the speech samples.

[0116] In step b1, for each iteration of training, the neural network has network parameters. The network parameters for the first iteration of training can be obtained through initialization or by manual setting.

[0117] The speech sample is input into the neural network, and the network parameters are used to predict the characters in the speech sample to obtain the possible characters and their corresponding character probabilities.

[0118] Specifically, for each speech sample in the first and second speech samples, before inputting the speech sample into the neural network, speech features need to be extracted from the speech sample. The extracted speech features are then input into the neural network to obtain the training results for character recognition of the speech sample. For details on speech feature extraction, please refer to the description in step S203; it will not be repeated here.

[0119] Before extracting speech features, the process also includes preprocessing and framing of the speech samples. Specific implementation details for the preprocessing and framing processes can be found in step S203 and will not be repeated here.

[0120] Step b2: Based on the training results of character recognition from the speech samples and the differences between the characters in the target language corresponding to the speech samples, adjust the network parameters of the neural network until training is complete.

[0121] Figure 9 A logic diagram illustrating the training process of the neural network provided in this application embodiment. For example... Figure 9 As shown, in one optional implementation, the loss function value is calculated based on the training results (predicted output) of character recognition from the speech samples and the characters (labels, such as real labels or pseudo labels) of the target language corresponding to the speech samples. It is then determined whether the loss function value has been minimized. If it has not been minimized, the gradient of the neural network is calculated, i.e., the direction and value of the network parameters are updated. Training continues based on the updated network parameters until the loss function is minimized, at which point training ends. The determination that the loss function has been minimized can be made by checking that the value of the loss function has not changed consecutively, i.e., the value of the loss function no longer decreases.

[0122] In another alternative implementation, a preset number of iterations can be set, and training ends when the preset number of iterations is reached.

[0123] Step b3: Determine the neural network corresponding to the network parameters at the end of training as the first acoustic model.

[0124] Figure 10 The flowchart of the training method for the acoustic model provided in the embodiments of this application Figure 2 .like Figure 10 As shown, step b1 specifically includes the following steps:

[0125] S1001. For each speech sample in the first speech sample and the second speech sample, feature extraction is performed on the speech sample to obtain the first speech feature of the speech sample.

[0126] It is understandable that the first speech sample and the second speech sample constitute a large speech sample set, which includes multiple language samples. This step is to extract features for each speech sample in the speech sample set.

[0127] For details on speech feature extraction from speech samples, please refer to the description of step S203, which will not be repeated here.

[0128] Before performing speech feature extraction, the speech samples are preprocessed and framed. For details on the specific implementation methods of preprocessing and framed processing, please refer to the description in step S203.

[0129] S1002. Input the speech features of the speech sample into the convolution unit to obtain the second speech features of the speech sample. The convolution unit includes a preset number of convolutional neural networks.

[0130] The convolutional unit extracts local features based on the speech features of the speech sample.

[0131] S1003. Input the second speech feature of the speech sample into the self-attention unit to obtain the third speech feature of the speech sample. The self-attention unit includes a preset number of self-attention blocks.

[0132] Among them, the self-attention unit can find the local features of interest among the local features extracted in step b12, and pay attention to the feature data at the feature position corresponding to the local feature during subsequent training.

[0133] S1004. Input the third speech feature of the speech sample into the fully connected unit to obtain the fourth speech feature of the speech sample. The fully connected unit includes a preset number of fully connected layers.

[0134] The fully connected unit combines features based on the local features of interest determined by the self-attention unit and classifies them through a classifier to obtain the fourth speech feature, which is the training result of the current iteration.

[0135] S1005. The fourth speech feature is determined as the training result for character recognition based on speech samples.

[0136] Figure 11 The flowchart of the training method for the acoustic model provided in the embodiments of this application Figure 3 .like Figure 11 As shown, the training method for this acoustic model includes the following steps:

[0137] S1101. Train the neural network based on the first speech sample to obtain the second acoustic model.

[0138] Specifically, step S1101 includes:

[0139] c1. For each of the multiple speech samples in the first speech sample, input the speech sample into the neural network to obtain the training result of character recognition of the speech sample.

[0140] In step c1, for each training iteration, the neural network has network parameters. The network parameters for the first training iteration can be obtained through initialization or by manual setting.

[0141] The speech sample is input into the neural network, and the network parameters are used to predict the characters in the speech sample to obtain the possible characters and their corresponding character probabilities.

[0142] c2. Based on the training results of character recognition from the speech samples and the differences between the characters in the target language corresponding to the speech samples, adjust the network parameters of the neural network until training is complete.

[0143] In one optional implementation, a loss function value is calculated based on the training results of character recognition from speech samples and the characters of the target language corresponding to the speech samples. It is then determined whether the loss function value has been minimized. If it has not been minimized, the gradient of the neural network is calculated, i.e., the direction and value of the network parameters are updated. Training continues based on the updated network parameters until the loss function is minimized, at which point training ends. The determination that the loss function has been minimized can be made by checking that the value of the loss function has not changed consecutively, i.e., the value of the loss function no longer decreases.

[0144] In another alternative implementation, a preset number of iterations can be set, and training ends when the preset number of iterations is reached.

[0145] c3. The neural network corresponding to the network parameters at the end of training is determined as the second acoustic model.

[0146] In this embodiment, the network parameters of the second acoustic model are the network parameters of the neural network at the end of training.

[0147] S1102. Based on multiple speech samples in the second speech sample and the second acoustic model, obtain the characters of the target language corresponding to the speech sample.

[0148] Specifically, step S1102 includes:

[0149] d1. For each speech sample in the multiple speech samples in the second speech sample, input the speech sample into the second acoustic model to obtain the character probability matrix corresponding to the speech sample. The character probability matrix is ​​used to represent the probability of the possible character corresponding to each pronunciation in the speech sample.

[0150] When inputting speech samples into the second acoustic model, feature extraction of the speech samples is required. For details on speech feature extraction, please refer to the introduction of step S203, which will not be repeated here.

[0151] Before extracting speech features, the process also includes preprocessing and framing of the speech samples. Specific implementation details for the preprocessing and framing processes can be found in step S203 and will not be repeated here.

[0152] d2. Input the character probability matrix corresponding to the speech sample into the decoder to obtain the text corresponding to the speech sample. The text includes the characters of the target language corresponding to each speech frame in the multiple speech samples.

[0153] Specifically, the decoder determines the text corresponding to a speech sample based on acoustic features and a language model. More specifically, the decoder constructs a sequence of possible texts from the possible characters of the speech sample, determines whether the possible text sequences conform to language conventions based on the language model, and identifies the most consistent possible text sequence as the text corresponding to the speech sample.

[0154] For details on how the decoder determines the text corresponding to a speech sample based on acoustic features and a language model, please refer to [link to relevant documentation]. Figure 6 Specific implementation of the illustrated embodiments.

[0155] Figure 12 A logic diagram illustrating the training process of the first acoustic model provided in this application embodiment.

[0156] The above embodiment trains a second acoustic model using a limited amount of labeled speech data (first speech samples). This second acoustic model is then used to transcribe a large amount of unlabeled data (multiple speech samples), meaning the unlabeled data acquires pseudo-labels (e.g., unlabeled speech 1, after transcription, becomes text 1, which serves as the pseudo-label). This unlabeled data is then merged with labeled data, and the acoustic model is retrained to obtain a better model. Experimental results within the CTC end-to-end speech recognition model framework are shown in Table 2 below.

[0157]

[0158] Table 2 shows that six test sets were used, comprising 90 hours of labeled data (LD), 450 hours of unlabeled data (ULD), a 1:5 ratio of labeled to unlabeled data, and speech samples collected in various scenarios. Improvements of 2.49% to 4.44% were achieved on these test sets. Furthermore, with the increase in collected unlabeled data, the recognition accuracy is expected to improve further.

[0159] Based on the above embodiments, the following two implementation methods can also be used to train the neural network based on unlabeled data to obtain the first acoustic model.

[0160] In one alternative implementation, the first acoustic model is trained using the following method:

[0161] e1. Input the first speech sample into the neural network to obtain the first feature vector;

[0162] e2. Input multiple speech samples from the second speech sample into the neural network to obtain the second feature vector;

[0163] e3. Calculate the first mean or the first variance based on the first eigenvector;

[0164] e4. Calculate the second mean or second variance based on the second eigenvector;

[0165] e5. Based on the error between the first mean and the second mean, or the error between the first variance and the second variance, adjust the network parameters of the preset network layer in the neural network until training is completed.

[0166] e6. The trained neural network is designated as the first acoustic model.

[0167] In another alternative implementation, the first acoustic model is trained using the following method:

[0168] f1. Train the neural network based on multiple speech samples in the second speech sample to obtain the feature vector model.

[0169] f2. Train the feature vector model based on the first speech sample to obtain the first acoustic model.

[0170] This implementation trains a feature vector model through pre-training, and then fine-tunes the feature vector model based on labeled first speech samples to obtain a first acoustic model with high recognition accuracy. This method can also improve the convergence speed of the training process.

[0171] Based on the training method of the acoustic model described above, the trained acoustic model can be stored in the server and the server can then execute... Figure 2 The steps of the method in the illustrated embodiment can also store the trained acoustic model in the terminal device, so that the terminal device can directly perform speech recognition according to the speech recognition request, eliminating the process of interacting with the server.

[0172] Based on the above method embodiments, Figure 13 This is a schematic diagram of the structure of a speech recognition device provided in an embodiment of this application. Figure 13As shown, the speech recognition device includes: a receiving module 131, a feature extraction module 132, an input module 133, a determining module 134, and an output module 135. The receiving module 131 receives a speech recognition request, which includes speech to be recognized in a target language. The feature extraction module 132 extracts features from the speech to be recognized to obtain speech features. The input module 133 inputs the speech features into a first acoustic model to obtain acoustic features, which include possible characters of the speech to be recognized. The first acoustic model is a model trained on a neural network based on a first speech sample and a second speech sample to recognize possible characters of the target language in speech data. The first speech sample includes multiple manually labeled speech samples and corresponding characters of the target language. The second speech sample includes multiple speech samples labeled using a second acoustic model trained with the first speech sample and corresponding characters of the target language. The determining module 134 determines the speech recognition result of the target language speech based on the acoustic features. The output module 135 is used to output the speech recognition result of the target language speech.

[0173] In some embodiments of this application, the input module 133 inputs the speech features into the first acoustic model to obtain acoustic features, specifically including: inputting the speech features into a convolutional unit to obtain a first speech feature, the convolutional unit including a preset number of convolutional neural networks; inputting the first speech features into a self-attention unit to obtain a second speech feature, the self-attention unit including a preset number of self-attention blocks; and inputting the second speech features into a fully connected unit to obtain the speech features, the fully connected unit including a preset number of fully connected layers.

[0174] In some embodiments of this application, the speech to be identified includes multiple speech frames, and the speech features include speech features corresponding to each speech frame; the input module 133 inputs the speech features into a first acoustic model to obtain acoustic features, specifically including: for each of the multiple speech frames, inputting the speech features corresponding to the speech frame into the first acoustic model to obtain the possible characters of the speech features corresponding to the speech frame, the possible characters corresponding to the possible characters having a character probability, the character probability being used to represent the probability that the character of the speech features corresponding to the speech frame is the possible character; combining the possible characters of the multiple speech frame features according to the time order of the speech frames to obtain a character feature matrix.

[0175] In some embodiments of this application, the determining module 134 determines the speech recognition result of the target language speech based on the acoustic features, specifically including: for the i-th speech frame, determining a target character, wherein the target character is a possible character among the possible characters of the speech frame with a probability greater than a preset probability, where i is a positive integer greater than 0; determining a new candidate path based on the target character and the generated candidate path, wherein the generated candidate path is used to represent the possible text sequence of the 1st to the (i-1th)th speech frames; scoring the new candidate path according to a language model; if the score of the new candidate path is greater than a preset score, then determining the new candidate path as the possible text sequence of the 1st to the i-th speech frames; repeating the above process until the last speech frame, to obtain the speech recognition result of the target language speech.

[0176] In some embodiments of this application, the device further includes: a framing module 136; the framing module 136 is used to perform framing processing on the speech to be recognized to obtain multiple speech frames; and a feature extraction module 132 is used to extract features from each of the multiple speech frames to obtain the speech frame features.

[0177] The speech recognition device provided in this application embodiment can be used to execute the speech recognition method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0178] Based on the above method embodiments, Figure 14 This is a schematic diagram of the structure of the training device for the acoustic model provided in an embodiment of this application. Figure 14 As shown, the training device for the acoustic model includes: an acquisition module 141 and a training module 142; wherein, the acquisition module 141 is used to acquire a first speech sample and a second speech sample, the first speech sample including multiple manually calibrated speech samples and corresponding characters of the target language, and the second speech sample including multiple speech samples calibrated by a second acoustic model trained using the first speech sample and corresponding characters of the target language; the training module 142 is used to train the neural network based on the first speech sample and the second speech sample to obtain a first acoustic model.

[0179] In some embodiments of this application, the device further includes: a determining module 143; a training module 142, further configured to train a neural network based on the first speech sample to obtain a second acoustic model; and a determining module 143, configured to determine the characters of the target language corresponding to the speech sample based on multiple speech samples in the second speech sample and the second acoustic model.

[0180] In some embodiments of this application, the determining module 143 determines the characters of the target language corresponding to the speech sample based on multiple speech samples in the second speech sample and the second acoustic model. Specifically, this includes: for each speech sample in the multiple speech samples in the second speech sample, inputting the speech sample into the second acoustic model to obtain a character probability matrix corresponding to the speech sample, wherein the character probability matrix is ​​used to represent the probability of a possible character corresponding to each pronunciation in the speech sample; and inputting the character probability matrix corresponding to the speech sample into the decoder to obtain the characters of the target language corresponding to the speech sample.

[0181] In some embodiments of this application, the training module 142 trains a neural network based on the first speech sample and the second speech sample to obtain a first acoustic model. Specifically, this includes: extracting features from each speech sample in the first and second speech samples to obtain a first speech feature; inputting the speech feature of the speech sample into a convolutional unit to obtain a second speech feature, the convolutional unit comprising a preset number of convolutional neural networks; inputting the second speech feature of the speech sample into a self-attention unit to obtain a third speech feature, the self-attention unit comprising a preset number of self-attention blocks; inputting the third speech feature of the speech sample into a fully connected unit to obtain a fourth speech feature, the fully connected unit comprising a preset number of fully connected layers; adjusting the network parameters of the convolutional unit, self-attention unit, and fully connected unit according to the difference between the fourth speech feature and the characters of the target language corresponding to the speech sample, until training is complete; and determining the first acoustic model based on the network parameters of the convolutional unit, self-attention unit, and fully connected unit obtained after training.

[0182] The acoustic model training device provided in this application embodiment can be used to execute the technical solution of the acoustic model training method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0183] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, module 134 can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element of the device. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0184] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 15 As shown, the electronic device may include: transceiver 151, processor 152, and memory 153.

[0185] Processor 152 executes computer execution instructions stored in memory, causing processor 152 to perform the scheme in the above embodiments. Processor 152 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0186] The memory 153 is connected to the processor 152 via the system bus and completes communication between them. The memory 153 is used to store computer program instructions.

[0187] Transceiver 151 can be used to receive speech recognition requests and return speech recognition results.

[0188] The system bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into address bus, data bus, control bus, etc. For ease of representation, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. Transceivers are used to enable communication between database access devices and other computers (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and may also include non-volatile memory.

[0189] The electronic device provided in this application embodiment can be the server described in the above embodiment.

[0190] This application also provides a chip for executing instructions, which is used to execute the technical solutions of the speech recognition method or acoustic model training method in the above embodiments.

[0191] This application also provides a computer-readable storage medium storing computer instructions. When the computer instructions are executed on a computer, the computer performs the technical solution of the speech recognition method or acoustic model training method described in the above embodiments.

[0192] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When the at least one processor executes the computer program, it can implement the technical solutions of the speech recognition method or acoustic model training method in the above embodiments.

[0193] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0194] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0196] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0197] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A speech recognition method, characterized in that, include: Receive a speech recognition request, wherein the speech recognition request includes speech to be recognized in the target language; The speech to be identified is subjected to feature extraction to obtain speech features; The speech features are input into a first acoustic model to obtain acoustic features, which include possible characters of the speech to be identified. The first acoustic model is a model trained on a neural network based on a first speech sample and a second speech sample to identify possible characters of the target language in speech data. The first speech sample includes multiple manually labeled speech samples and corresponding characters of the target language. The second speech sample includes multiple speech samples labeled by a second acoustic model trained with the first speech sample and corresponding characters of the target language. Based on the acoustic features, determine the speech recognition result of the target language speech; Output the speech recognition results of the target language.

2. The method according to claim 1, characterized in that, The step of inputting the speech features into the first acoustic model to obtain acoustic features includes: The speech features are input into a convolutional unit to obtain the first speech features. The convolutional unit includes a preset number of convolutional neural networks. The first speech feature is input into the self-attention unit to obtain the second speech feature, wherein the self-attention unit includes a preset number of self-attention blocks; The second speech feature is input into a fully connected unit to obtain the speech feature. The fully connected unit includes a preset number of fully connected layers.

3. The method according to claim 1, characterized in that, The speech to be identified includes multiple speech frames, and the speech features include the speech features corresponding to each speech frame. Accordingly, inputting the speech features into the first acoustic model to obtain acoustic features includes: For each of the plurality of speech frames, the speech features corresponding to the speech frame are input into the first acoustic model to obtain the possible characters of the speech features corresponding to the speech frame. Each possible character has a character probability, and the character probability is used to represent the probability that the character of the speech feature corresponding to the speech frame is the possible character. The possible characters of multiple speech frame features are combined according to the time order of the speech frames to obtain the character feature matrix.

4. The method according to claim 3, characterized in that, The step of determining the speech recognition result of the target language speech based on the acoustic features includes: For the i-th speech frame, a target character is determined. The target character is a possible character among the possible characters of the speech frame whose probability is greater than a preset probability, where i is a positive integer greater than 0. Based on the target character and the generated candidate paths, a new candidate path is determined, wherein the generated candidate paths are used to represent the possible text sequences of the first to the (i-1)th speech frames; The new candidate paths are scored based on the language model; If the score of the new candidate path is greater than the preset score, then the new candidate path is determined as the possible text sequence of the first to the i-th speech frames; Repeat the above process until the last speech frame to obtain the speech recognition result of the target language speech.

5. The method according to claim 3 or 4, characterized in that, The speech features corresponding to each speech frame are determined according to the following method: The speech to be recognized is segmented into frames to obtain multiple speech frames; For each of the multiple speech frames, feature extraction is performed on the speech frame to obtain the speech frame features.

6. A method for training an acoustic model, characterized in that, include: Acquire a first speech sample and a second speech sample. The first speech sample includes multiple manually calibrated speech samples and corresponding characters of the target language. The second speech sample includes multiple speech samples calibrated by a second acoustic model trained using the first speech sample and corresponding characters of the target language. The neural network is trained based on the first speech sample and the second speech sample to obtain the first acoustic model.

7. The method according to claim 6, characterized in that, The method further includes: The neural network is trained based on the first speech sample to obtain the second acoustic model; Based on multiple speech samples in the second speech sample and the second acoustic model, the characters of the target language corresponding to the speech sample are determined.

8. The method according to claim 7, characterized in that, The step of determining the characters of the target language corresponding to the speech samples based on multiple speech samples in the second speech sample and the second acoustic model includes: For each of the multiple speech samples in the second speech sample, the speech sample is input into the second acoustic model to obtain the character probability matrix corresponding to the speech sample. The character probability matrix is ​​used to represent the probability of the possible character corresponding to each pronunciation in the speech sample. The character probability matrix corresponding to the speech sample is input into the decoder to obtain the characters of the target language corresponding to the speech sample.

9. The method according to claim 6, characterized in that, The step of training the neural network based on the first speech sample and the second speech sample to obtain the first acoustic model includes: For each speech sample in the first speech sample and the second speech sample, feature extraction is performed on the speech sample to obtain the first speech feature of the speech sample; The speech features of the speech sample are input into a convolutional unit to obtain the second speech features of the speech sample. The convolutional unit includes a preset number of convolutional neural networks. The second speech feature of the speech sample is input into the self-attention unit to obtain the third speech feature of the speech sample. The self-attention unit includes a preset number of self-attention blocks. The third speech feature of the speech sample is input into a fully connected unit to obtain the fourth speech feature of the speech sample. The fully connected unit includes a preset number of fully connected layers. Based on the differences between the fourth speech feature and the characters of the target language corresponding to the speech sample, the network parameters of the convolutional unit, self-attention unit, and fully connected unit are adjusted respectively until the training ends; The first acoustic model is determined based on the network parameters of the convolutional units, self-attention units, and fully connected units obtained after training.

10. A voice recognition device, characterized in that, include: A receiving module is used to receive a speech recognition request, wherein the speech recognition request includes speech to be recognized in the target language; The feature extraction module is used to extract features from the speech to be recognized to obtain speech features; An input module is used to input the speech features into a first acoustic model to obtain acoustic features, wherein the acoustic features include possible characters of the speech to be identified. The first acoustic model is a model trained on a neural network based on a first speech sample and a second speech sample for identifying possible characters of the target language in speech data. The first speech sample includes multiple manually labeled speech samples and corresponding characters of the target language. The second speech sample includes multiple speech samples labeled by a second acoustic model trained with the first speech sample and corresponding characters of the target language. The determining module is used to determine the speech recognition result of the target language speech based on the acoustic features; The output module is used to output the speech recognition results of the target language speech.

11. A training device for an acoustic model, characterized in that, include: The acquisition module is used to acquire a first speech sample and a second speech sample. The first speech sample includes multiple manually calibrated speech samples and corresponding characters of the target language. The second speech sample includes multiple speech samples calibrated by a second acoustic model trained using the first speech sample and corresponding characters of the target language. The training module is used to train the neural network based on the first speech sample and the second speech sample to obtain the first acoustic model.

12. An electronic device, characterized in that, include: Memory, processor; Memory, used to store the processor-executable instructions; The processor is configured to implement the method as described in any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-10.

14. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 1-10.

Images