Acoustic model construction method and system, speech recognition method, device and medium

Abstract

An acoustic model construction method and system, a speech recognition method, a device and a storage medium, wherein the construction method comprises: obtaining unlabeled training data and labeled training data; constructing an acoustic model, the acoustic model being used to obtain conditional probabilities corresponding to phonemes; the acoustic model comprising an audio representation model, the audio representation model being used to obtain representation vectors based on input speech; using a training speech set, pre-training the audio representation model based on the correlation between frames; after pre-training the audio representation model, using labeled training data, training the acoustic model based on annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and speech data. The embodiments of the present invention improve the training effect of the acoustic model.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of speech processing, and in particular to an acoustic model construction method and system, a speech recognition method, device and storage medium. Background Art

[0002] The acoustic model (AM) is one of the most important parts of the speech recognition / evaluation system, and its performance greatly affects the accuracy of the evaluation system. Currently, most mainstream systems use hidden Markov models for modeling. For speech recognition / evaluation systems, the output value of the acoustic model is usually the acoustic features calculated from each frame.

[0003] However, the performance of the acoustic models currently constructed still needs to be improved. Summary of the invention

[0004] The problem solved by the embodiments of the present invention is to provide an acoustic model construction method and system, a speech recognition method, device and storage medium, which improve the speech recognition performance of the acoustic model and reduce the dependence on the language model.

[0005] To solve the above problems, an embodiment of the present invention provides a method for constructing an acoustic model, comprising: obtaining unlabeled training data and labeled training data, the labeled training data comprising speech data and corresponding annotated text, the unlabeled training data and the speech data constituting a training speech set; constructing an acoustic model, the acoustic model being used to obtain conditional probabilities corresponding to phonemes; the acoustic model comprising an audio representation model, the audio representation model being used to obtain representation vectors based on input speech; using the training speech set, based on the correlation between frames, pre-training the audio representation model; after pre-training the audio representation model, using the labeled training data, based on the annotated text, the correlation between words in the annotated text, and the correlation between words and representation vectors corresponding to the speech data, training the acoustic model.

[0006] Correspondingly, an embodiment of the present invention also provides an acoustic model construction system, including: a training data acquisition module, used to acquire unlabeled training data and labeled training data, the labeled training data includes speech data and corresponding annotated text, and the unlabeled training data and the speech data constitute a training speech set; a model construction module, used to construct an acoustic model, the acoustic model is used to obtain the conditional probability corresponding to the phoneme; the acoustic model includes an audio representation model, the audio representation model is used to obtain a representation vector based on the input speech; a pre-training module, used to use the training speech set, based on the correlation between frames, to pre-train the audio representation model; an acoustic model training module, used to use the labeled training data, based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data, to train the acoustic model.

[0007] Correspondingly, an embodiment of the present invention also provides a speech recognition method, comprising: obtaining speech data to be recognized; inputting the speech data into an acoustic model constructed using the acoustic model construction method of an embodiment of the present invention to obtain phoneme conditional probability; and obtaining a speech recognition result based on the phoneme conditional probability.

[0008] Correspondingly, an embodiment of the present invention also provides a device, comprising at least one memory and at least one processor, wherein the memory stores one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the acoustic model construction method provided by the embodiment of the present invention.

[0009] Correspondingly, an embodiment of the present invention further provides a storage medium, characterized in that the storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the acoustic model construction method provided by the embodiment of the present invention.

[0010] Compared with the prior art, the technical solution of the embodiment of the present invention has the following advantages:

[0011] The acoustic model construction method provided by the embodiment of the present invention comprises an audio representation model in the process of constructing the acoustic model; accordingly, the audio representation model is pre-trained based on the training speech set and the correlation between frames, so that not only the audio representation model can learn the correlation of the audio context, but also the pronunciation representation learning using the unlabeled training speech data can effectively avoid the problem of "sparse" context data, and accordingly expand the expression ability of the audio representation model for phonemes; and after the audio representation model is pre-trained, the acoustic model is trained using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data, so that the acoustic model can learn the correlation of the text context and combine the pronunciation representation with the text context for learning, and because the audio representation model is trained using the unlabeled training speech data alone before the acoustic model is trained, accordingly, less labeled training data is required when training the acoustic model; in summary, the embodiment of the present invention avoids the frame independence assumption in the traditional modeling method, and can also better grasp the contextual information of the audio. Information), thereby improving the training effect of the acoustic model, correspondingly improving the speech recognition performance of the acoustic model, and reducing the dependence on the language model.

[0012] In the acoustic model construction system provided by the embodiment of the present invention, the acoustic model constructed by the model construction module includes an audio representation model; accordingly, the pre-training module uses the training speech set and the correlation between frames to pre-train the audio representation model, so that not only the audio representation model can learn the correlation of the audio context, but also the pronunciation representation learning using the unlabeled training speech data can effectively avoid the problem of "sparse" context data, and correspondingly expands the expression ability of the audio representation model for phonemes; and the acoustic model training module uses the labeled training data to pre-train the audio representation model based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data. The acoustic model is trained so that the acoustic model can learn the relevance of the text context and combine the pronunciation representation with the text context for learning. Moreover, before the acoustic model is trained, the pre-training module trains the audio representation model using the unlabeled training speech data alone. Accordingly, the acoustic model training module requires less labeled training data when training the acoustic model. In summary, the embodiments of the present invention avoid the frame independence assumption in the traditional modeling method, and can also better grasp the contextual information (Contextual Information) of the audio, thereby improving the training effect of the acoustic model, correspondingly improving the speech recognition performance of the acoustic model, and reducing the dependence on the language model. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a flow chart of an embodiment of an acoustic model building method of the present invention;

[0014] Figure 2 is a schematic diagram of a network structure of an embodiment of an audio representation model;

[0015] Figure 3 is a structural schematic diagram of an embodiment of the acoustic model;

[0016] Figure 4 is a schematic diagram of the network structure of an embodiment of a decoding module;

[0017] Figure 5 Shows Figure 1 Flow chart of an embodiment of step S3;

[0018] Figure 6 A schematic diagram of a process flow of an embodiment of a conversion process is shown;

[0019] Figure 7 Shows Figure 1 A schematic diagram of a flow chart of an embodiment of step S4 in FIG.

[0020] Figure 8 Shows Figure 7 Schematic diagram of an embodiment of step S44

[0021] Fig. 9 is a functional block diagram of an embodiment of an acoustic model building system of the present invention;

[0022] Fig.10 is a flow chart of an embodiment of a speech recognition method of the present invention;

[0023] Fig.11 It is a hardware structure diagram of a device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0024] As can be seen from the background technology, the performance of the currently constructed acoustic model still needs to be improved.

[0025] Specifically, in the field of spoken language assessment, training an acoustic model with a high recognition rate requires a large amount of annotated speech data. Acoustic models often use standard phonemes, such as tri-phones or more factor phonemes, such as quin-phones, to describe the impact of the context of pronunciation on phonemes, and divide the same phoneme into multiple states to represent temporal changes.

[0026] A common approach in the prior art is to use a three-factor phoneme (Tri-phone) for modeling, and each phoneme has three states. The states of the phonemes are usually bound according to data features. Common state binding methods include data-driven clustering and decision tree clustering.

[0027] Taking the decision tree clustering method as an example, different phoneme models are expanded for each phoneme in the context of each phoneme. Since the available annotated speech data is often small, it is impossible to cover all phoneme models. For some phonemes, the corresponding data is often sparse, resulting in poor results in training the acoustic model.

[0028] In order to solve the above technical problems, an embodiment of the present invention provides a method for constructing an acoustic model. Figure 1 , showing a flow chart of an embodiment of an acoustic model construction method of the present invention

[0029] In this embodiment, the acoustic model construction method includes the following basic steps:

[0030] Step S1: Obtain unlabeled training data and labeled training data, wherein the labeled training data includes speech data and corresponding annotated text, and the unlabeled training data and the speech data constitute a training speech set;

[0031] Step S2: constructing an acoustic model, the acoustic model being used to obtain conditional probabilities corresponding to phonemes; the acoustic model comprising an audio representation model, the audio representation model being used to obtain a representation vector based on input speech;

[0032] Step S3: using the training speech set, based on the correlation between frames, pre-training the audio representation model;

[0033] Step S4: After pre-training the audio representation model, the acoustic model is trained using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the words and the representation vectors corresponding to the speech data.

[0034] The acoustic model construction method, in the process of constructing the acoustic model, the acoustic model includes an audio representation model; accordingly, the audio representation model is pre-trained based on the training speech set and the correlation between frames, so that not only the audio representation model can learn the correlation of the audio context, but also the pronunciation representation learning using unlabeled training speech data can effectively avoid the problem of "sparse" context data, and correspondingly expand the expression ability of the audio representation model for phonemes; and after the audio representation model is pre-trained, the acoustic model is trained using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data, so that the acoustic model can learn the correlation of the text context and combine the pronunciation representation with the text context for learning, and because the audio representation model is trained using the unlabeled training speech data alone before the acoustic model is trained, accordingly, less labeled training data is required when training the acoustic model; in summary, the embodiment of the present invention avoids the frame independence assumption in the traditional modeling method, and can also better grasp the contextual information of the audio. Information), thereby improving the training effect of the acoustic model, correspondingly improving the speech recognition performance of the acoustic model, and reducing the dependence on the language model.

[0035] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0036] refer to Figure 1 , execute step S1: obtain unlabeled training data and labeled training data, the labeled training data includes speech data and corresponding annotated text, and the unlabeled training data and the speech data constitute a training speech set.

[0037] Unlabeled training data and labeled training data are used to form training samples for training acoustic models.

[0038] Among them, unlabeled training data does not need to be manually labeled, so unlabeled training data is easier to obtain than labeled training data, so that a large amount of unlabeled training data can be obtained, which is beneficial to avoid the problem of insufficient "sparse" context data, and accordingly ensures the training effect of the acoustic model while also helping to reduce costs.

[0039] Specifically, in this embodiment, the acoustic model is applied to the field of spoken language assessment. Spoken language assessment refers to the overall evaluation of a person's spoken language level using speech recognition technology given text and audio.

[0040] Therefore, the unlabeled training data is unlabeled spoken pronunciation data. The labeled training data includes spoken pronunciation data and corresponding annotated text.

[0041] In this embodiment, both the unlabeled training data and the labeled training data include: pronunciation data of native speakers of the target language, pronunciation data of second language learners, and native language pronunciation data of second language learners.

[0042] Among them, the pronunciation data of the target language native speaker refers to the pronunciation data of a person whose native language is the target language; the pronunciation data of the second language learner refers to the pronunciation data of a person learning the target language speaking the target language; and the second language learner native language pronunciation data refers to the pronunciation data of a person learning the target language speaking his or her native language.

[0043] For example, in the scenario of Chinese people learning English, the pronunciation data of the target language native speaker refers to the pronunciation data of English spoken by a native English speaker, the pronunciation data of the second language learner refers to the pronunciation data of Chinese people speaking English, and the second language learner native language pronunciation data refers to the pronunciation data of Chinese people speaking Chinese.

[0044] By making the unlabeled training data and the labeled training data include the pronunciation data of the target language native speakers, the pronunciation data of the second language learners, and the native language pronunciation data of the second language learners, in the subsequent pre-training of the audio representation model based on the training speech set and the correlation between each frame and all frames in the speech, similar pronunciation phonemes between different languages ​​will share representation vectors, and language-specific pronunciation phonemes will independently form different representation vectors. These representation vectors will be combined into a "cross-language", rich and huge representation space, thereby expanding the model's ability to express phonemes, so that the pronunciation representation and phonemes have a high correlation.

[0045] Moreover, in the subsequent pre-training process of the audio representation model, it is possible to learn the pronunciation of second language learners that is biased by their native language, and accordingly, it can additionally output the phoneme conditional probability of the erroneous phonemes, which can provide diversified feature selection for subsequent oral evaluation tasks.

[0046] Continue to refer Figure 1 , execute step S2: construct an acoustic model, the acoustic model is used to obtain the conditional probability corresponding to the phoneme; the acoustic model includes an audio representation (Speech Representation) model, and the audio representation model is used to obtain a representation vector based on the input speech.

[0047] The audio representation model is used to represent the input audio, thereby representing the characteristics of the phonemes, the differences between the phonemes, etc.

[0048] The acoustic model includes an audio representation model, so that the audio representation model can be trained using unlabeled training speech data alone in the future. Accordingly, less labeled training data is required when training the acoustic model. In addition, the subsequent use of unlabeled training speech data for pronunciation representation learning can effectively avoid the problem of "sparse" context data, and correspondingly expand the ability of the audio representation model to express phonemes.

[0049] Combined with reference Figure 2 , shows a schematic diagram of the network structure of an embodiment of the audio model 10.

[0050] In this embodiment, the audio representation model 10 includes: a convolution layer 101, which is used to perform convolution processing on the speech to obtain a hidden layer representation Z; a quantization layer 102, which is used to quantize the hidden layer representation to obtain a quantized representation corresponding to the current frame; a mask converter 103, which obtains an audio context representation V2 based on the correlation between the hidden layer representation of each frame and the hidden layer representation of the frame within a preset range; a splicing layer 104, which is used to splice and combine the quantized representation and the audio context representation to obtain the representation vector.

[0051] Combined with reference Figure 3 , showing a structural schematic diagram of an embodiment of the acoustic model.

[0052] refer to Figure 3 In this embodiment, the acoustic model also includes: a text embedding layer 60, which is used to obtain a word vector based on the annotated text; a position encoding layer 50, which is used to output a corresponding position encoding vector based on the word vector; an overlay layer 70, which is used to overlay the word vector with the corresponding position encoding vector; a decoding module 20, which is connected to the audio representation model 10 and the overlay layer 70; a linear transformation layer 30, which is connected to the decoding module 20; and a normalization layer 40, which is connected to the linear transformation layer 30, which is used to output the conditional probability corresponding to the phoneme.

[0053] Combined with reference Figure 4 , shows a schematic diagram of the network structure of an embodiment of the decoding module.

[0054] As one embodiment, the decoding module 20 includes: a first attention module 21, connected to the superposition layer 70; a first residual normalization layer 22, connected to the superposition layer 70 and the first attention module 21; a second attention module 23, connected to the first residual normalization layer 22; a second residual normalization layer 24, connected to the second attention module 23 and the first residual normalization layer 22; a feedforward layer 25, connected to the second residual normalization layer 22; and a third residual normalization layer 26, connected to the feedforward layer 25 and the second residual normalization layer 22.

[0055] It should be noted that, in this embodiment, the acoustic model construction method further includes: before constructing the acoustic model, obtaining a phoneme set, wherein the phoneme set includes standard phonemes and erroneous phonemes. Therefore, in this embodiment, based on the phoneme set, an acoustic model is constructed, and the acoustic model is used to obtain the conditional probability corresponding to the standard phonemes and the conditional probability corresponding to the erroneous phonemes, which can provide diversified feature selection for subsequent oral evaluation tasks.

[0056] Continue to refer Figure 1 , execute step S3: use the training speech set to pre-train the audio representation model based on the correlation between frames.

[0057] The audio representation model is pre-trained using the training speech set based on the correlation between frames, so that not only can the audio representation model learn the correlation of the audio context, but also the pronunciation representation learning is performed using unlabeled training speech data, which reduces the dependence on labeled data during the pre-training process, can effectively avoid the problem of "sparse" context data, and correspondingly expands the audio representation model's ability to express phonemes.

[0058] Accordingly, in this embodiment, the audio representation model performs self-supervised representation learning.

[0059] Specifically, please refer to Figure 5 , showing Figure 1 Flow chart of an embodiment of step S3 in FIG.

[0060] As an embodiment, the step S3 of pre-training the audio representation model based on the training speech set and the correlation between each frame and all frames in the speech includes:

[0061] like Figure 5 As shown, step S31 is executed: convolution processing is performed on the speech in the training speech set to obtain a hidden layer representation.

[0062] Specifically, in conjunction with reference Figure 2In this embodiment, performing convolution processing on the speech in the training speech set includes: inputting the speech in the training speech set into a convolutional neural network (i.e., the convolution layer 101) for convolution processing to obtain the hidden layer representation Z.

[0063] Specifically, the convolution layer 101 is a multi-layer one-dimensional convolution layer.

[0064] As an embodiment, the convolution kernels of the convolution layer 101 are (10, 3, 3, 3, 3, 2, 2), the strides are (5, 2, 2, 2, 2, 2, 2), and the number of channels is 512. The convolution layer 101 outputs the hidden layer representation Z, and the receptive field of the hidden layer representation Z is 400 sampling points (16k sampling, i.e., 25ms), and the time span is 320 sampling points (16k sampling, i.e., 20ms), which is equivalent to the frame length and frame shift of traditional speech acoustic features.

[0065] like Figure 5 As shown, step S32 is executed: quantizing the hidden layer representation to obtain the quantized representation corresponding to the current frame. Quantizing is performed to learn the representation vector, and then the corresponding phoneme representation can be obtained from the quantization code table.

[0066] Specifically, in this embodiment, the quantization processing of the hidden layer representation includes: mapping the hidden layer representation corresponding to each phoneme to a quantization codebook, so that the input vector can be replaced by the index of the codeword in the codebook that best matches the input vector for transmission and storage, and then the corresponding audio representation vector can be extracted by only performing a simple table lookup operation using the index.

[0067] As an example, assuming that the dimension of the hidden layer representation Z is D, and there are V quantization codebooks E = [e1, e2, ..., eV], then the distance (logit distance) between the input vector Z and any entry is l v The definition is as follows:

[0068] 1 v =log[1+exp(-Z T e v )],v=1,2,...,V (I)

[0069] The logit distance is defined to calculate which entry the input vector Z is closer to, so as to select the corresponding codebook entry.

[0070] In the specific implementation, the D dimensions of the hidden layer representation Z are usually divided into G groups, and the quantization codebook is also G groups, each group includes V entries, the encoding dimension is D / G, and the logit distance l between any group of inputs and any entry in the same group g,v The calculation method is similar. In this embodiment, the selection process of the coding entry P g,v Calculated by formula (II):

[0071]

[0072] Where τ is a non-negative temperature constant (usually set to 2, with a decay factor of 0.999995 as the number of iterations increases), n v is Gumbel random noise, through the formula n v =-log(-log(u)), calculated from u to U(0,1), where u is uniformly distributed in the interval (0,1), and k∈[1,V].

[0073] In which, by introducing random noise n v , which makes the training more stable and thus improves the robustness.

[0074] In this embodiment, the method of using Gumbel Softmax to make the selection process of coding items differentiable is used as an example for description.

[0075] In this embodiment, in the specific training process, the forward calculation selects the corresponding codebook entry according to argmaxpg,v, and the gradient back propagation is strictly based on p g,v To calculate. When the training is finished and the module is used, n v Set to 0.

[0076] like Figure 5 As shown, step S33 is performed: based on the correlation between the hidden layer representation of each frame and the hidden layer representation of the frames within a preset range, the audio context representation is obtained.

[0077] Based on the correlation between the hidden layer representation of each frame and the hidden layer representation of the frames within a preset range, the audio representation model can learn the correlation between the current frame and other frames within the preset range, that is, learn the correlation of the audio context.

[0078] Combined with reference Figure 2 In a specific implementation, the hidden layer representation can be input into the mask converter 103 to obtain the correlation between the hidden layer representation of each frame and the hidden layer representation of the frames within a preset range.

[0079] In this embodiment, obtaining the audio context representation includes performing one or more conversion processes in sequence. Figure 6, showing a flow chart of an embodiment of conversion processing.

[0080] In this embodiment, the conversion process includes:

[0081] like Figure 6 As shown, step S331 is executed: a weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frames within a preset range is calculated, and a weighted hidden layer representation obtained by superimposing the weight relationship and the hidden layer representation of the current frame is output.

[0082] Specifically, the attention mechanism can be used to calculate the weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frames within the preset range. More specifically, the multi-head attention mechanism can be used to calculate the weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frames within the preset range, so that the weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frames within the preset range can be paid attention to from different dimensions.

[0083] like Figure 6 As shown, step S332 is executed: performing a first normalization process on the weighted hidden layer representation and outputting a first normalized representation. The first normalization process is used to suppress the vanishing of the deep network gradient and accelerate convergence.

[0084] like Figure 6 As shown, step S333 is executed: the first normalized representation is input into a feedforward network (FeedForward) for processing.

[0085] like Figure 6 As shown, step S334 is executed: the output of the feedforward network is superimposed with the first normalized representation.

[0086] like Figure 6 As shown, step S335 is executed: after superimposing the output of the feedforward network with the first normalized representation, a second normalization process is performed to output a second normalized representation. The second normalization process is used to suppress the vanishing of the deep network gradient and accelerate convergence.

[0087] In a specific embodiment, after performing one or more conversion processes, the last second normalized representation output is used as the audio context representation.

[0088] Continue to refer Figure 5 , executing step S34: obtaining a loss function (Loss Function) based on the quantization representation and the audio context representation.

[0089] The loss function is used to measure the gap between the output of the model and the true value, and then the parameters of the audio representation model can be adjusted based on the loss function, and the audio representation model can be optimized accordingly to obtain a trained audio representation model.

[0090] In this embodiment, the audio representation model is subjected to self-supervised representation learning, and the loss function is shown in formula (III):

[0091]

[0092] Among them, L m is the contrast loss function, L d is the diversity loss function, and α is a hyperparameter. In this embodiment, α is set to 0.1.

[0093] In this embodiment, any time span (time-step) is selected as the starting position with a certain probability p (usually set to 0.065), and then the subsequent M consecutive time spans (usually set to 10) are masked (replaced with a common fixed vector), the speech representation output corresponding to the center position of the mask is ct, and the corresponding selected quantization entry is qt, which together with K (usually set to 100) quantization entries randomly selected from other mask segments constitute a set Qt.

[0094] In this embodiment, one purpose of performing self-supervised representation learning on the audio representation model is to make the similarity between ct and qt as large as possible as possible from the similarity between other quantization items in ct and Qt, which is called contrast loss. The contrast loss is calculated using the contrast loss function. Another purpose is to make the quantization items as diverse as possible, that is, g,v The entropy is the largest, which is called diversity loss, that is, it is calculated using the diversity loss function.

[0095] In this embodiment, the contrast loss function L m Calculated by formula (IV):

[0096]

[0097] Wherein, K is another non-negative temperature constant (usually set to 0.1), sim(a, b) is a similarity function, and the similarity function is usually calculated using cosine distance, that is, in this embodiment, the formula sim(a, b)=a is used. T b / ||a||||b||calculate the similarity function.

[0098] In this embodiment, the diversity loss function L d Calculated by formula (V):

[0099]

[0100] Among them, g is the number of groups of items, v is the number of items in each group, H(.) is the information entropy,

[0101] Continue to refer Figure 5 , execute step S35: adjust the parameters of the audio representation model according to the loss function, so as to optimize the parameters of the audio representation model.

[0102] Specifically, in this embodiment, the loss is calculated according to the loss function, the gradient of the loss function is calculated, and the parameters of the audio representation model are updated through back propagation.

[0103] The audio representation model is used to perform data fusion on the quantized representation and the audio context representation, and output a representation vector corresponding to each frame.

[0104] In this embodiment, the audio representation model performs data fusion on the quantization representation and the audio context representation, including: concatenating and combining the quantization representation and the audio context representation to obtain the representation vector.

[0105] It should be noted that, in this embodiment, the step of pre-training the audio representation model using the training speech set based on the correlation between frames also includes: after obtaining the hidden layer representation and before obtaining the correlation between the hidden layer representation of each frame and the hidden layer representation of all frames, performing a first position encoding process on the hidden layer representation.

[0106] In this embodiment, the step of performing first position coding processing includes: obtaining a corresponding first position coding vector based on the input speech; and superimposing the hidden layer representation with the corresponding first position coding vector.

[0107] Continue to refer Figure 1 , execute step S4: after pre-training the audio representation model, use the labeled training data to train the acoustic model based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data.

[0108] After pre-training the audio representation model, the acoustic model is trained using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data, so that the acoustic model can learn the correlation of the text context and combine the pronunciation representation and the text context for learning. Moreover, since the audio representation model is trained using the unlabeled training speech data alone before training the acoustic model, correspondingly, less labeled training data is required when training the acoustic model.

[0109] In summary, this embodiment avoids the frame independence assumption in the traditional modeling method, and can also better capture the contextual information of the audio, thereby improving the training effect of the acoustic model, correspondingly improving the speech recognition performance of the acoustic model, and reducing the dependence on the language model.

[0110] In this embodiment, the acoustic model is applied to the field of oral evaluation; the labeled training data includes pronunciation data of second language learners; and the acoustic model is trained using the pronunciation data of second language learners.

[0111] By using the pronunciation data of second language learners to train the acoustic model, the training data is enriched, and when using the acoustic model constructed in this embodiment for speech recognition or oral evaluation, it is possible to better handle the situation where the actual user pronunciation is biased by the native language.

[0112] In this embodiment, the labeled training speech data also includes: pronunciation data of native speakers of the target language and pronunciation data of native speakers of the second language learners.

[0113] As an embodiment, the acoustic model is trained based on the pronunciation data of the target language native speaker, the pronunciation data of the second language learner, and the native language pronunciation data of the second language learner.

[0114] In addition to using the pronunciation data of the second language learner, the acoustic model is also trained using the pronunciation data of the target language native speaker and the native language pronunciation data of the second language learner, thereby enriching the training data. When the acoustic model constructed in this embodiment is used for speech recognition or oral evaluation, it is possible to better handle the situation where the actual user pronunciation is biased by the native language, and to take into account the erroneous phonemes in all the training data.

[0115] In other embodiments, training the acoustic model may further include: performing a first sub-training on the acoustic model based on pronunciation data of a native speaker of the target language and native pronunciation data of a second language learner; and after the first sub-training, performing a second sub-training on the acoustic model based on the pronunciation data of the second language learner.

[0116] By performing a first sub-training on the acoustic model using the pronunciation data of the target language native speakers and the native language pronunciation data of the second language learners, and then performing a second sub-training on the acoustic model using the pronunciation data of the second language learners, it is possible to specifically train the pronunciation data of the second language learners in a specific country or region, which is more targeted and can improve the training effect.

[0117] Combined with reference Figure 7 , which shows a flow chart of an embodiment of step S4. The specific steps of training the acoustic model in this embodiment are described in detail below in conjunction with the accompanying drawings.

[0118] refer to Figure 7 , execute step S41, input the speech data of the labeled training data into the audio representation model to obtain a representation vector. Specifically, input the speech data of the labeled training data into the audio representation model 30 to obtain a representation vector corresponding to each frame.

[0119] It can be seen from the above records that the audio representation model is pre-trained based on the correlation between frames using the training speech set. Therefore, the audio representation model can learn the correlation of the audio context. Accordingly, the representation vectors corresponding to each frame obtained by the audio representation model 30 can well represent the correlation between the audio contexts.

[0120] Continue to refer Figure 7 , execute step S42 to obtain a word vector based on the annotated text.

[0121] Specifically, the annotated text is input into the text embedding layer 60 for text embedding processing to obtain a word vector. The annotated text is embedded to obtain a matrix consisting of vectors corresponding to each phoneme in the annotated text, so that when using the acoustic model, the corresponding vector can be found based on the word.

[0122] Continue to refer Figure 7 , execute step S43. In this embodiment, the step of training the acoustic model also includes: after obtaining the word vector, based on the representation vector and the word vector, before outputting the hidden layer vector, performing a second position encoding process on the word vector.

[0123] The word vector is subjected to a second position encoding process, thereby combining the word order information and the word vector to form a new representation, thereby being able to learn the word order information.

[0124] In this embodiment, the step of performing the second position encoding process includes: obtaining a corresponding second position encoding vector based on the word vector corresponding to the annotated text; and superimposing the word vector with the second position encoding vector. Specifically, the position encoding layer 50 obtains a corresponding second position encoding vector based on the word vector corresponding to the annotated text; and the superimposing layer 70 superimposes the word vector with the corresponding second position encoding vector.

[0125] Specifically, the second position encoding vector is obtained using the following formula:

[0126] PE(pos,2i)=sin(pos / (10000^(2i / d)))

[0127] PE(pos,2i+1)=cos(pos / (10000^(2i / d)))

[0128] Among them, d is the output dimension of the text embedding layer, pos is the position corresponding to the word vector, i∈[0,d / 2).

[0129] In this embodiment, when in use, the corresponding second position encoding vector is obtained according to the corresponding position of the currently processed minimum modeling unit in the annotated text and the length of the text embedding layer, wherein the minimum modeling unit is a phoneme.

[0130] Continue to refer Figure 7 , execute step S44, output a hidden layer vector based on the representation vector and the word vector, and the hidden layer vector learns the correlation between the word vectors and the word vectors, and the correlation between the word vectors and the representation vector.

[0131] Specifically, the representation vector and the word vector are input into the decoding module 20, and a hidden layer vector is output.

[0132] Combined with reference Figure 8 , showing Figure 7 Specifically, in this embodiment, based on the representation vector and the word vector, the step of outputting the hidden layer vector includes:

[0133] like Figure 8 As shown, step S441 is executed: based on the output of the second position encoding process and the correlation between the word vectors, a first sub-hidden layer vector is obtained.

[0134] Specifically, in this embodiment, the first attention module 21 obtains the first sub-hidden layer vector based on the output of the second position encoding processing and the correlation between word vectors.

[0135] Based on the output of the second position encoding process and the correlation between word vectors, a first sub-hidden layer vector is obtained, so that the first sub-hidden layer vector can learn the dependency between text contexts.

[0136] Specifically, the weight relationship between the current word and the words in the preset range in the annotated text is calculated, and the first sub-hidden layer vector obtained by superimposing the weight relationship and the word vector is output.

[0137] It should be noted that, in this embodiment, after obtaining the first sub-hidden layer vector, based on the representation vector and the word vector, the step of outputting the hidden layer vector further includes: executing step S442: performing a first residual normalization process to superimpose the first sub-hidden layer vector with the word vector. Specifically, the first residual normalization layer is respectively connected to the position encoding layer and the first attention module, so as to superimpose the first sub-hidden layer vector with the word vector to obtain the second sub-hidden layer vector.

[0138] like Figure 8 As shown, step S443 is performed: based on the representation vector and the second sub-hidden layer vector, a third sub-hidden layer vector is obtained, and the third sub-hidden layer vector learns the correlation between the word vector and the representation vector. Specifically, the representation vector and the second sub-hidden layer vector are input into the second attention module to obtain the third sub-hidden layer vector.

[0139] Therefore, in this embodiment, not only the context of the audio is considered, but also the context of the text, so that a stronger context dependency can be learned. When the acoustic model constructed using the modeling method of this embodiment is used for speech recognition, speech recognition can take into account the context dependency of pronunciation and text, thereby improving the accuracy of speech recognition accordingly.

[0140] As an embodiment, an attention mechanism is used to calculate the weight relationship between the second sub-hidden layer vector and the characterization vector based on the following formula, and a third sub-hidden layer vector after the weight relationship and the characterization vector are superimposed is output:

[0141]

[0142] Among them, K and V are the input matrices from the audio representation model, that is, the input matrices composed of all frame representation vectors, and K and V are the same; dk is the number of rows of the input matrix K, and Q is the matrix composed of the second sub-hidden layer vectors corresponding to all words.

[0143] like Figure 8As shown, in this embodiment, based on the representation vector and the word vector, the step of outputting the hidden layer vector further includes: executing step S444 to perform a second residual normalization process to superimpose the third sub-hidden layer vector with the second sub-hidden layer vector. Specifically, the second residual normalization layer is connected to the position encoding layer and the first attention module respectively, so as to superimpose the third sub-hidden layer vector with the second sub-hidden layer vector and output a fourth sub-hidden layer vector.

[0144] It should also be noted that, in this embodiment, the step of outputting the hidden layer vector based on the representation vector and the word vector also includes: executing step S445: inputting the fourth sub-hidden layer vector into the feedforward layer and the third residual normalization layer in sequence, and outputting the hidden layer vector.

[0145] Continue to refer Figure 7 , execute step S45 to perform linear transformation on the hidden layer vector.

[0146] The hidden layer vector is linearly transformed, so as to transform the dimension of the target output sequence to a desired dimension. Specifically, the hidden layer vector is input into the linear transformation layer for linear transformation.

[0147] Continue to refer Figure 7 , execute step S46, perform a linear transformation on the hidden layer vector, perform a third normalization process, and output the conditional probability corresponding to each phoneme.

[0148] Specifically, in this embodiment, the softmax activation function is used to perform the third normalization process to obtain the conditional probability corresponding to each phoneme. Assuming that the input X = [x1, x2, ..., xD] has a dimension of D, then the output Y = [y1, y2, ..., yD] of the softmax activation function is also D-dimensional, where

[0149]

[0150] As an embodiment, the acoustic model is applied to the field of oral evaluation; the unlabeled training data and the labeled training data both include: pronunciation data of native speakers of the target language, pronunciation data of second language learners, and native language pronunciation data of second language learners.

[0151] Therefore, in this embodiment, the conditional probability of the standard phoneme and the conditional probability corresponding to the erroneous phoneme are obtained. By additionally outputting the conditional probability of the erroneous phoneme, diversified feature selection is provided for subsequent evaluation tasks.

[0152] Correspondingly, the present invention also provides an acoustic model building system. Fig. 9 It is a functional block diagram of an embodiment of the acoustic model building system of the present invention.

[0153] refer to Fig. 9 In this embodiment, the acoustic model construction system includes: a training data acquisition module 510, which is used to acquire unlabeled training data and labeled training data, the labeled training data includes speech data and corresponding annotated text, and the unlabeled training data and the speech data constitute a training speech set; a model construction module 520, which is used to construct an acoustic model, and the acoustic model is used to obtain the conditional probability corresponding to the phoneme; the acoustic model includes an audio representation model, and the audio representation model is used to obtain a representation vector based on the input speech; a pre-training module 530, which is used to use the training speech set to pre-train the audio representation model based on the correlation between frames; an acoustic model training module 540, which is used to use the labeled training data to train the acoustic model based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data.

[0154] The acoustic model constructed by the model construction module 520 includes an audio representation model; accordingly, the pre-training module 530 pre-trains the audio representation model based on the training speech set and the correlation between frames, so that not only the audio representation model can learn the correlation of the audio context, but also the pronunciation representation learning using the unlabeled training speech data can effectively avoid the problem of "sparse" context data, and correspondingly expands the audio representation model's ability to express phonemes; and the acoustic model training module 540 uses the labeled training data to train the acoustic model based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data, so that the acoustic model can learn the correlation of the text context and combine the pronunciation representation with the text context for learning, and because the pre-training module 530 uses the unlabeled training speech data alone to train the audio representation model before the acoustic model training module 540 trains the acoustic model, accordingly, the acoustic model training module 540 requires less labeled training data when training the acoustic model.

[0155] In summary, the acoustic model construction system provided in this embodiment avoids the frame independence assumption in the traditional modeling method, and can also better grasp the contextual information (Contextual Information) of the audio, thereby improving the training effect of the acoustic model, and correspondingly improving the speech recognition performance of the acoustic model, and reducing the dependence on the language model.

[0156] The training data acquisition module 510 is used to acquire unlabeled training data and labeled training data, wherein the labeled training data includes speech data and corresponding annotated text, and the unlabeled training data and the speech data constitute a training speech set.

[0157] Unlabeled training data and labeled training data are used to form training samples for training acoustic models.

[0158] Among them, unlabeled training data does not need to be manually labeled, so unlabeled training data is easier to obtain than labeled training data, so that a large amount of unlabeled training data can be obtained, which is beneficial to avoid the problem of insufficient "sparse" context data, and accordingly ensures the training effect of the acoustic model while also helping to reduce costs.

[0159] Specifically, in this embodiment, the acoustic model is applied to the field of spoken language assessment. Spoken language assessment refers to the overall evaluation of a person's spoken language level using speech recognition technology given text and audio.

[0160] Therefore, the unlabeled training data is unlabeled spoken pronunciation data. The labeled training data includes spoken pronunciation data and corresponding annotated text.

[0161] In this embodiment, the unlabeled training data and the labeled training data both include: pronunciation data of native speakers of the target language, pronunciation data of second language learners, and native language pronunciation data of second language learners.

[0162] Among them, the pronunciation data of the target language native speaker refers to the pronunciation data of a person whose native language is the target language; the pronunciation data of the second language learner refers to the pronunciation data of a person learning the target language speaking the target language; and the second language learner native language pronunciation data refers to the pronunciation data of a person learning the target language speaking his or her native language.

[0163] For example, in the scenario of Chinese people learning English, the pronunciation data of the target language native speaker refers to the pronunciation data of English spoken by a native English speaker, the pronunciation data of the second language learner refers to the pronunciation data of Chinese people speaking English, and the second language learner native language pronunciation data refers to the pronunciation data of Chinese people speaking Chinese.

[0164] By making the unlabeled training data and the labeled training data include the pronunciation data of the target language native speakers, the pronunciation data of the second language learners, and the native language pronunciation data of the second language learners, in the subsequent pre-training of the audio representation model based on the training speech set and the correlation between each frame and all frames in the speech, similar pronunciation phonemes between different languages ​​will share representation vectors, and language-specific pronunciation phonemes will independently form different representation vectors. These representation vectors will be combined into a "cross-language", rich and huge representation space, thereby expanding the model's ability to express phonemes, so that the pronunciation representation and phonemes have a high correlation.

[0165] Moreover, in the process of pre-training the audio representation model by the pre-training module 530, the pronunciation of second language learners that is biased by their mother tongue can be learned, and the phoneme conditional probability of the erroneous phonemes can be output accordingly, which can provide diversified feature selection for oral evaluation tasks.

[0166] The model building module 520 is used to build an acoustic model, and the acoustic model is used to obtain the conditional probability corresponding to the phoneme; the acoustic model includes an audio representation (Speech Representation) model, and the audio representation model is used to obtain a representation vector based on the input speech.

[0167] The audio representation model is used to represent the input audio, thereby representing the characteristics of the phonemes, the differences between the phonemes, etc.

[0168] The acoustic model includes an audio representation model, so that the pre-training module 530 can train the audio representation model using unlabeled training speech data alone, and accordingly, less labeled training data is required when the training module trains the acoustic model. In addition, the subsequent use of unlabeled training speech data for pronunciation representation learning can effectively avoid the problem of "sparse" context data, and correspondingly expand the ability of the audio representation model to express phonemes.

[0169] Combined with reference Figure 2 In this embodiment, the audio representation model 10 includes: a convolution layer 101, which is used to perform convolution processing on the speech to obtain a hidden layer representation Z; a quantization layer 102, which is used to perform quantization processing on the hidden layer representation Z to obtain a quantized representation corresponding to the current frame; a mask converter 103, which obtains an audio context representation V2 based on the correlation between the hidden layer representation of each frame and the hidden layer representation of the frame within a preset range; a splicing layer 104, which is used to splice and combine the quantized representation and the audio context representation to obtain the representation vector.

[0170] Combined with reference Figure 3In this embodiment, the acoustic model also includes: a text embedding layer 60, which is used to obtain a word vector based on the annotated text; a position encoding layer 50, which is used to obtain a corresponding first position encoding vector based on the word vector corresponding to the annotated text; an overlay layer 70, which is used to overlay the word vector with the first position encoding vector; a decoding module 20, which is connected to the audio representation model 10 and the overlay layer 70; a linear transformation layer 30, which is connected to the decoding module 20; and a normalization layer 40, which is connected to the linear transformation layer 30, for outputting the conditional probability corresponding to the phoneme.

[0171] Combined with reference Figure 4 The decoding module 20 includes: a first attention module 21, connected to the superposition layer 70; a first residual normalization layer 22, connected to the superposition layer 70 and the first attention module 21; a second attention module 23, connected to the first residual normalization layer 22; a second residual normalization layer 24, connected to the second attention module 23 and the first residual normalization layer 22; a feedforward layer 25, connected to the second residual normalization layer 24; a third residual normalization layer 26, connected to the feedforward layer 25 and the second residual normalization layer 24.

[0172] It should be noted that, in this embodiment, the acoustic model construction system further includes: a phoneme set acquisition module (not shown), which is used to obtain a phoneme set, and the phoneme set includes standard phonemes and erroneous phonemes. Therefore, in this embodiment, the model construction module 520 uses language based on the phoneme set to construct an acoustic model, and the acoustic model is used to obtain the conditional probability corresponding to the standard phoneme and the conditional probability corresponding to the erroneous phoneme, which can provide diversified feature selection for subsequent oral evaluation tasks.

[0173] The pre-training module 530 uses the training speech set to pre-train the audio representation model based on the correlation between frames, so that not only can the audio representation model learn the correlation of the audio context, but also use the unlabeled training speech data to perform pronunciation representation learning, thereby reducing the dependence on labeled data during the pre-training process, and can effectively avoid the problem of "sparse" context data, and correspondingly expand the audio representation model's ability to express phonemes.

[0174] Accordingly, in this embodiment, self-supervised representation learning is performed on the audio representation model.

[0175] As an embodiment, the pre-training module 530 inputs the speech in the training speech set into the convolutional neural network (ie, the convolutional layer 101) for convolution processing to obtain the hidden layer representation Z.

[0176] Specifically, the convolution layer 101 is a multi-layer one-dimensional convolution layer.

[0177] As an embodiment, the convolution kernels of the convolution layer 101 are (10, 3, 3, 3, 3, 2, 2), the strides are (5, 2, 2, 2, 2, 2, 2), and the number of channels is 512. The convolution layer 101 outputs the hidden layer representation Z, and the receptive field of the hidden layer representation Z is 400 sampling points (16k sampling, i.e., 25ms), and the time span is 320 sampling points (16k sampling, i.e., 20ms), which is equivalent to the frame length and frame shift of traditional speech acoustic features.

[0178] In this embodiment, the pre-training module 530 is also used to input the hidden layer representation into the quantization layer 102 (such as Figure 2 Quantize the current frame to obtain the quantized representation of the current frame. Quantize the current frame to learn the representation vector, and then obtain the corresponding phoneme representation from the quantization code table.

[0179] Specifically, in this embodiment, the quantization layer 102 quantizes the hidden layer representation, including: mapping the hidden layer representation corresponding to each phoneme to a quantization codebook, so that the input vector can be replaced by the index of the codeword in the codebook that best matches the input vector for transmission and storage, so that the corresponding audio representation vector can be extracted by only performing a simple table lookup operation using the index.

[0180] As an example, assuming that the dimension of the hidden layer representation Z is D, and there are V quantization codebooks E = [e1, e2, ..., eV], then the distance (logit distance) between the input vector Z and any entry is l v The definition is as follows:

[0181] 1 v =log[1+exp(-Z T e v )],v=1,2,...,V (I)

[0182] The logit distance is defined to calculate which entry the input vector Z is closer to, so as to select the corresponding codebook entry.

[0183] In the specific implementation, the D dimensions of the hidden layer representation Z are usually divided into G groups, and the quantization codebook is also G groups, each group includes V entries, the encoding dimension is D / G, and the logit distance l between any group of inputs and any entry in the same group g,v The calculation method is similar. In this embodiment, the selection process of the coding entry P g,v Calculated by formula (II):

[0184]

[0185] Where τ is a non-negative temperature constant (usually set to 2, with a decay factor of 0.999995 as the number of iterations increases), n v is Gumbel random noise, through the formula n v =-log(-log(u)), calculated from u to U(0,1), where u is uniformly distributed in the interval (0,1), and k∈[1,V].

[0186] In which, by introducing random noise n v , which makes the training more stable and thus improves the robustness.

[0187] In this embodiment, the method of using Gumbel Softmax to make the selection process of coding items differentiable is used as an example for description.

[0188] In this embodiment, in the specific training process, the forward calculation selects the corresponding codebook entry according to argmaxpg,v, and the gradient back propagation is strictly based on p g,v To calculate. When the training is finished and the module is used, n v Set to 0.

[0189] The pre-training module 530 is also used to input the hidden layer representation into the mask converter 103 (such as Figure 2 As shown), the audio context representation is obtained based on the correlation between the hidden layer representation of each frame and the hidden layer representation of the frames within a preset range.

[0190] Based on the correlation between the hidden layer representation of each frame and the hidden layer representation of the frames within a preset range, the audio representation model can learn the correlation between the current frame and other frames within the preset range, that is, learn the correlation of the audio context.

[0191] In this embodiment, the mask converter 103 includes one or more layers of sub-converters (not shown) connected in sequence. In this embodiment, the sub-converter includes: a weighted hidden layer representation acquirer, which is used to calculate the weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frame within a preset range, and output the weighted hidden layer representation after the weight relationship is superimposed with the hidden layer representation of the current frame; a first normalization layer, which is used to perform a first normalization process on the weighted hidden layer representation and output a first normalized representation; a feedforward network, which is connected to the first normalization layer and is used to process the first normalized representation; a superimposer, which is used to superimpose the output of the feedforward network and the first normalized representation; a second normalization layer, which is used to perform a second normalization process on the output of the superimposer and output a second normalized representation.

[0192] Specifically, the attention mechanism can be used to calculate the weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frames within the preset range. More specifically, the multi-head attention mechanism can be used to calculate the weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frames within the preset range, so that the weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frames within the preset range can be paid attention to from different dimensions.

[0193] The first normalization layer performs a first normalization process on the weighted representation, thereby suppressing the vanishing of the deep network gradient and accelerating the convergence.

[0194] The second normalization layer performs the second normalization process to suppress the vanishing of the deep network gradient and accelerate the convergence.

[0195] In a specific embodiment, the last of the second normalized representations output by one or more layers of sequentially connected sub-converters is used as the audio context representation.

[0196] In this embodiment, the pre-training module 530 further includes a loss function acquisition unit (not shown) for obtaining a loss function based on the quantization representation and the audio context representation.

[0197] The loss function is used to measure the gap between the output of the model and the true value, and then based on the loss function, the parameters of the audio representation model can be adjusted, and the audio representation model can be optimized accordingly to obtain a trained audio representation model.

[0198] In this embodiment, the audio representation model performs self-supervised representation learning, and the loss function is shown in formula (III):

[0199]

[0200] Among them, L m is the contrast loss function, L d is the diversity loss function, and α is a hyperparameter. In this embodiment, α is set to 0.1.

[0201] In this embodiment, any time span is selected as the starting position with a certain probability p (usually set to 0.065), and then the subsequent consecutive M (usually set to 10) time spans are masked (replaced with a common fixed vector), the speech representation output corresponding to the center position of the mask is ct, and the corresponding selected quantization entry is qt, which together with K (usually set to 100) quantization entries randomly selected from other mask segments constitute a set Qt.

[0202] In this embodiment, one purpose of performing self-supervised representation learning on the audio representation model is to make the similarity between ct and qt as large as possible as possible from the similarity between other quantized items in ct and Qt, which is called contrastive loss and is calculated using the contrastive loss function. Another purpose is to make the quantized items as diverse as possible, that is, p g,v The entropy is the largest, which is called diversity loss, that is, it is calculated using the diversity loss function.

[0203] In this embodiment, the contrast loss function L m Calculated by formula (IV):

[0204]

[0205] Wherein, K is another non-negative temperature constant (usually set to 0.1), sim(a, b) is a similarity function, and the similarity function is usually calculated using cosine distance, that is, in this embodiment, the formula sim(a, b)=a is used. T b / ||a||||b||calculate the similarity function.

[0206] In this embodiment, the diversity loss function L d Calculated by formula (V):

[0207]

[0208] Among them, g is the number of groups of items, v is the number of items in each group, H(.) is the information entropy,

[0209] In this embodiment, the pre-training module 530 further includes: a parameter optimization unit (not shown), which is used to adjust the parameters of the audio representation model according to the loss function, thereby optimizing the parameters of the audio representation model.

[0210] Specifically, in this embodiment, the parameter optimization unit calculates the loss according to the loss function, calculates the gradient of the loss function, and updates the parameters of the audio representation model through back propagation.

[0211] The audio representation model is used to perform data fusion on the quantized representation and the audio context representation, and output a representation vector corresponding to each frame.

[0212] In this embodiment, the audio representation model performs data fusion on the quantization representation and the audio context representation, including: concatenating and combining the quantization representation and the audio context representation to obtain the representation vector.

[0213] It should be noted that, in this embodiment, the pre-training module 530 is also used to input the hidden layer representation into the first position encoding unit for first position encoding processing.

[0214] The acoustic model training module 540 is used to train the acoustic model using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the words and the representation vectors corresponding to the speech data.

[0215] After the pre-training module 530 pre-trains the audio representation model, the acoustic model training module 540 uses the labeled training data to train the acoustic model based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data, so that the acoustic model can learn the correlation of the text context and combine the pronunciation representation and the text context for learning. Moreover, since the audio representation model is trained using unlabeled training speech data alone before training the acoustic model, correspondingly, less labeled training data is required when training the acoustic model.

[0216] In summary, this embodiment avoids the frame independence assumption in the traditional modeling method, and can also better grasp the contextual information of the audio, thereby improving the training effect of the acoustic model, correspondingly improving the speech recognition performance of the acoustic model, and reducing the dependence on the language model.

[0217] In this embodiment, the acoustic model is applied to the field of oral evaluation; the labeled training data includes pronunciation data of second language learners; and the acoustic model is trained using the pronunciation data of second language learners.

[0218] The acoustic model training module 540 trains the acoustic model by using the pronunciation data of the second language learner, so that the training data is richer, and when the acoustic model constructed by this embodiment is used for speech recognition or oral evaluation, it can better handle the situation where the actual user's pronunciation is biased by the native language.

[0219] In this embodiment, the labeled training speech data also includes: pronunciation data of native speakers of the target language and pronunciation data of native speakers of the second language learners.

[0220] As an embodiment, the acoustic model training module 540 trains the acoustic model based on the pronunciation data of the target language native speaker, the pronunciation data of the second language learner, and the native language pronunciation data of the second language learner.

[0221] In addition to using the pronunciation data of the second language learner, the acoustic model training module 540 also uses the pronunciation data of the native speaker of the target language and the native language pronunciation data of the second language learner to train the acoustic model, thereby enriching the training data. When the acoustic model constructed in this embodiment is used for speech recognition or oral evaluation, it can better handle the situation where the actual user pronunciation is biased by the native language, and can take into account the erroneous phonemes in all the training data.

[0222] In other embodiments, the acoustic model training module may further include: a first sub-training unit, used to perform a first sub-training on the acoustic model based on the pronunciation data of the native speaker of the target language and the native pronunciation data of the second language learner; and a second sub-training unit, used to perform a second sub-training on the acoustic model based on the pronunciation data of the second language learner after the first sub-training.

[0223] By performing a first sub-training on the acoustic model using the pronunciation data of the target language native speakers and the native language pronunciation data of the second language learners, and then performing a second sub-training on the acoustic model using the pronunciation data of the second language learners, it is possible to specifically train the pronunciation data of the second language learners in a specific country or region, which is more targeted and can improve the training effect.

[0224] Specifically, the acoustic model training module 540 inputs the speech data of the labeled training data into the audio representation model 30 to obtain the representation vector corresponding to each frame.

[0225] It can be seen from the above records that the pre-training module 530 uses the training speech set to pre-train the audio representation model based on the correlation between frames. Therefore, the audio representation model can learn the correlation of the audio context. Accordingly, the representation vectors corresponding to each frame obtained by the audio representation model 30 can well represent the correlation between the audio contexts.

[0226] Specifically, the acoustic model training module 540 is also used to input the annotated text into the text embedding layer 60 for text embedding processing to obtain a word vector. The annotated text is embedded to obtain a matrix consisting of word vectors corresponding to each phoneme in the annotated text, so that when using the acoustic model, the corresponding vector can be found based on the word.

[0227] The acoustic model training module 540 is also used to input the word vector into the second position encoding unit to perform second position encoding processing on the word vector, thereby combining the word order information and the word vector to form a new representation, and then being able to learn the word order information.

[0228] Specifically, in this embodiment, the position encoding layer 50 obtains the corresponding second position encoding vector based on the word vector corresponding to the annotated text; and the superposition layer 70 superimposes the word vector with the corresponding second position encoding vector.

[0229] The acoustic model training module 540 is also used to input the representation vector and the word vector into the decoding module 20. The decoding module 20 outputs a hidden layer vector based on the representation vector and the word vector. The hidden layer vector learns the correlation between the word vectors and the word vectors, and the correlation between the word vectors and the representation vector.

[0230] The acoustic model training module 540 is also used to input the hidden layer vector into the linear transformation layer for linear transformation. The acoustic model training module 540 is also used to input the hidden layer vector into the normalization layer for third normalization processing, and output the conditional probability corresponding to each phoneme.

[0231] Specifically, in this embodiment, the softmax activation function is used to perform the third normalization process to obtain the conditional probability corresponding to each phoneme. Assuming that the input X = [x1, x2, ..., xD] has a dimension of D, then the output Y = [y1, y2, ..., yD] of the softmax activation function is also D-dimensional, where

[0232]

[0233] As an embodiment, the acoustic model is applied to the field of oral evaluation; the unlabeled training data and the labeled training data both include: pronunciation data of native speakers of the target language, pronunciation data of second language learners, and native language pronunciation data of second language learners.

[0234] Therefore, in this embodiment, based on the target output sequence, the conditional probability of the standard phoneme and the conditional probability corresponding to the erroneous phoneme are obtained. By additionally outputting the conditional probability of the erroneous phoneme, diversified feature selection is provided for subsequent evaluation tasks.

[0235] Correspondingly, an embodiment of the present invention also provides a speech recognition method. Fig.10 It is a flow chart of an embodiment of a speech recognition method of the present invention.

[0236] In this embodiment, the speech recognition method includes the following basic steps:

[0237] Step S10: Acquire voice data to be recognized;

[0238] Step S20: inputting the speech data into the acoustic model constructed by the acoustic model construction method provided by the embodiment of the present invention to obtain the phoneme conditional probability;

[0239] Step S30: Obtaining speech recognition results based on phoneme conditional probability.

[0240] It can be seen from the above records that in the acoustic model construction method provided by the embodiment of the present invention, the audio representation model is pre-trained based on the training speech set and the correlation between frames, so that not only the audio representation model can learn the correlation of the audio context, but also the pronunciation representation learning using unlabeled training speech data can effectively avoid the problem of "sparse" context data, and correspondingly expand the audio representation model's ability to express phonemes; and, after the audio representation model is pre-trained, the acoustic model is trained using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the representation vectors corresponding to the words and the speech data, so that the acoustic model can learn the correlation of the text context and combine the pronunciation representation with the text context for learning, and because the audio representation model is trained using the unlabeled training speech data alone before the acoustic model is trained, correspondingly, less labeled training data is required when training the acoustic model.

[0241] Accordingly, when constructing an acoustic model, the embodiment of the present invention avoids the frame independence assumption in the traditional modeling method, and can also better grasp the contextual information of the audio, thereby improving the training effect of the acoustic model, and correspondingly improving the speech recognition performance of the acoustic model, and reducing the dependence on the language model, thereby improving the accuracy and recognition effect of speech recognition.

[0242] In order to solve the above-mentioned problem, an embodiment of the present invention further provides a device, which can implement the acoustic model building method provided by the embodiment of the present invention by loading the above-mentioned acoustic model building method in the form of a program.

[0243] An optional hardware structure of the device provided by the embodiment of the present invention can be as follows Fig.11 As shown, it includes: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04.

[0244] In the embodiment of the present invention, the number of the processor 01 , the communication interface 02 , the memory 03 , and the communication bus 04 is at least one, and the processor 01 , the communication interface 02 , and the memory 03 communicate with each other through the communication bus 04 .

[0245] Optionally, the communication interface 02 may be an interface of a communication module for network communication, such as an interface of a GSM module.

[0246] Optionally, the processor 01 may be a central processing unit CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0247] Optionally, the memory 03 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0248] The memory 03 stores one or more computer instructions, and the one or more computer instructions are executed by the processor 01 to implement the acoustic model building method provided in the embodiment of the present invention.

[0249] It should be noted that the above-mentioned implementation terminal device may also include other devices (not shown) that may not be necessary for understanding the contents disclosed in the embodiments of the present invention; given that these other devices may not be necessary for understanding the contents disclosed in the embodiments of the present invention, the embodiments of the present invention will not introduce them one by one.

[0250] Correspondingly, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the acoustic model construction method described in the embodiment of the present invention.

[0251] The storage medium is a computer-readable storage medium, and the storage medium can be a read-only memory (ROM), a random access memory (RAM), a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk, or other medium that can store program codes.

[0252] The above-mentioned embodiments of the present invention are combinations of elements and features of the present invention. Unless otherwise mentioned, the elements or features may be considered as optional. Each element or feature may be put into practice without being combined with other elements or features. In addition, the embodiments of the present invention may be constructed by combining some elements and / or features. The order of operations described in the embodiments of the present invention may be rearranged. Some constructions of any one embodiment may be included in another embodiment, and may be replaced by the corresponding construction of another embodiment. It is obvious to those skilled in the art that claims that do not have a clear reference relationship to each other in the attached claims may be combined into embodiments of the present invention, or may be included as new claims in the amendment after submitting this application.

[0253] The embodiments of the present invention can be implemented by various means such as hardware, firmware, software or a combination thereof. In a hardware configuration, the method according to the exemplary embodiment of the present invention can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0254] In a firmware or software configuration, the embodiments of the present invention may be implemented in the form of modules, procedures, functions, etc. The software code may be stored in a memory unit and executed by a processor. The memory unit is located inside or outside the processor and may send data to and receive data from the processor via various known means.

[0255] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A method for constructing an acoustic model, characterized in that: include: Obtaining unlabeled training data and labeled training data, wherein the labeled training data includes speech data and corresponding annotated text, and the unlabeled training data and the speech data constitute a training speech set; Constructing an acoustic model, the acoustic model being used to obtain conditional probabilities corresponding to phonemes; the acoustic model comprising an audio representation model, the audio representation model being used to obtain a representation vector based on input speech; Using the training speech set, based on the correlation between frames, pre-training the audio representation model; After pre-training the audio representation model, the acoustic model is trained using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the words and the representation vectors corresponding to the speech data.

2. The acoustic model construction method according to claim 1, characterized in that: The acoustic model is applied to the field of oral evaluation; the labeled training data includes pronunciation data of second language learners; The acoustic model is trained using the pronunciation data of second language learners.

3. The acoustic model construction method according to claim 2, characterized in that: The labeled training speech data also includes: pronunciation data of a native speaker of the target language and pronunciation data of a native speaker of a second language learner; The training of the acoustic model comprises: performing a first sub-training on the acoustic model based on the pronunciation data of the target language native speaker and the native pronunciation data of the second language learner; after the first sub-training, performing a second sub-training on the acoustic model based on the pronunciation data of the second language learner; or, The acoustic model is trained based on the target language native speaker pronunciation data, the second language learner pronunciation data, and the second language learner native speaker pronunciation data.

4. The acoustic model construction method according to claim 1, characterized in that: The acoustic model construction method further includes: before constructing the acoustic model, obtaining a phoneme set, wherein the phoneme set includes standard phonemes and erroneous phonemes; Based on the phoneme set, an acoustic model is constructed, and the acoustic model is used to obtain the conditional probability corresponding to the standard phoneme and the conditional probability corresponding to the erroneous phoneme.

5. The acoustic model construction method according to claim 1, characterized in that: The step of pre-training the audio representation model using the training speech set based on the correlation between frames includes: Performing convolution processing on the speech in the training speech set to obtain a hidden layer representation; Quantizing the hidden layer representation to obtain a quantized representation corresponding to the current frame; Obtaining an audio context representation based on a correlation between a hidden layer representation of each frame and a hidden layer representation of a frame within a preset range; Obtaining a loss function based on the quantized representation and the audio context representation; Adjusting parameters of the audio representation model according to the loss function; The audio representation model is used to perform data fusion on the quantized representation and the audio context representation, and output a representation vector corresponding to each frame.

6. The acoustic model construction method according to claim 5, characterized in that: Performing convolution processing on the speech in the training speech set includes: inputting the speech in the training speech set into a convolutional neural network for convolution processing to obtain the hidden layer representation.

7. The acoustic model construction method according to claim 5, characterized in that: Obtaining the audio context representation includes performing one or more transformation processes in sequence; The steps of the conversion process include: Calculate the weight relationship between the hidden layer representation of the current frame and the hidden layer representation of the frames within a preset range, and output the weighted hidden layer representation after the weight is superimposed on the hidden layer representation of the current frame; Performing a first normalization process on the weighted hidden layer representation, and outputting a first normalized representation; Input the first normalized representation into the feedforward network for processing; Superimposing the output of the feedforward network with the first normalized representation; After superimposing the output of the feedforward network with the first normalized representation, a second normalization process is performed to output a second normalized representation.

8. The acoustic model construction method according to any one of claims 5 to 7, characterized in that: The step of pre-training the audio representation model using the training speech set based on the correlation between frames also includes: after obtaining the hidden layer representation and before obtaining the correlation between the hidden layer representation of each frame and the hidden layer representation of frames within a preset range, performing a first position encoding process on the hidden layer representation.

9. The acoustic model construction method according to claim 5, characterized in that: The data fusion of the quantization representation and the audio context representation includes: concatenating and combining the quantization representation and the audio context representation to obtain the representation vector.

10. The acoustic model construction method according to claim 1, characterized in that: The step of training the acoustic model using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the words and the representation vectors corresponding to the speech data comprises: Inputting the speech data with the labeled training data into the audio representation model to obtain a representation vector; Based on the annotated text, obtain word vectors; Based on the representation vector and the word vector, output a hidden layer vector, wherein the hidden layer vector learns the correlation between the word vectors and the word vectors, and the correlation between the word vectors and the representation vector; Performing a linear transformation on the hidden layer vector; After the hidden layer vector is linearly transformed, a third normalization process is performed to output the conditional probability corresponding to each phoneme.

11. The acoustic model construction method according to claim 10, characterized in that: The step of training the acoustic model also includes: after obtaining the word vector based on the annotated text, and before outputting the hidden layer vector based on the representation vector and the word vector, performing a second position encoding process on the word vector.

12. An acoustic model building system, characterized in that: include: A training data acquisition module, used to acquire unlabeled training data and labeled training data, wherein the labeled training data includes speech data and corresponding annotated text, and the unlabeled training data and the speech data constitute a training speech set; A model building module, used to build an acoustic model, the acoustic model is used to obtain conditional probabilities corresponding to phonemes; the acoustic model includes an audio representation model, the audio representation model is used to obtain a representation vector based on input speech; A pre-training module, used to pre-train the audio representation model based on the correlation between frames using the training speech set; The acoustic model training module is used to train the acoustic model using the labeled training data based on the annotated text, the correlation between words in the annotated text, and the correlation between the words and the representation vectors corresponding to the speech data.

13. A speech recognition method, characterized in that: include: Acquire voice data to be recognized; Inputting the speech data into an acoustic model constructed by using the acoustic model construction method according to any one of claims 1 to 11 to obtain a phoneme conditional probability; Based on the phoneme conditional probability, a speech recognition result is obtained.

14. A device, characterized in that It includes at least one memory and at least one processor, wherein the memory stores one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the acoustic model construction method as described in any one of claims 1-11.

15. A storage medium, characterized in that: The storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the acoustic model construction method as described in any one of claims 1-11.

Images