An English auxiliary pronunciation training method and system based on multi-task learning

Abstract

The present invention belongs to the field of speech recognition and speech evaluation, and provides an English auxiliary pronunciation training method and system based on multi-task learning, which obtains English pronunciation audio to be evaluated; pre-processes the English pronunciation audio, extracts audio features, and obtains a deep feature vector of English pronunciation; uses a pre-trained automatic pronunciation evaluation model to perform multi-granularity evaluation on the deep feature vector to determine pronunciation evaluation results at the phoneme level, word level, and sentence level; uses a pre-trained pronunciation detection and diagnosis model to perform phoneme recognition on the deep feature vector and calculate the phoneme error rate of English pronunciation; in the training process of the automatic pronunciation evaluation model and the pronunciation detection and diagnosis model, jointly optimizes the loss of automatic pronunciation evaluation and wrong pronunciation detection and diagnosis. The present invention makes full use of the correlation between the automatic pronunciation evaluation and wrong pronunciation detection and diagnosis tasks, and improves the accuracy of pronunciation evaluation.

Description

Technical Field

[0001] The present invention belongs to the field of speech recognition and speech evaluation, and in particular relates to an English auxiliary pronunciation training method and system based on multi-task learning. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] Computer-aided Pronunciation Training (CAPT) technology is a tool that uses computers and software technology to help second language learners improve their pronunciation. The CAPT system can score and detect errors in learners' pronunciation, thereby helping learners correct their pronunciation errors. The CAPT system mainly consists of two tasks, namely Automatic Pronunciation Assessment (APA) and Mispronunciation Detection and Diagnosis (MDD).

[0004] As branch tasks of CAPT, APA and MDD also use similar methods, such as methods based on the Goodness of Pronunciation (GOP) feature, methods based on pre-trained self-supervised learning models, and methods based on deep features. However, the method based on the GOP feature needs to first input the acoustic features into the acoustic model based on the deep neural network-hidden Markov model to obtain the log-likelihood value of the target phoneme. Then the GOP feature is input into the neural network for learning to obtain the final evaluation score. Using the method based on the pre-trained self-supervised learning model, the acoustic features can be input into the fine-tuned pre-trained self-supervised learning model to obtain the deep features, and then the deep features are input into the scoring head to predict the score. The Pearson correlation coefficient of the method based on the pre-trained self-supervised learning model on the APA task is higher than that of the method based on the GOP feature, and because the scale of the data set used for speech evaluation is generally small, the method based on the pre-trained self-supervised learning model can solve the problem of insufficient training data to a certain extent.

[0005] Many studies on automatic pronunciation evaluation focus on scoring the phonemes of pronunciation, or separately modeling the accuracy, stress, fluency, rhythm, and completeness of words. However, the acoustic features between phonemes, words, and sentences are interrelated, and modeling only a single granularity cannot fully utilize the rich feature information in a piece of speech. In practical applications, there is also a need for methods that can evaluate multiple aspects and multi-granularity scores with only one model.

[0006] Mispronunciation detection and diagnosis is a technique used to identify and analyze second language learners' pronunciation errors. Specifically, it is a phoneme recognition task that identifies learners' pronunciation errors at the phoneme level, including insertion errors, deletion errors, and substitution errors.

[0007] Automatic pronunciation assessment and mispronunciation detection and diagnosis have been treated as two independent tasks in previous studies. However, there is a linguistic correlation between these two tasks, and treating them separately cannot comprehensively assess learners' pronunciation levels. Summary of the invention

[0008] In order to solve the above problems, the present invention proposes an English auxiliary pronunciation training method and system based on multi-task learning. The present invention makes full use of the correlation between automatic pronunciation evaluation and wrong pronunciation detection and diagnosis tasks, and improves the accuracy of pronunciation evaluation.

[0009] According to some embodiments, the present invention adopts the following technical solutions:

[0010] An English auxiliary pronunciation training method based on multi-task learning comprises the following steps:

[0011] Get the English pronunciation audio to be evaluated;

[0012] Preprocessing the English pronunciation audio, extracting audio features, and obtaining a deep feature vector of the English pronunciation;

[0013] Use the pre-trained automatic pronunciation evaluation model to perform multi-granularity evaluation on the deep feature vector to determine the pronunciation evaluation results at the phoneme level, word level, and sentence level;

[0014] Use the pre-trained pronunciation detection and diagnosis model to perform phoneme recognition on the deep feature vectors and calculate the phoneme error rate of English pronunciation;

[0015] During the training process of the automatic pronunciation evaluation model and the pronunciation detection and diagnosis model, the losses of the automatic pronunciation evaluation and the mispronunciation detection and diagnosis are jointly optimized.

[0016] As an optional implementation, the English pronunciation audio is preprocessed, and the process of extracting audio features includes: extracting audio features using a trained self-supervised learning model, the self-supervised learning model includes a convolutional neural network encoder, a quantization module and a Transformer context network, the convolutional neural network encoder uses multiple convolutional layers to process the original audio signal to generate a low-level continuous feature representation, the quantization module uses a vector quantizer to convert the continuous feature representation into a discrete feature encoding for contrastive learning, and the Transformer context network uses a Transformer network to model the quantized features, generate context representations, capture long-distance dependencies, and generate acoustic deep features with semantic information.

[0017] As a further implementation method, the model is fine-tuned during the training process of the self-supervised learning model, a fully connected layer and a CTC decoder are added after the Transformer context network, and the training results are optimized using CTC loss.

[0018] As an optional implementation, the automatic pronunciation evaluation model includes a convolution-enhanced Transformer encoder and a multi-aspect and multi-granularity scoring module, wherein the convolution-enhanced Transformer encoder is used to encode deep feature vectors and extract scoring information shared at the phoneme level, word level and sentence level; the multi-aspect and multi-granularity scoring module is used to score each sentence at three granularities: phoneme level, word level and sentence level; the sentence level includes five aspects: completeness, accuracy, fluency, rhythm and total score; the word level includes three aspects: accuracy, stress and total score; and the phoneme level includes one aspect of accuracy.

[0019] As a further implementation, the convolution-enhanced Transformer encoder includes a convolution module, a self-attention module and a feedforward network connected in sequence, and each layer is connected by a residual structure; the convolution module is used to capture local context information of the deep feature vector to extract local information between phonemes and words; the self-attention module is used to capture global context information, assign different weights to the input information to model the sentence level; the feedforward network is used to perform nonlinear transformation and enhanced feature representation on the input features, and fuse the acoustic features passed through the convolution module and the self-attention module.

[0020] As a further implementation, the multi-aspect and multi-granularity scoring module is provided with a scoring head for each aspect of each granularity, and two multi-layer perceptrons are designed to adapt to different scoring heads, the first multi-layer perceptron is used for phoneme and word scoring heads, and the second multi-layer perceptron is used for sentence scoring heads, both multi-layer perceptrons are composed of a linear layer and a one-dimensional adaptive average pooling, and the output dimensions of the two are different.

[0021] As a further implementation method, the multi-aspect and multi-granularity scoring module pre-fills the phoneme-level and word-level scores. During the training process, the automatic pronunciation evaluation task is regarded as a regression task to predict the scores of each aspect at each granularity. The mean square error loss is used as the loss function, and when calculating the prediction score loss, the filled positions are ignored so that the filling does not participate in the loss calculation. The predicted scores at the phoneme level, word level, and sentence level are respectively subjected to mean square error operations with the true labels, and finally the sum is taken as the total loss of the automatic pronunciation evaluation task.

[0022] As an optional implementation, the pronunciation detection and diagnosis model includes a CTC decoder and a Transformer decoder, wherein:

[0023] The CTC decoder includes a fully connected layer and a CTC prediction head. The fully connected layer is a fully connected layer added when the self-supervised learning model is fine-tuned by extracting audio features, sharing the fine-tuned weights, and obtaining an unnormalized score; the CTC prediction head converts the unnormalized score into a logarithmic probability to obtain the probability of each frame of speech corresponding to a phoneme in the vocabulary;

[0024] The Transformer decoder includes multiple layers of decoder blocks, at least one layer of decoder blocks includes a masked multi-head self-attention module, a multi-head cross-attention module and a feedforward module connected in sequence.

[0025] As an alternative implementation, the loss of the CTC decoder is , the loss of the Transformer decoder is , the total loss of hybrid CTC / Attention joint training for:

[0026] ;

[0027] in, is the loss weight of the MDD task, , used to balance the loss of CTC and Attention;

[0028] In the task of mispronunciation detection and diagnosis, the total loss is used The results of optimizing both CTC decoder and Transformer decoder are simultaneously performed.

[0029] As an optional implementation, the process of jointly optimizing the loss of automatic pronunciation evaluation and mispronunciation detection and diagnosis includes: multi-task learning of the total loss for:

[0030] ;

[0031] in, is the loss weight for multi-task learning, , used to balance the losses of automatic pronunciation evaluation and mispronunciation detection and diagnosis.

[0032] An English auxiliary pronunciation training system based on multi-task learning, comprising:

[0033] A data acquisition module, configured to acquire English pronunciation audio to be evaluated;

[0034] An audio deep feature extraction module is configured to preprocess the English pronunciation audio, extract audio features, and obtain a deep feature vector of the English pronunciation;

[0035] An automatic pronunciation evaluation module, configured to perform multi-granularity evaluation on the deep feature vector using a pre-trained automatic pronunciation evaluation model to determine pronunciation evaluation results at the phoneme level, word level, and sentence level;

[0036] an incorrect pronunciation detection and diagnosis module, configured to perform phoneme recognition on the deep feature vector using a pre-trained pronunciation detection and diagnosis model and calculate a phoneme error rate of English pronunciation;

[0037] The joint optimization module is configured to jointly optimize the losses of automatic pronunciation evaluation and mispronunciation detection and diagnosis during the training process of the automatic pronunciation evaluation model and the pronunciation detection and diagnosis model.

[0038] A computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the steps in the above method are completed.

[0039] An electronic device comprises a memory and a processor, and computer instructions stored in the memory and executed on the processor. When the computer instructions are executed by the processor, the steps in the above method are completed.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] (1) The present invention uses a multi-task learning approach to jointly train automatic pronunciation evaluation and mispronunciation detection and diagnosis, solving the problem of not being able to fully utilize the language correlation between the two tasks.

[0042] (2) The present invention utilizes the transfer learning method to transfer the pre-trained self-supervised learning model used in the field of speech recognition to the field of speech evaluation, and uses the pre-trained self-supervised learning model as the backbone network for extracting acoustic deep features of the present invention, thereby solving the problem of scarce training data in the field of speech evaluation, and the Pearson correlation coefficient of the score is significantly improved compared with the method based on GOP features.

[0043] (3) For the automatic pronunciation evaluation task, the present invention designs a multi-faceted and multi-granular scoring standard, specifically three granularities: phoneme level, word level and sentence level; the phoneme level has one aspect: accuracy; the word level has three aspects: accuracy, stress and total score; the sentence level has five aspects: completeness, accuracy, fluency, rhythm and total score.

[0044] (4) For the enhanced feature extraction layer in the automatic pronunciation evaluation task, the present invention designs a convolution-enhanced Transformer module, which can obtain higher scores in multi-faceted and multi-granular speech evaluation compared with the bidirectional LSTM module.

[0045] (5) For the task of mispronunciation detection and diagnosis, the present invention designs a hybrid CTC / Attention architecture decoding module, which can improve the accuracy of phoneme recognition, reduce the phoneme error rate, and obtain higher indicators in the results of mispronunciation detection and diagnosis.

[0046] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0048] Figure 1 is a flow chart of an English auxiliary pronunciation training method based on multi-task learning in an embodiment;

[0049] Figure 2 is a framework diagram of an English auxiliary pronunciation training system based on multi-task learning in an embodiment;

[0050] Figure 3 yes Figure 2 Schematic diagram of the automatic pronunciation evaluation module in ;

[0051] Figure 4 yes Figure 2 Schematic diagram of the mispronunciation detection and diagnosis module in;

[0052] Figure 5is a schematic diagram of a self-supervised learning model fine-tuning process according to an embodiment;

[0053] Figure 6 is a flow chart of a convolution module of an embodiment;

[0054] Figure 7 is a schematic diagram of a self-attention module flow chart of an embodiment;

[0055] Figure 8 The feedforward module flow chart of one embodiment. DETAILED DESCRIPTION

[0056] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0057] It should be noted that the following detailed descriptions are all illustrative and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0058] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0059] In the absence of conflict, the embodiments in this application and the features in the embodiments may be combined with each other.

[0060] Embodiment 1

[0061] An English auxiliary pronunciation training method based on multi-task learning, such as Figure 1 As shown, the following steps are included:

[0062] Get the English pronunciation audio to be evaluated;

[0063] Preprocessing the English pronunciation audio, extracting audio features, and obtaining a deep feature vector of the English pronunciation;

[0064] Use the pre-trained automatic pronunciation evaluation model to perform multi-granularity evaluation on the deep feature vector to determine the pronunciation evaluation results at the phoneme level, word level, and sentence level;

[0065] Use the pre-trained pronunciation detection and diagnosis model to perform phoneme recognition on the deep feature vectors and calculate the phoneme error rate of English pronunciation;

[0066] During the training process of the automatic pronunciation evaluation model and the pronunciation detection and diagnosis model, the losses of the automatic pronunciation evaluation and the mispronunciation detection and diagnosis are jointly optimized.

[0067] The multi-task learning framework of this embodiment mainly includes three task links: acoustic deep feature extraction, automatic pronunciation evaluation, and incorrect pronunciation detection and diagnosis, which are introduced in detail below.

[0068] Acoustic deep feature extraction is to extract the acoustic deep features of English speech, which is mainly achieved through a fine-tuned pre-trained self-supervised learning model. The fine-tuned pre-trained self-supervised learning model is Wav2Vec2-robust.

[0069] In this embodiment, the model is trained with 63,000 hours of unlabeled speech data. Its structure can be mainly divided into three layers, namely, convolutional neural network encoder, quantization module, and Transformer context network. The model is fine-tuned on the TIMIT dataset to better extract phoneme features. Compared with the traditional phoneme feature extraction method, this method is trained directly on the original audio waveform, and the acoustic depth feature vector of the phoneme can be obtained without extracting MFCC or FBank features.

[0070] The automatic pronunciation evaluation task mainly processes acoustic deep features, captures shared scoring information, and then obtains multi-faceted and multi-granular scores of English speech through three scoring heads. This step uses a convolution-enhanced Transformer encoder to extract shared scoring information and inputs the scoring information into a multi-granular scoring head to predict multiple scores. Here, a multi-layer perceptron (MLP) is used as a scoring head to standardize the dimension of the scoring information.

[0071] The task of mispronunciation detection and diagnosis is to perform phoneme recognition and calculate the phoneme error rate of English pronunciation. This task uses CTC and attention joint decoding to perform phoneme recognition, which consists of a CTC decoder and a Transformer decoder.

[0072] In addition, this embodiment also uses a multi-task learning mechanism to jointly optimize the losses of automatic pronunciation evaluation and mispronunciation detection and diagnosis during the training phase. Using this mechanism, the relevant information between the two can be fully utilized.

[0073] The following is a detailed description of each step or task link:

[0074] First is acoustic feature extraction. This embodiment uses a fine-tuned pre-trained self-supervised learning model, Wav2Vec2-robust. Because the computer-assisted pronunciation training task has the problem of data scarcity, the idea of ​​transfer learning is used to obtain a model with rich data resources and huge scale from another field to adapt to the CAPT field, thereby solving the problem of insufficient training data in the CAPT field, and extracting high-quality acoustic features for various downstream tasks. And compared to the method of extracting GOP features, the method based on pre-trained self-supervised learning models is simpler to implement, and can be used by directly calling the model in the HuggingFace library. Moreover, when the extracted acoustic depth features are used for APA tasks, the Pearson correlation coefficient of the score is higher than that based on the GOP method.

[0075] The model of this embodiment directly operates on the original audio with a sampling rate of 16kHz and a length of L It is trained on CNNs without extracting MFCC or FBank features, and learns meaningful audio representations through self-supervised pre-training methods. The dimension is ,in B is the batch size batch_size, L is the audio length length. The architecture of the model includes three main modules: convolutional neural network encoder, quantization module and Transformer context network. The convolutional neural network encoder uses a series of convolutional layers with a feature dimension of 512 to process the original audio signal and generate low-level continuous feature representations. The quantization module uses a vector quantizer to convert continuous feature representations into discrete feature encodings for contrastive learning. The Transformer context network uses the Transformer network to model the quantized features and generate context representations; the Transformer network can capture long-distance dependencies and generate acoustic deep features with rich semantic information. ,in T is the time step length after the feature extractor has processed it, which is relative to the original length of the input audio L In general, the time step is reduced due to the stride of the convolution operation. D is the dimension of the acoustic depth feature vector; h is the abbreviation of hidden, which means the calculated intermediate state vector, which is used to distinguish it from the original vector x. The above process is shown in formula (1):

[0076] (1)

[0077] in, SSL It is a pre-trained self-supervised learning model.

[0078] In order to enhance the performance of the pre-trained self-supervised learning model in phoneme recognition, this embodiment performs supervised fine-tuning on the model. The fine-tuning dataset is the TIMIT dataset, which is a native language dataset containing recordings of 630 speakers, each of whom recorded 10 sentences, covering 8 American English dialects, and using 61 phoneme sets for speech transcription. In order to perform fine-tuning, it is only necessary to add a fully connected layer and a CTC decoder after the Transformer context network of the pre-trained self-supervised learning model, and use the CTC loss to optimize the training results. The flowchart of fine-tuning is shown in the figure. Figure 5 shown.

[0079] Acoustic deep features are used to perform automatic pronunciation evaluation for multi-faceted and multi-granular speech scoring. The automatic pronunciation evaluation task can further encode the acoustic deep features and predict the scores of a speech at the phoneme level, word level, and sentence level.

[0080] The model is mainly composed of a convolution-enhanced Transformer encoder and a multi-faceted and multi-granularity scoring module.

[0081] First, we introduce the convolution-enhanced Transformer encoder. This embodiment uses the convolution-enhanced Transformer encoder to extract the scoring information shared by the phoneme level, word level, and sentence level. Specifically, the encoder is used to extract the output of the pre-trained self-supervised learning model. Further encoding is performed. The encoder consists of a convolution module, a self-attention module, and a feedforward network, and each layer is connected by a residual structure.

[0082] The first module of the encoder is the convolution module. This embodiment is inspired by the Conformer structure in the field of automatic speech recognition and designs a convolution module. The convolution module can effectively capture acoustic depth features. Therefore, the introduction of the convolution module can better extract the local information between phonemes and words. The specific process of the convolution module is as follows: Figure 6 shown.

[0083] It can be obtained that for the acoustic depth feature , the calculation process in the convolution module is as follows,

[0084] (2)

[0085] (3)

[0086] (4)

[0087] in, Represents point-by-point convolution, with a kernel size of 1, a stride of 1, and an output dimension of 2 D , so after back, The dimension is ;go through After the activation function, the dimension is restored to ; Represents a one-dimensional depth convolution with a kernel size of 3, a stride of 1, and a padding of 1; represents batch normalization; Representation layer normalization; It is also point-by-point convolution, but does not change the vector dimension. After the convolution module, the acoustic features are obtained. .

[0088] Next is the self-attention module, and its process is as follows Figure 7 As shown. Because different parts of a sentence have different effects on the final scoring results, the parts with greater influence need to be given greater weights. Therefore, in order to better model the sentence level, it is necessary to introduce a self-attention mechanism to capture global context information and give different weights to the input information. For the structural design of the self-attention module, since the dataset used in this embodiment is SpeechOcean762 and the dataset size is small, for this reason, the self-attention module only designs one self-attention head. And in order to make the encoder more robust to changes in speech length, this embodiment also uses Dropout and residual mechanisms, which help to train and regularize deeper models.

[0089] For the acoustic features after the convolution module , the calculation process in the self-attention module is as follows:

[0090] (5)

[0091] (6)

[0092] in, Represents transpose(1,0). In order to match the network structure, The 0th and 1st dimensions of are exchanged, and the dimensions become , represents the self-attention head, Represents batch normalization. After the self-attention module, the acoustic features are obtained .

[0093] here The formula is shown in formula (7), assuming for ,

[0094] (7)

[0095] in ; are the linear transformation matrices of query, key, and value respectively; It is the characteristic dimension of the key value.

[0096] The last is the feedforward module, whose process is as follows Figure 8 As shown. The role of the feedforward module is to It performs nonlinear transformation and enhances feature representation. It can fuse acoustic features passed through the convolution module and the self-attention module, and can capture complex patterns and relationships in the input features. And the input features can be scaled through the feedforward module to reduce useless features, which helps the model better understand and utilize the input features in subsequent processing steps.

[0097] For the acoustic features after the self-attention module , the calculation process in the feedforward module is as follows:

[0098] (8)

[0099] (9)

[0100] in, Represents a linear layer, expanding the feature dimension to 2 D , which is 2048; then The linear layer scales the features to D / 8, that is, 256, recorded as H ; means transpose(1,0), The 0th and 1st dimensions of are exchanged, and the dimensions become After the feedforward module, the acoustic features are obtained .

[0101] Because the dataset is small, in order to avoid overfitting, this example only uses one layer of convolution-enhanced Transformer encoder. After the encoder, the acoustic enhancement feature is obtained .

[0102] This embodiment performs multi-faceted and multi-granular scoring based on the scoring label format of the SpeechOcean762 dataset. Each sentence in the dataset has three granularity scores, namely, phoneme level, word level, and sentence level. One sentence has five aspects of scoring: completeness, accuracy, fluency, rhythm, and total score; a sentence has multiple words, and each word has three aspects of scoring: accuracy, stress, and total score; each word has multiple phonemes, and each phoneme has one aspect of scoring: accuracy. In view of this note feature of the dataset, this embodiment designs multi-faceted and multi-granular scoring, so as to achieve comprehensive scoring of a sentence of speech.

[0103] According to the labeling of the above dataset, we can know that the length of the sentence-level score of a sentence is fixed, but the number of words and phonemes is variable, so the length of the phoneme-level and word-level scores is not fixed. Therefore, during training, the phoneme-level and word-level scores need to be padded when loading the score labels. Because the number of phonemes in each sentence does not exceed 50, the phoneme-level and word-level score labels are padded with -1, and the length after padding is 50. And the number of aspects of the phoneme-level, word-level and sentence-level scores are 1, 3, and 5 respectively, so the dimension of the padded phoneme-level score label phn_labels is , the dimension of word-level score label word_labels is , the dimension of sentence-level score label utt_labels is .

[0104] In order to match the acoustic feature dimensions with the scoring labels during training, this embodiment designs two multi-layer perceptrons (MLPs) to adapt to different scoring heads. The main function of the scoring head is to extract the acoustic enhancement features extracted from the convolution-enhanced Transformer encoder. The two MLPs are further processed and combined to generate the final score. In this embodiment, two MLPs are designed, MLP1 is used for the phoneme and word scoring head, and MLP2 is used for the sentence scoring head. Both MLPs consist of a linear layer and a one-dimensional adaptive average pooling (AdaptiveAvgPool1d). The difference is that they The dimension of is handled differently. To be consistent with the scoring label dimension of phonemes and words, the output dimension of AdaptiveAvgPool1d of MLP1 is set to 50, and the output dimension of AdaptiveAvgPool1d of MLP2 is set to 1.

[0105] One scoring head is set at the phoneme level: p_acc_head predicts phoneme accuracy; three scoring heads are set at the word level: word_acc_head predicts word accuracy, word_stress_head predicts word stress, and word_tot_head predicts word total score; five scoring heads are set at the sentence level: utt_com_head predicts sentence completeness, utt_acc_head predicts sentence accuracy, utt_pros_head predicts sentence rhythm, utt_flu_head predicts sentence fluency, and utt_tot_head predicts sentence total score.

[0106] During training, acoustic enhancement features are input into these 9 scoring heads respectively. Among them, the dimensions of the predicted scoring output by the phoneme and word scoring heads are , the dimension of the predicted score output by the sentence scoring head is Then the phoneme-level score phn_preds is squeezed (2) and the dimension becomes ; The outputs of the three word-level scoring heads are concatenated on the last dimension, set to word_preds, and its dimension is ; The outputs of the five sentence-level scoring heads are concat-operated on the last dimension, which is set to word_preds, and its dimension is .

[0107] This embodiment regards the automatic pronunciation evaluation task as a regression task, and lets the model predict the score of each aspect of each granularity. Therefore, in the automatic pronunciation evaluation task, this embodiment uses the mean square error (MSE) loss as the loss function, and its calculation formula is shown in formula (10):

[0108] (10)

[0109] in, N is the number of samples, is the true value of the sample label, is the predicted value of the sample label.

[0110] When calculating the prediction score loss, it is necessary to ignore the positions filled with -1 so that the filling does not participate in the loss calculation to avoid the resulting errors. The predicted scores at the phoneme level, word level, and sentence level are respectively calculated with the true labels for mean square error, and finally the sum is taken as the total loss of the automatic pronunciation evaluation task, as shown in formula (11):

[0111] (11)

[0112] in, represents the loss of phoneme score prediction, represents the loss of word score prediction, represents the loss of sentence rating prediction.

[0113] In order to evaluate the effect of the model proposed in this embodiment on the automatic pronunciation evaluation task, this embodiment uses the Pearson Correlation Coefficient (PCC) as an evaluation indicator to measure the correlation between the predicted score and the actual score. The calculation formula of PCC is shown in formula (12):

[0114] (12)

[0115] in, Indicates the first i scores, The corresponding first i scores, N is the total number of samples.

[0116] During the inference phase, an English speech is processed through acoustic deep feature extraction and a convolution-enhanced Transformer encoder to obtain acoustically enhanced features. The acoustically enhanced features are then input into each scoring head to obtain multi-faceted and multi-granular scores.

[0117] Multi-faceted and multi-granular automatic speech evaluation can input deep acoustic features and output predicted speech scores.

[0118] This embodiment uses acoustic deep features to detect and diagnose mispronunciation. Based on the hybrid CTC / Attention architecture, the CTC decoder and the Transformer decoder are used to decode the acoustic deep features. Perform joint decoding to predict the phonemes of a sentence of speech, and then calculate the phoneme error rate of a sentence of speech.

[0119] The advantage of CTC decoding is that it does not require explicit alignment information. The model can learn directly from unaligned input and target sequences and automatically find the optimal alignment, which is very useful when processing variable-length input and output sequences. However, because CTC relies on the conditional independence assumption, it may be difficult to capture long-distance dependencies for very long sequences. The Attention mechanism can more flexibly handle the importance of different inputs by dynamically assigning weights, capturing the correlation of different positions in the input sequence, and excels in modeling long-distance dependencies and processing contextual information. It can solve the problem of insufficient ability to model contextual relationships in CTC-based methods. Therefore, using a hybrid CTC / Attention architecture can obtain better alignment capabilities and improve the accuracy of model decoding. In addition, combining CTC and Attention mechanisms can achieve multi-task learning. The model can optimize CTC loss and Attention loss at the same time, thereby taking advantage of the advantages of both mechanisms and improving overall performance.

[0120] The CTC decoder consists of a fully connected layer and a CTC prediction head. This fully connected layer is the fully connected layer added when fine-tuning the pre-trained self-supervised learning model, which can share the fine-tuned weights. Acoustic depth features The fully connected layer outputs the unnormalized score ctc_logits, denoted as ,in V is the size of the phoneme vocabulary vocab_size.

[0121] Then To enter The main function of this layer is to Convert to logarithmic probability. This can improve numerical stability and avoid the problem of numerical underflow or overflow caused by directly calculating softmax and then taking the logarithm. The calculation formula is shown in formula (13):

[0122] (13)

[0123] in, Representation sequence Middle i elements, N Representation sequence The total number of elements in . Operation, get , denoted as , which represents the probability that each frame of speech corresponds to a phoneme in the vocabulary.

[0124] During the training phase, because the number of phonemes in each sentence is different, the phoneme labels ctc_labels used for CTC need to be padded with -1. After padding, the dimension of ctc_labels is , and ignore the padding elements when calculating the loss. The CTC branch uses ctc_loss as the loss function, with Calculate the loss with ctc_labels and record the loss as .

[0125] This embodiment uses the Transformer decoder as the Attention-based method, in which the structure of a decoder block consists of a masked multi-head self-attention module, a multi-head cross-attention module, and a feedforward module. For the best effect, this embodiment sets a 3-layer decoder block, and each layer of the masked multi-head self-attention module and the multi-head cross-attention module has 4 attention heads, and the dimension of the feedforward module is 2048. These modules are similar to the self-attention module and feedforward module structures in the convolution-enhanced Transformer encoder.

[0126] The input of the first layer decoder is the acoustic depth feature and phoneme labels phn_labels (denoted as ), first of all, Word embedding is performed , and then add the position embedding to get .Then Input to the masked multi-head self-attention module, and get . As a query, acoustic depth features As key and value, it is input into the multi-head cross attention module to obtain .at last After the feedforward module, we get . It is used as the input of the decoder block of the next layer. The calculation process of each encoder block is as follows:

[0127] (14)

[0128] (15)

[0129] (16)

[0130] in, represents the multi-head self-attention module, represents a multi-head cross attention module, Represents a feed-forward module.

[0131] During training, the output of the Transformer decoder passes through the fully connected layer and the softmax layer, and the cross entropy loss is used to calculate its loss, which is recorded as .

[0132] When jointly trained, the loss of the CTC decoder is , the loss of the Transformer decoder is , the total loss of hybrid CTC / Attention joint training The definition is as follows:

[0133] (17)

[0134] in, is the loss weight of the MDD task, , used to balance the losses of CTC and Attention.

[0135] In the task of mispronunciation detection and diagnosis, we use To optimize the decoding results of CTC and Transformer at the same time.

[0136] During inference, this embodiment uses the attention re-scoring method to predict the phoneme results. Specifically, the CTC prefix beam search algorithm is first executed to initialize the prefix prefix and its corresponding score , for each time step t, expand the current prefix set, and then obtain the n best candidate sequences (nbest) from the model. Then for each candidate prefix prefix, use the Attention mechanism to calculate the attention score, where the query comes from the current state of the decoder, the key and value come from the output of the encoder, and then calculate the attention weight and generate the context vector. The Transformer decoder combines the context vector and the current state to generate the attention score. The final combined score is as follows:

[0137] (18)

[0138] in, is the combined score, is the CTC score, is the Attention score, It is the same weight parameter as the loss calculation, which is used to balance the CTC and Attention scores.

[0139] Finally, according to the combined score The prefix with the highest score is selected as the final output sequence. The final output sequence needs to be compared with the original phoneme sequence to calculate the phoneme error rate.

[0140] When APA and MDD are jointly trained, the loss of APA is calculated as shown in formula (11), where the loss is ; The loss calculation of MDD is shown in formula (17), the loss is The total loss of APA and MDD multi-task learning is As shown below:

[0141] (19)

[0142] in, is the loss weight for multi-task learning, , used to balance the losses of APA and MDD.

[0143] Embodiment 2

[0144] An English auxiliary pronunciation training system based on multi-task learning, comprising:

[0145] A data acquisition module, configured to acquire English pronunciation audio to be evaluated;

[0146] An audio deep feature extraction module is configured to preprocess the English pronunciation audio, extract audio features, and obtain a deep feature vector of the English pronunciation;

[0147] An automatic pronunciation evaluation module, configured to perform multi-granularity evaluation on the deep feature vector using a pre-trained automatic pronunciation evaluation model to determine pronunciation evaluation results at the phoneme level, word level, and sentence level;

[0148] an incorrect pronunciation detection and diagnosis module, configured to perform phoneme recognition on the deep feature vector using a pre-trained pronunciation detection and diagnosis model and calculate a phoneme error rate of English pronunciation;

[0149] The joint optimization module is configured to jointly optimize the losses of automatic pronunciation evaluation and mispronunciation detection and diagnosis during the training process of the automatic pronunciation evaluation model and the pronunciation detection and diagnosis model.

[0150] like Figure 2As shown, in this embodiment, the audio deep feature extraction module specifically executes the steps including: preprocessing the English pronunciation audio, and the process of extracting audio features includes: extracting audio features using a trained self-supervised learning model, wherein the self-supervised learning model includes a convolutional neural network encoder, a quantization module, and a Transformer context network, wherein the convolutional neural network encoder uses multiple convolutional layers to process the original audio signal to generate a low-level continuous feature representation, and the quantization module uses a vector quantizer to convert the continuous feature representation into a discrete feature encoding for comparative learning, and the Transformer context network uses a Transformer network to model the quantized features, generate a context representation, capture long-distance dependencies, and generate acoustic deep features with semantic information;

[0151] During the training process of the self-supervised learning model, the model is fine-tuned, a fully connected layer and CTC decoder are added after the Transformer context network, and the CTC loss is used to optimize the training results.

[0152] like Figure 3 As shown, in this embodiment, the automatic pronunciation evaluation model of the automatic pronunciation evaluation module includes a convolution-enhanced Transformer encoder and a multi-aspect and multi-granularity scoring module, wherein the convolution-enhanced Transformer encoder is used to encode the deep feature vector and extract the scoring information shared by the phoneme level, word level and sentence level; the multi-aspect and multi-granularity scoring module is used to score each sentence at three granularities: the phoneme level, the word level and the sentence level; the sentence level includes five aspects: completeness, accuracy, fluency, rhythm and total score; the word level includes three aspects: accuracy, stress and total score; and the phoneme level includes one aspect of accuracy.

[0153] In this embodiment, the convolution-enhanced Transformer encoder includes a convolution module, a self-attention module and a feedforward network connected in sequence, and each layer is connected by a residual structure; the convolution module is used to capture the local context information of the deep feature vector to extract the local information between phonemes and words; the self-attention module is used to capture the global context information, and assign different weights to the input information to model the sentence level; the feedforward network is used to perform nonlinear transformation and enhanced feature representation on the input features, and fuse the acoustic features passed through the convolution module and the self-attention module.

[0154] In this embodiment, the multi-aspect and multi-granularity scoring module is provided with a scoring head for each aspect of each granularity, and two multi-layer perceptrons are designed to adapt to different scoring heads, the first multi-layer perceptron is used for phoneme and word scoring heads, and the second multi-layer perceptron is used for sentence scoring heads, and both multi-layer perceptrons are composed of a linear layer and a one-dimensional adaptive average pooling, and the output dimensions of the two are different;

[0155] The multi-aspect and multi-granularity scoring module pre-fills the phoneme-level and word-level scores. During the training process, the automatic pronunciation evaluation task is regarded as a regression task to predict the scores of each aspect of each granularity. The mean square error loss is used as the loss function, and when calculating the prediction score loss, the filled positions are ignored so that the filling does not participate in the loss calculation. The predicted scores at the phoneme level, word level, and sentence level are respectively calculated with the true labels for mean square error, and finally the sum is taken as the total loss of the automatic pronunciation evaluation task.

[0156] like Figure 4 As shown, in this embodiment, the pronunciation detection and diagnosis model of the mispronunciation detection and diagnosis module includes a CTC decoder and a Transformer decoder, wherein:

[0157] The CTC decoder includes a fully connected layer and a CTC prediction head. The fully connected layer is a fully connected layer added when the self-supervised learning model is fine-tuned by extracting audio features, sharing the fine-tuned weights, and obtaining an unnormalized score; the CTC prediction head converts the unnormalized score into a logarithmic probability to obtain the probability of each frame of speech corresponding to a phoneme in the vocabulary;

[0158] The Transformer decoder includes multiple layers of decoder blocks, at least one layer of decoder blocks includes a masked multi-head self-attention module, a multi-head cross-attention module and a feedforward module connected in sequence.

[0159] It can be understood that the above-mentioned units / modules can be separately or completely combined into one or several other units / modules, or one (some) of the units can be further divided into multiple functionally smaller units, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application.

[0160] The above modules of the system are divided based on logical functions. In practical applications, the function of one module can also be implemented by multiple modules, or the functions of multiple modules can be implemented by one module. For example, the audio depth feature extraction module of this embodiment may include:

[0161] A convolutional neural network encoder that processes the raw audio signal using multiple convolutional layers to generate low-level continuous feature representations;

[0162] A quantization module for converting continuous feature representations into discrete feature encodings using a vector quantizer for contrastive learning;

[0163] Transformer context network, which is used to model the quantized features, generate contextual representations, capture long-range dependencies, and generate acoustic deep features with semantic information.

[0164] Similarly, in other embodiments of the present application, the system may also include other units / modules. In actual applications, these functions may also be implemented with the assistance of other units, and may be implemented by collaboration of multiple units.

[0165] According to another embodiment of the present application, the system described in this embodiment can be constructed and the method of Embodiment 1 can be implemented by running a computer program (including program code) capable of executing the steps involved in the corresponding method described in Embodiment 1 on a general-purpose computing device such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, and loaded into the above-mentioned computing device through the computer-readable recording medium and run therein.

[0166] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of an all-hardware embodiment, an all-software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of one or more computer-usable storage media (including but not limited to disk storage, CD - ROM , optical storage, etc.).

[0167] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0168] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0169] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0170] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made by those skilled in the art within the spirit and principle of the present invention without creative labor shall be included in the protection scope of the present invention.

Claims

1. An English auxiliary pronunciation training method based on multi-task learning, characterized in that: The following steps are involved: Get the English pronunciation audio to be evaluated; Preprocessing the English pronunciation audio, extracting audio features, and obtaining a deep feature vector of the English pronunciation; Use a pre-trained automatic pronunciation evaluation model to perform multi-granularity evaluation on the deep feature vector to determine the pronunciation evaluation results at the phoneme level, word level, and sentence level; Use the pre-trained pronunciation detection and diagnosis model to perform phoneme recognition on the deep feature vectors and calculate the phoneme error rate of English pronunciation; During the training of the automatic pronunciation evaluation model and the pronunciation detection and diagnosis model, jointly optimizing the losses of the automatic pronunciation evaluation and the mispronunciation detection and diagnosis; The total loss of the automatic pronunciation evaluation task is: ;in, represents the loss of phoneme score prediction, represents the loss of word score prediction, represents the loss of sentence score prediction; When jointly trained, the loss of the CTC decoder is , the loss of the Transformer decoder is , the total loss of hybrid CTC / Attention joint training The definition is as follows: ;in, is the loss weight of the MDD task, , used to balance the loss of CTC and Attention; in the task of mispronunciation detection and diagnosis, the total loss is used Optimize the results of CTC decoder and Transformer decoder at the same time; The process of jointly optimizing the loss of automatic pronunciation evaluation and mispronunciation detection and diagnosis includes: multi-task learning total loss for: ;in, is the loss weight for multi-task learning, , used to balance the losses of automatic pronunciation evaluation and mispronunciation detection and diagnosis.

2. The English auxiliary pronunciation training method based on multi-task learning as claimed in claim 1, characterized in that: The English pronunciation audio is preprocessed, and the process of extracting audio features includes: extracting audio features using a trained self-supervised learning model, wherein the self-supervised learning model includes a convolutional neural network encoder, a quantization module, and a Transformer context network, wherein the convolutional neural network encoder uses multiple convolutional layers to process the original audio signal to generate a low-level continuous feature representation, and the quantization module uses a vector quantizer to convert the continuous feature representation into a discrete feature encoding for contrastive learning, and the Transformer context network uses a Transformer network to model the quantized features, generate context representations, capture long-distance dependencies, and generate acoustic deep features with semantic information.

3. The English auxiliary pronunciation training method based on multi-task learning as claimed in claim 2, characterized in that: During the training process of the self-supervised learning model, the model is fine-tuned, a fully connected layer and CTC decoder are added after the Transformer context network, and the CTC loss is used to optimize the training results.

4. The English auxiliary pronunciation training method based on multi-task learning as claimed in claim 1, characterized in that: The automatic pronunciation evaluation model includes a convolution-enhanced Transformer encoder and a multi-aspect and multi-granularity scoring module, wherein the convolution-enhanced Transformer encoder is used to encode deep feature vectors and extract scoring information shared by phoneme level, word level and sentence level; the multi-aspect and multi-granularity scoring module is used to score each sentence at three granularities: phoneme level, word level and sentence level; the sentence level includes five aspects: completeness, accuracy, fluency, rhythm and total score; the word level includes three aspects: accuracy, stress and total score; and the phoneme level includes one aspect of accuracy.

5. The English auxiliary pronunciation training method based on multi-task learning as claimed in claim 4, characterized in that: The convolution-enhanced Transformer encoder includes a convolution module, a self-attention module and a feedforward network connected in sequence, and each layer is connected by a residual structure; the convolution module is used to capture the local context information of the deep feature vector to extract the local information between phonemes and words; the self-attention module is used to capture the global context information, and assign different weights to the input information to model the sentence level; the feedforward network is used to perform nonlinear transformation and enhanced feature representation on the input features, and fuse the acoustic features passed through the convolution module and the self-attention module.

6. The English auxiliary pronunciation training method based on multi-task learning as claimed in claim 4, characterized in that: The multi-aspect and multi-granularity scoring module is provided with a scoring head for each aspect of each granularity, and two multi-layer perceptrons are designed to adapt to different scoring heads, the first multi-layer perceptron is used for phoneme and word scoring heads, and the second multi-layer perceptron is used for sentence scoring heads, and the two multi-layer perceptrons are both composed of a linear layer and a one-dimensional adaptive average pooling, and the output dimensions of the two are different; The multi-aspect and multi-granularity scoring module pre-fills the phoneme-level and word-level scores. During the training process, the automatic pronunciation evaluation task is regarded as a regression task to predict the scores of each aspect of each granularity. The mean square error loss is used as the loss function, and when calculating the prediction score loss, the filled positions are ignored so that the filling does not participate in the loss calculation. The predicted scores at the phoneme level, word level, and sentence level are respectively calculated with the true labels for mean square error, and finally the sum is taken as the total loss of the automatic pronunciation evaluation task.

7. The English auxiliary pronunciation training method based on multi-task learning as claimed in claim 1, characterized in that: The pronunciation detection and diagnosis model includes a CTC decoder and a Transformer decoder, wherein: The CTC decoder includes a fully connected layer and a CTC prediction head. The fully connected layer is a fully connected layer added when the self-supervised learning model is fine-tuned by extracting audio features, sharing the fine-tuned weights, and obtaining an unnormalized score; the CTC prediction head converts the unnormalized score into a logarithmic probability to obtain the probability of each frame of speech corresponding to a phoneme in the vocabulary; The Transformer decoder includes multiple layers of decoder blocks, at least one layer of decoder blocks includes a masked multi-head self-attention module, a multi-head cross-attention module and a feedforward module connected in sequence.

8. An English auxiliary pronunciation training system based on multi-task learning, characterized in that: include: A data acquisition module, configured to acquire English pronunciation audio to be evaluated; An audio deep feature extraction module is configured to preprocess the English pronunciation audio, extract audio features, and obtain a deep feature vector of the English pronunciation; An automatic pronunciation evaluation module, configured to perform multi-granularity evaluation on the deep feature vector using a pre-trained automatic pronunciation evaluation model to determine pronunciation evaluation results at the phoneme level, word level, and sentence level; an incorrect pronunciation detection and diagnosis module, configured to perform phoneme recognition on the deep feature vector using a pre-trained pronunciation detection and diagnosis model and calculate a phoneme error rate of English pronunciation; The joint optimization module is configured to jointly optimize the losses of automatic pronunciation evaluation and mispronunciation detection and diagnosis during the training of the automatic pronunciation evaluation model and the pronunciation detection and diagnosis model; the total loss of the automatic pronunciation evaluation task is: ;in, represents the loss of phoneme score prediction, represents the loss of word score prediction, represents the loss of sentence score prediction; When jointly trained, the loss of the CTC decoder is , the loss of the Transformer decoder is , the total loss of hybrid CTC / Attention joint training The definition is as follows: ;in, is the loss weight of the MDD task, , used to balance the loss of CTC and Attention; in the task of mispronunciation detection and diagnosis, the total loss is used Optimize the results of CTC decoder and Transformer decoder at the same time; The process of jointly optimizing the loss of automatic pronunciation evaluation and mispronunciation detection and diagnosis includes: multi-task learning total loss for: ;in, is the loss weight for multi-task learning, , used to balance the losses of automatic pronunciation evaluation and mispronunciation detection and diagnosis.

Images