A deep learning-based pronunciation evaluation scoring method

Abstract

The present application relates to the technical field of speech evaluation, in particular to a pronunciation evaluation and scoring method based on deep learning.The present application uses a speech recognition model to recognize the true text result of the audio.Then an HMM-DNN model is used to obtain the posterior probability of the audio.Finally, a scoring model is used to score the phonemes.Before forced alignment, the speech recognition model is used to identify the correct text of the audio, avoiding the situation that the audio and the text are inconsistent and cannot be aligned to the correct position during the forced alignment process.Meanwhile, the scoring model is constructed using a deep neural network, which can fit multiple information such as posterior probability, initial and final consonants, part of speech, tone, pronunciation duration, etc., making the phoneme scoring more reasonable and more accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech evaluation, in particular to a pronunciation evaluation scoring method based on deep learning, which uses a deep learning method to realize the scoring mechanism of phonemes in the pronunciation evaluation process, so that the phoneme scoring in pronunciation evaluation is more reasonable and accurate. BACKGROUND

[0002] Spoken English is increasingly valued in language education courses, and one-on-one communication and teaching between teachers and students is the most effective way to improve English speaking skills, but it is difficult to meet the needs of a large number of spoken English learners. Thanks to the rapid progress of computer technology and pronunciation evaluation technology, various spoken English evaluation schemes based on artificial intelligence technology have been implemented. Providing additional learning opportunities and rich learning materials for students can assist or replace teachers in guiding students to conduct more targeted pronunciation practice, point out students' pronunciation errors, provide effective diagnostic feedback information, and evaluate students' overall pronunciation level, effectively improving students' spoken English learning efficiency and spoken English level.

[0003] The current mainstream method of pronunciation evaluation is to obtain the posterior probability of the speech based on a hidden Markov-deep neural network (HMM-DNN) model, and then perform forced alignment with the evaluation text, and use the GOP method to score.

[0004] The forced alignment method can achieve high accuracy, but it must meet a prerequisite: the given text and audio must match. If a user reads "I am a teacher" as "I was a teacher", when processing the audio segment corresponding to "was", it will incorrectly align the phonemes corresponding to "am", which may cause the subsequent "a" and "teacher" to be unable to align to the correct position, thereby affecting the accuracy of the scoring. SUMMARY

[0005] To solve the above problems, the present application proposes a pronunciation evaluation scoring method based on deep learning. First, a speech recognition model is used to recognize the text of the audio, and then the recognized text is used for forced alignment, so that the alignment result is more accurate. Finally, a scoring model constructed by a deep neural network is used to predict the score of the phonemes, and the score of the words and sentences is calculated according to the score of the phonemes.

[0006] This paper proposes a pronunciation evaluation scoring method based on deep learning. First, a speech recognition model is used to recognize the true text result of the audio. Second, an HMM-DNN model is used to obtain the posterior probability of the audio. Then, the recognized text result of the audio and the posterior probability of the audio are used for forced alignment to determine the time boundary of each phoneme. Finally, a scoring model is used to score the phonemes.

[0007] The specific technical solutions are as follows:

[0008] Step one, the acoustic features of the speech to be evaluated are extracted and sent into a speech recognition model to identify the real text result of the speech to be evaluated.

[0009] Step two, the acoustic features of the speech to be evaluated extracted in step one are sent into an HMM-DNN model to predict the posterior probability of each frame.

[0010] Step three, according to the text result identified in step one and the posterior probability of each frame obtained in step two, forced alignment is performed to determine the time boundary of each phoneme.

[0011] Step four, according to the time boundary of each phoneme obtained in step three and the posterior probability of each frame obtained in step two, the average value of the posterior probability of each phoneme is calculated, and then the average value of the posterior probability of the phoneme and the feature information of the phoneme, such as initial consonant, part of speech, tone, pronunciation duration, etc. are spliced together and sent into a scoring model to obtain the score of the phoneme.

[0012] Step five, according to the text result identified in step one and the reference text, phoneme alignment is performed to determine which phonemes are overread and missed.

[0013] Step six, the final score is calculated, and the score of the word and the score of the entire sentence are calculated according to the overread and missed conditions in step five.

[0014] Beneficial effects

[0015] The application uses a speech recognition model to identify the correct text of the audio before forced alignment, avoiding the situation that the audio and the text are inconsistent and cannot be aligned to the correct position in the forced alignment process. At the same time, a scoring model is constructed using a deep neural network, which can fit multiple information such as posterior probability, initial consonant, part of speech, tone, pronunciation duration, etc., making the phoneme scoring more reasonable and accurate.

[0016] 1. A speech recognition model is used to identify the correct text of the audio, avoiding the situation that the audio and the text are inconsistent and cannot be aligned to the correct position in the forced alignment process.

[0017] 2. A scoring model is constructed using a deep neural network, which can fit multiple information such as posterior probability, initial consonant, part of speech, tone, pronunciation duration, etc., making the phoneme scoring more reasonable and accurate. BRIEF DESCRIPTION OF DRAWINGS

[0018] Figure 1 is a flowchart. DETAILED DESCRIPTION

[0019] The application will be described in further detail below with reference to the drawings.

[0020] Figure 1 is a flowchart of the pronunciation evaluation method of the application based on deep learning. As shown in Figure 1 , it mainly includes the following steps:

[0021] Step one, extract the acoustic features of the speech to be evaluated. The extracted acoustic features can be Fbank features. When extracting Fbank features, the sampling frequency is 16000, the window length is set to 25ms, and the frame shift is set to 10ms. After extracting the features, they are sent to a speech recognition model. The speech recognition model can use a wenet model to identify the true text result of the speech to be evaluated.

[0022] Step two, send the acoustic features of the speech to be evaluated extracted in step one, such as Fbank features, to an HMM-DNN model to predict the posterior probability of each frame. Assuming there are m frames and n phonemes in total, an m*n posterior probability matrix will be generated finally.

[0023] Step three, according to the text result identified in step one and the posterior probability of each frame obtained in step two, perform forced alignment using a greedy or Viterbi algorithm to find a path with the maximum probability and finally determine the time boundary of each phoneme.

[0024] Step four, according to the time boundary of each phoneme obtained in step three and the posterior probability of each frame obtained in step two, the average value of the posterior probability of each phoneme can be calculated.

[0025] Then the feature information of each phoneme needs to be obtained, such as vowel, consonant, tone, pronunciation duration, etc.

[0026] 1. Vowel and consonant features: determine whether the current phoneme is a vowel or a consonant. The scoring of vowels and consonants will be different for different phoneme types, and vowels will be more important.

[0027] 2. Part of speech feature: determine the part of speech of the word in which the current phoneme is located. According to the importance of the part of speech of the word, the scoring results will be different, and the importance of real words such as verbs and nouns is higher than that of virtual words.

[0028] 3. Tone feature: determine whether the current phoneme contains a tone. If the tone information is marked in the text but not read in the audio, the final score will be reduced.

[0029] 4. Pronunciation duration feature: calculate the duration of the current phoneme and normalize it with the standard phoneme pronunciation duration.

[0030] If the normalized pronunciation duration feature is too low or too high, the score will be reduced.

[0031] Finally, the average value of the posterior probability of the phoneme and the feature information of the phoneme, such as the initial consonant, the part of speech, the tone, and the pronunciation duration, are spliced together and sent to the scoring model to obtain the score of the phoneme.

[0032] The scoring model can be modeled using a DNN, using a three-layer deep neural network DNN, with a hidden layer dimension of 128 and a last layer dimension of 1, outputting the final scoring result.

[0033] By using a deep neural network DNN to model various features of the phoneme, a complex function is fitted to score the phoneme, making the phoneme scoring more reasonable and accurate.

[0034] Step five, according to the text result identified in step one and the reference text, perform phoneme alignment to determine which phonemes are multi-read phonemes, missing-read phonemes, and normal phonemes.

[0035] The alignment method can be implemented using a method of calculating the edit distance. First, convert the identified text result into a corresponding phoneme string list X[1,…,n], and convert the reference text into a corresponding phoneme string list Y[1,…,m]. Define the distance D(i,j) as the distance between X[1,…,i] and Y[1,…,j], and the edit distance between X and Y is D(n,m). Use the dynamic programming method to solve it, and the state transition equation is:

[0036] D(i,0)=i

[0037] D(0,j)=j

[0038]

[0039] Where insert represents multi-read phonemes, delete represents missing-read phonemes, and norm represents normal phonemes.

[0040] By recording which sub-problem each sub-result is derived from, the results can be backtracked, and finally the alignment results of the two strings can be obtained.

[0041] Step six, calculate the final score, and calculate the word score and sentence score according to the multi-read and missing-read conditions in step five.

[0042] The word score is related to the scores of all normal phonemes within the word and the multi-read and missing-read ratio. The sentence score is related to the scores of all normal phonemes within the sentence and the multi-read and missing-read ratio, as well as the number of continuous frames of the word and the number of silent frames between words. The number of silent frames between words only calculates the silent frames greater than the normal pause time.

[0043] The score of a word is calculated using the following formula:

[0044]

[0045] where score norm_phone represents the score of normal phonemes within the word, n represents the number of all phonemes in the reference text, norn cnt represents the number of normal phonemes, ins cnt represents the number of extra phonemes, del cnt represents the number of missing phonemes.

[0046] The score of a sentence is calculated using the following formula:

[0047]

[0048] where score norm_phone represents the score of normal phonemes within the sentence, n represents the number of all phonemes in the reference text, norn cnt represents the number of normal phonemes, ins cnt represents the number of extra phonemes, del cnt represents the number of missing phonemes. frame word represents the number of frames occupied by all words, frame sil represents the number of silence frames between words.

Claims

1. A deep learning-based pronunciation evaluation scoring method, characterized in that, The method comprises the following steps: Firstly, the real text result of the audio is recognized by a voice recognition model; Secondly, the posterior probability of the audio is obtained by an HMM-DNN model; Then, forced alignment is performed using the recognized text result of the audio and the posterior probability of the audio to determine the time boundary of each phoneme; Finally, a scoring model is used to score the phonemes; The specific steps are as follows: Step one: acoustic features of the speech to be evaluated are extracted and input into a speech recognition model to recognize the real text result of the speech to be evaluated; Step two: the acoustic features of the speech to be evaluated extracted in step one are input into an HMM-DNN model to predict the posterior probability of each frame; Step three: forced alignment is performed according to the text result recognized in step one and the posterior probability of each frame obtained in step two to determine the time boundary of each phoneme; Step four: the average value of the posterior probability of each phoneme is calculated according to the time boundary of each phoneme obtained in step three and the posterior probability of each frame obtained in step two, and then the average value of the posterior probability of the phoneme and the feature information of the phoneme, such as initial consonant, word class, tone and pronunciation duration, are spliced together and input into a scoring model to obtain the score of the phoneme; Step five: phoneme alignment is performed according to the text result recognized in step one and the reference text to determine which phonemes are over-read phonemes, missing-read phonemes and normal phonemes; Step six: the final score is calculated, and the score of a word and the score of a sentence are calculated according to the over-read and missing-read conditions in step five; The step three is performed by using a greedy algorithm or a Viterbi algorithm to find a path with the maximum probability and finally determine the time boundary of each phoneme; The feature used in the scoring model in step four is the average value of the posterior probability of the phoneme and the feature information of the phoneme, such as initial consonant, word class, tone and pronunciation duration; The phoneme alignment method in step five is realized by calculating the edit distance, wherein the recognized text result is first converted into a corresponding phoneme string list X[1,…,n], the reference text is converted into a corresponding phoneme string list Y[1,…,m], the distance D(i,j) is defined as the distance between X[1,…,i] and Y[1,…,j], and the edit distance between X and Y is D(n,m); The dynamic programming method is used to solve the problem, and the state transition equation is as follows: D(i,0)=i D(0,j)=j Wherein, insert represents over-read phonemes, delete represents missing-read phonemes, and norm represents normal phonemes; The alignment result of the two strings is obtained by backtracking the result by recording which sub-problem each sub-result is solved from; The score of a word is calculated by using the following formula: wherein score norm_phone represents the score of a normal phoneme within a word, n represents the number of all phonemes in the reference text, norn cnt represents the number of normal phonemes, ins cnt represents the number of multi-reading phonemes, del cnt represents the number of missed phonemes; The score of a sentence is calculated by using the following formula: where score norm_phone represents the score of normal phonemes within a sentence, n represents the number of all phonemes in the reference text, norn cnt represents the number of normal phonemes, ins cnt represents the number of multi-reading phonemes, del cnt represents the number of missed phonemes, frame word represents the number of frames occupied by all words, frame sil represents the number of silent frames between words.

2. The method of claim 1, wherein the method further comprises: The acoustic features extracted in step one are Fbank features.

3. The method of claim 1, wherein the method further comprises: The speech recognition model used in step one is a wenet model.

Images