Learning evaluation method and system for English word pronunciation through audio recognition

Abstract

The invention relates to a learning evaluation method for English word pronunciation through audio recognition, which comprises the following steps: constructing a speech recognition system, preprocessing received user pronunciation audio, and then realizing phonetic symbol recognition of recognized words through vowel phonetic symbol recognition and screening and consonant phonetic symbol recognition and screening or full phoneme recognition. The word recognized by the final recognition phonetic symbol sequence obtained by the speech recognition system is compared with the stored standard pronunciation of the word, the pronunciation of the practicer is scored, and specific practice suggestions are given.

Description

A learning assessment method and system for English word pronunciation through audio recognition Technical Field

[0001] This invention relates to the fields of speech recognition and education technology, and in particular to a learning assessment method and system for English word pronunciation through audio recognition. Background Technology

[0002] With globalization, the importance of spoken English proficiency has become increasingly prominent as an international language. Standard pronunciation is the foundation of effective communication. Traditional English pronunciation learning mainly relies on face-to-face instruction from teachers or self-imitation of audio recordings, both of which have significant limitations. Teacher instruction is costly and inflexible in terms of time; while self-imitation lacks an effective feedback mechanism, making it difficult for learners to accurately understand the specific differences between their pronunciation and standard pronunciation, resulting in incorrect pronunciation habits being difficult to correct once formed.

[0003] While existing speech recognition technology is widely used for voice input and simple command recognition, its application depth is insufficient in language learning, especially in pronunciation calibration down to the phonetic symbol level. Most pronunciation assessment software can only provide a general score for the overall pronunciation of words or sentences, unable to pinpoint which phoneme (phonetic symbol) is mispronounced or in which specific pronunciation dimension (such as mouth shape, tongue position, vowel length, etc.) the problem exists. This makes it difficult for learners to make targeted improvements even if they know their pronunciation is not standard. Existing speech recognition algorithms also suffer from high computational load, slow recognition speed, and low efficiency. Furthermore, there are differences in speaking speed and pronunciation patterns between different regions and people, which makes word pronunciation recognition difficult, affecting the accuracy of phonetic symbol recognition and resulting in poor recognition performance of existing methods.

[0004] Therefore, there is an urgent need in this field for an English word learning method that can accurately break down word pronunciation to the phonetic symbol level and quantitatively score and specifically correct each phonetic symbol. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a learning and assessment method for English word pronunciation through audio recognition. This method can accurately decompose the user's word pronunciation into individual phonetic symbols and independently evaluate and score the pronunciation quality of each phonetic symbol, thereby achieving accurate and efficient pronunciation calibration.

[0006] To achieve the above-mentioned objectives, this invention provides a method for learning and evaluating the pronunciation of English words through audio recognition, comprising the following steps: Step S1: Constructing a speech recognition system, which has a memory pre-stored standard audio data of all phonetic symbols and standard phonetic symbol sequences for each word. The database contains idealized pronunciation samples for each phonetic symbol as a benchmark for comparison.

[0007] The phonetic symbol sequence of each word is explained here as a collection of commonly used words and a phonetic symbol classification annotation set for each word; the phonetic symbol classification annotation set: the phonetic symbol of each word is broken down into "vowel sequence" and "consonant sequence". For example, the phonetic symbol classification annotation of "banana" has three columns: the complete phonetic symbol: / bəˈnɑːnə / , the vowel sequence [ə,ɑː,ə], and the consonant sequence [b,n,n].

[0008] Step S2: The received user pronunciation audio is preprocessed using the speech recognition system. First, the input raw user pronunciation audio signal x(t) is pre-emphasized, framed, and windowed to eliminate the effects of the vocal cords and lips during pronunciation and reduce spectral leakage. Mel-frequency cepstral coefficients (MFCCs) are used to extract acoustic features from each frame of the audio signal; these features can simulate the auditory perception characteristics of the human ear. The entire word, after acoustic feature extraction, is represented as a series of feature sequences O=(o1,o2,...o...). t ...,o T ).

[0009] Where t is the continuous time variable of the original audio, x(t) is the amplitude of the original audio signal at time t, and o t Let O represent the acoustic feature vector of the audio at frame t, where T represents the total number of frames after the word audio is segmented, and O is the time-frequency feature sequence of the entire audio, which is a T×D matrix, where D is the dimension of the feature vector.

[0010] Step S3: Vowel phonetic symbol recognition and consonant phonetic symbol recognition. Step S3 includes the following two steps: S31 and S32.

[0011] Step S31: Vowel Phonetic Symbol Recognition and Vowel Sequence Set Filtering; A phonetic symbol recognizer based on a Hidden Markov Model (HMM) is specifically trained for vowel phonetic symbols (ɑː, ɔː, ɜː, iː, uː, æ, e, ɪ, ɒ, ʊ, ə, etc.); Input the preprocessed feature sequence O, output the vowel sequence of the recognized word pronunciation, and store this vowel sequence as a vowel sequence set; Database retrieval: Based on the recognized vowel sequence set, retrieve all words in the phonetic symbol database that completely match the recognized vowel sequence set, and store this word set as set A; If the number of retrieved words is 1, then proceed to step S32 without further processing, and use this unique word as the recognized word to proceed to step S5; If the number of retrieved words is ≥2, then proceed to step S32.

[0012] Step S32: Consonant Phonetic Symbol Recognition and Consonant Sequence Set Filtering; Optimize the model based on the pronunciation features of consonant phonetic symbols (p, b, t, d, k, g, f, v, s, z, etc.) (such as the temporal impact of plosives and fricatives) to improve consonant recognition and obtain a trained SVM model; Input the preprocessed feature sequence O, output the consonant sequence of the recognized word pronunciation, and store the consonant sequence as a consonant sequence set; Search the database, and based on the recognized consonant sequence set, search the phonetic symbol database for all words that completely match the recognized consonant sequence set, and store the word set as set B; If the number of retrieved words is 1, use this unique word as the recognized word and proceed to the next step S5; If the number of retrieved words is ≥2, intersect set B and set A to obtain set C. If the number of words in set C is 1, use this unique word as the recognized word and proceed to the next step S5; If the number of words in set C is ≥2, proceed to the next step S4.

[0013] Step S4: Full phoneme recognition.

[0014] Acoustic modeling and phoneme sequence prediction: The feature sequence O is input into a Bi-LSTM (Bidirectional Long Short-Term Memory) network to obtain the hidden vector state ht containing contextual information for each frame, which can represent the dependencies between frames in the audio signal; the hidden vector state ht output for each frame is then processed. t A fully connected layer and a softmax layer are fed in to calculate the probability distribution P of each phoneme (phonetic symbol) at time t in the frame: in, Let be the phoneme label predicted at time t, which is a random variable. It could be any of the 48 English phonetic symbols; 'a' represents each of the monosyllabic phonetic symbols (excluding the 8 diphthongs) in the 48 English phonetic symbols; K is the total size of the set of phonetic symbols 'a', where K is 40; W aLet b be the weight matrix corresponding to the phonetic symbol a. a W is the bias term corresponding to the phonetic symbol 'a'. a b a The model was learned from a large amount of labeled data through backpropagation during the previous model training phase; h t Let be the hidden state vector output by the Bi-LSTM network at time t.

[0015] The maximum probability P for selecting each frame. t The corresponding phonetic symbols are used as the phonetic symbols identified at the corresponding time frame t, thus enabling the identification of the entire word's frame phonetic symbol sequence.

[0016] The above describes the feature vector o of each frame in the feature sequence O. t The input is fed into a Bi-LSTM to obtain the hidden vector state ht containing contextual information for each frame. The regular iterative computation steps of Bi-LSTM are as follows: First, the forward and backward states are obtained through bidirectional LSTM processing: the forward LSTM scans the sequence from left to right, updating its internal state at each step. This state is a memory of the "past". The backward LSTM scans the sequence from right to left, updating its internal state at each step. This state is a memory of the "future"; both of the above LSTMs use forget gates, input gates, and output gates to determine what to remember, what to forget, and what to output; then at each time point t, the state representing the "past" is... and representing the "future" These are concatenated together to form the final context-aware representation of the hidden vector state ht. At this point, if... and If the dimensions of each element are d, then the dimensions of the concatenated ht will be 2d.

[0017] It is worth noting that ht is an information-rich vector that not only contains information about the current frame x(t), but also encodes all the important context related to the current moment in the pronunciation of the word. This ensures that the subsequent Softmax layer can accurately determine what the specific phoneme is in the current context.

[0018] The integration process adopts a simple integration approach, which mainly solves the problem of mismatch between frame-level prediction and phoneme sequence length through experience and simple settings. It includes two steps: (1) First, in the above-mentioned frame phonetic symbol sequence that identifies the entire word, adjacent phonetic symbols identified between adjacent frames are merged, that is, multiple adjacent phonetic symbols can be merged into one phonetic symbol. For example, the phonetic symbols identified in frames 15-28 are all |ɪ|, so they are merged into one phonetic symbol |ɪ| to obtain the merged phonetic symbol sequence; (2) At the same time, the pronunciation duration of each phonetic symbol can be roughly determined in advance based on the habitual pronunciation speed when learning English. Based on the phonetic symbol sequence after merging the above phonetic symbols, the system intelligently integrates the matter of whether two monophthongs need to be merged into a diphthong phonetic symbol. According to the recognition results, the phonetic symbols recognized in frames 23-34 are all |ɪ| (the recognition results before frame 33 and after frame 34 are not this phonetic symbol). Therefore, frames 23-34 are merged into a single phonetic symbol |ɪ|. However, it is found that the duration of the merged phonetic symbol |ɪ| is too short. Therefore, it is considered to merge the merged phonetic symbol |ɪ| with the phonetic symbols recognized before or after its duration (which must be monophthongs) to form a diphthong phonetic symbol.

[0019] After obtaining the final sequence of phonetic symbols as described above, the identified word is obtained.

[0020] The pronunciation duration of each phonetic symbol can be roughly determined based on the habitual pronunciation and speaking speed when learning English. Combined with the phonetic symbol sequence after merging the phonetic symbols, the system intelligently integrates the matter of whether two monophthongs need to be merged into a diphthong phonetic symbol. The habitual pronunciation and speaking speed when learning English can be determined through the following pre-arranged steps: Before step S1, step S0 is also included: The speech recognition system provides the user with a certain training English word stored in the system for reading aloud and provides a pronunciation speed judgment for the phonetic symbols. Specifically, the learner reads the word given by the speech recognition system at their usual reading or speaking speed. Since the speech recognition system knows in advance what word the user is reading, the speech recognition system only needs to identify the pronunciation duration of each phonetic symbol in the phonetic symbol sequence of the word pronunciation (this identification can be obtained through conventional methods such as the audio feature signals at each time point in the pronunciation signal, such as amplitude and frequency), and then store and record it to obtain the speech speed characteristics of each learner. This facilitates more targeted and accurate recognition of each learner's pronunciation practice in the future, effectively eliminating the influence of each person's speaking, pronunciation, and reading speed on the recognition results.

[0021] This invention provides a learning assessment method for English word pronunciation through audio recognition. It further involves comparing the word identified by the final phonetic sequence obtained through the aforementioned speech recognition system with its stored standard pronunciation, scoring the learner's pronunciation, and providing specific practice suggestions. Specifically, it includes the following step S5: English word pronunciation learning assessment. Step S5 specifically includes the following steps: Step 51: Based on the standard phonetic sequence of the identified word, this standard phonetic sequence has standard features corresponding to its time-frequency domain (the pre-stored standard phonetic sequence of the word is actually the time-frequency feature sequence of this standard phonetic sequence). Assume we obtain a user audio segment of the i-th phonetic symbol of a word read aloud by the user.

[0022] Step 52: Feature Extraction and Scoring. For each phonetic symbol segment, extract key acoustic features and compare them with the standard features of the standard phonetic symbol sequence to calculate the sub-score S of the phonetic symbol. i .

[0023] Step 53: Calculate the word score. Average or weighted average the sub-scores of all phonetic symbols to obtain the total phonetic accuracy score S for the entire word.

[0024] The method may further include step S54: for sub-fractions S i For phonetic symbols below a preset threshold, the system will provide a highlight (e.g., red highlight) and automatically play the standard pronunciation of the phonetic symbol to guide the user to repeat and imitate, thus achieving closed-loop correction.

[0025] Correspondingly, this invention proposes a learning and assessment system for English word pronunciation through audio recognition. This system includes a speech recognition system and a word pronunciation learning and assessment system. The speech recognition system includes a memory, an audio signal preprocessing module, a vowel sequence recognition module, a consonant sequence recognition module, a database retrieval and processing module, and a whole phoneme recognition module. The memory pre-stores standard audio data for all phonetic symbols and the standard phonetic sequence for each word. The audio signal preprocessing module performs the aforementioned step S2 to perform preliminary processing of the audio signal data. The vowel sequence recognition module performs step S31, and the consonant sequence recognition module performs step S42... Step S32 involves recognizing vowel and consonant sequences respectively. The database retrieval and processing module performs the retrieval and data processing of the pre-stored data in the vowel and consonant phonetic symbol recognition steps in step S3. The whole phoneme recognition module performs precise and refined recognition of whole phonemes in step S4. The word pronunciation learning and evaluation system intelligently evaluates and scores the pronunciation of the recognized words read by the user, i.e., step S5 is executed. In addition, the English word pronunciation learning and evaluation system proposed in this invention through audio recognition can also have a speech rate calculation module to perform step S0, which identifies each person's pronunciation and reading habits.

[0026] Compared with existing technologies, this invention has the following significant advantages: In the process of word phonetic recognition, this invention adopts a hierarchical and precision-based sequential recognition approach. First, it performs coarse-precision recognition of words by recognizing vowel sequences or consonant sequences individually. If a user's word can be recognized by only vowel sequence recognition, consonant sequence recognition, or both, it directly proceeds to the word pronunciation learning and evaluation step. This method of first coarse recognition and then fine-grained recognition significantly improves recognition efficiency. This invention accurately locates pronunciation problems by breaking down word pronunciation to the phonetic symbol level and scoring it independently, precisely identifying the user's weak points in pronunciation and avoiding the ambiguity of overall scoring. The feedback is intuitive and effective, providing users with quantified scores and specific phonetic symbol problems, making the feedback information very intuitive and greatly improving learning efficiency and targeting. The phonetic symbol recognition method of this invention fully considers the differences in speaking speed and pronunciation habits among individuals, making speech recognition more accurate and efficient. Furthermore, the system can record frequently misused phonetic symbols by the user, thereby recommending words or exercises focusing on these phonetic symbols, achieving personalized learning. Attached Figure Description

[0027] Figure 1 is a flowchart of the method for learning English words through audio recognition provided in an embodiment of the present invention.

[0028] Figure 2 is a detailed flowchart of step S4 (phonetic symbol comparison and scoring) in an embodiment of the present invention. Detailed Implementation

[0030] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0031] A learning and evaluation method for English word pronunciation through audio recognition includes the following steps: Step S1: Construct a speech recognition system with a memory that pre-stores standard audio data for all phonetic symbols and standard phonetic symbol sequences for each word. The database contains idealized pronunciation samples for each phonetic symbol as a benchmark for comparison.

[0032] The phonetic symbol sequence of each word is explained here as a collection of commonly used words and a phonetic symbol classification annotation set for each word; the phonetic symbol classification annotation set: the phonetic symbol of each word is broken down into "vowel sequence" and "consonant sequence". For example, the phonetic symbol classification annotation of "banana" has three columns: the complete phonetic symbol: / bəˈnɑːnə / , the vowel sequence [ə,ɑː,ə], and the consonant sequence [b,n,n].

[0033] Step S2: The received user pronunciation audio is preprocessed using the speech recognition system. First, the input raw audio signal x(t) is pre-emphasized, framed, and windowed to eliminate the effects of the vocal cords and lips during phonation and reduce spectral leakage. Mel-frequency cepstral coefficients (MFCCs) are used to extract acoustic features from each frame of the audio signal; these features simulate the auditory perception characteristics of the human ear. The entire word, after acoustic feature extraction, is represented as a series of feature sequences O=(o1,o2,...o...). t ...,o T ).

[0034] Where t is the continuous time variable of the original audio, x(t) is the amplitude of the original audio signal at time t, and o t Let O represent the acoustic feature vector of the audio at frame t, where T represents the total number of frames after the word audio is segmented, and O is the time-frequency feature sequence of the entire audio, which is a T×D matrix, where D is the dimension of the feature vector.

[0035] Step S3: Vowel phonetic symbol recognition and consonant phonetic symbol recognition. Step S3 includes the following two steps: S31 and S32.

[0036] Step S31: Vowel Phonetic Symbol Recognition and Vowel Sequence Set Filtering; A phonetic symbol recognizer based on a Hidden Markov Model (HMM) is specifically trained for vowel phonetic symbols (ɑː, ɔː, ɜː, iː, uː, æ, e, ɪ, ɒ, ʊ, ə, etc.); Input the preprocessed feature sequence O, output the vowel sequence of the recognized word pronunciation, and store this vowel sequence as a vowel sequence set; Database retrieval: Based on the recognized vowel sequence set, retrieve all words in the phonetic symbol database that completely match the recognized vowel sequence set, and store this word set as set A; If the number of retrieved words is 1, then proceed to step S32 without further processing, and use this unique word as the recognized word to proceed to step S5; If the number of retrieved words is ≥2, then proceed to step S32.

[0037] Step S32: Consonant Phonetic Symbol Recognition and Consonant Sequence Set Filtering; Optimize the model based on the pronunciation features of consonant phonetic symbols (p, b, t, d, k, g, f, v, s, z, etc.) (such as the temporal impact of plosives and fricatives) to improve consonant recognition and obtain a trained SVM model; Input the preprocessed feature sequence O, output the consonant sequence of the recognized word pronunciation, and store the consonant sequence as a consonant sequence set; Search the database, and based on the recognized consonant sequence set, search the phonetic symbol database for all words that completely match the recognized consonant sequence set, and store the word set as set B; If the number of retrieved words is 1, use this unique word as the recognized word and proceed to the next step S5; If the number of retrieved words is ≥2, intersect set B and set A to obtain set C. If the number of words in set C is 1, use this unique word as the recognized word and proceed to the next step S5; If the number of words in set C is ≥2, proceed to the next step S4.

[0038] Step S4: Full phoneme recognition.

[0039] Acoustic modeling and phoneme sequence prediction: The feature sequence O is input into a Bi-LSTM (Bidirectional Long Short-Term Memory) network to obtain the hidden vector state ht containing contextual information for each frame, which can represent the dependencies between frames in the audio signal; the hidden vector state ht output for each frame is then processed. t A fully connected layer and a softmax layer are fed in to calculate the probability distribution P of each phoneme (phonetic symbol) at time t in the frame: in, Let be the phoneme label predicted at time t, which is a random variable. It could be any of the 48 English phonetic symbols; 'a' represents each of the monosyllabic phonetic symbols (excluding the 8 diphthongs) in the 48 English phonetic symbols; K is the total size of the set of phonetic symbols 'a', where K is 40; W a Let b be the weight matrix corresponding to the phonetic symbol a. a W is the bias term corresponding to the phonetic symbol 'a'. a b a The model was learned from a large amount of labeled data through backpropagation during the previous model training phase; h t Let be the hidden state vector output by the Bi-LSTM network at time t.

[0040] The maximum probability P for selecting each frame. t The corresponding phonetic symbols are used as the phonetic symbols identified at the corresponding time frame t, thus enabling the identification of the entire word's frame phonetic symbol sequence.

[0041] The above describes the feature vector o of each frame in the feature sequence O. tThe input is fed into a Bi-LSTM to obtain the hidden vector state h containing context information for each frame. t The standard iterative computation steps for Bi-LSTM involve first obtaining the forward and backward states through bidirectional LSTM processing: the forward LSTM scans the sequence from left to right, updating its internal state at each step. This state is a memory of the "past". The backward LSTM scans the sequence from right to left, updating its internal state at each step. This state is a memory of the "future"; both of the above LSTMs use forget gates, input gates, and output gates to determine what to remember, what to forget, and what to output; then at each time point t, the state representing the "past" is... and representing the "future" These are concatenated together to form the final context-aware representation of the hidden vector state ht. At this point, if... and If the dimension of each element is d, then the concatenated h... t The dimension is 2D.

[0042] It is worth noting that ht is an information-rich vector that not only contains information about the current frame x(t), but also encodes all the important context related to the current moment in the pronunciation of the word. This ensures that the subsequent Softmax layer can accurately determine what the specific phoneme is in the current context.

[0043] The integration process adopts a simple integration approach, which mainly solves the problem of mismatch between frame-level prediction and phoneme sequence length through experience and simple settings. It includes two steps: (1) First, in the above-mentioned frame phonetic symbol sequence that identifies the entire word, adjacent phonetic symbols identified between adjacent frames are merged, that is, multiple adjacent phonetic symbols can be merged into one phonetic symbol. For example, the phonetic symbols identified in frames 15-28 are all |ɪ|, so they are merged into one phonetic symbol |ɪ| to obtain the merged phonetic symbol sequence; (2) At the same time, the pronunciation duration of each phonetic symbol can be roughly determined in advance based on the pronunciation speed and habitual speech when learning English. Based on the phonetic symbol sequence after merging the above phonetic symbols, the system intelligently integrates the matter of whether two monophthongs need to be merged into a diphthong phonetic symbol. According to the recognition results, the phonetic symbols recognized in frames 23-34 are all |ɪ| (the recognition results before frame 33 and after frame 34 are not this phonetic symbol). Therefore, frames 23-34 are merged into a single phonetic symbol |ɪ|. However, it is found that the duration of the merged phonetic symbol |ɪ| is too short. Therefore, it is considered to merge the merged phonetic symbol |ɪ| with the phonetic symbols recognized before or after its duration (which must be monophthongs) to form a diphthong phonetic symbol.

[0044] After obtaining the final sequence of phonetic symbols as described above, the identified word is obtained.

[0045] The pronunciation duration of each phonetic symbol can be roughly determined based on the habitual pronunciation and speaking speed when learning English. Combined with the phonetic symbol sequence after merging the phonetic symbols, the system intelligently integrates the matter of whether two monophthongs need to be merged into a diphthong phonetic symbol. The habitual pronunciation and speaking speed when learning English can be determined through the following pre-arranged steps: Before step S1, step S0 is also included: The speech recognition system provides the user with a certain training English word stored in the system for reading aloud and provides a pronunciation speed judgment for the phonetic symbols. Specifically, the learner reads the word given by the speech recognition system at their usual reading or speaking speed. Since the speech recognition system knows in advance what word the user is reading, the speech recognition system only needs to identify the pronunciation duration of each phonetic symbol in the phonetic symbol sequence of the word pronunciation (this identification can be obtained through conventional methods such as the audio feature signals at each time point in the pronunciation signal, such as amplitude and frequency), and then store and record it to obtain the speech speed characteristics of each learner. This facilitates more targeted and accurate recognition of each learner's pronunciation practice in the future, effectively eliminating the influence of each person's speaking, pronunciation, and reading speed on the recognition results.

[0046] This invention provides a learning assessment method for English word pronunciation through audio recognition. It further involves comparing the word identified by the final phonetic sequence obtained through the aforementioned speech recognition system with its stored standard pronunciation, scoring the learner's pronunciation, and providing specific practice suggestions. Specifically, it includes the following step S5: English word pronunciation learning assessment. Step S5 specifically includes the following steps: Step 51: Based on the standard phonetic sequence of the identified word, this standard phonetic sequence has standard features corresponding to its time-frequency domain (the pre-stored standard phonetic sequence of the word is actually the time-frequency feature sequence of this standard phonetic sequence). Assume we obtain a user audio segment of the i-th phonetic symbol of a word read aloud by the user.

[0047] Step 52: Feature Extraction and Scoring. For each phonetic symbol segment, extract key acoustic features and compare them with the standard features of the standard phonetic symbol sequence to calculate the sub-score S of the phonetic symbol. i .

[0048] Step 53: Calculate the word score. Average or weighted average the sub-scores of all phonetic symbols to obtain the total phonetic accuracy score S for the entire word.

[0049] The method may further include step S54: for sub-fractions S i For phonetic symbols below a preset threshold, the system will provide a highlight (e.g., red highlight) and automatically play the standard pronunciation of the phonetic symbol to guide the user to repeat and imitate, thus achieving closed-loop correction.

[0050] Corresponding to the aforementioned English word pronunciation learning and assessment method, this invention also proposes an English word pronunciation learning and assessment system through audio recognition. This system includes a speech recognition system and a word pronunciation learning and assessment system. The speech recognition system includes a memory, an audio signal preprocessing module, a vowel sequence recognition module, a consonant sequence recognition module, a database retrieval and processing module, and a whole phoneme recognition module. The memory pre-stores standard audio data for all phonetic symbols and the standard phonetic symbol sequences for each word. The audio signal preprocessing module performs the aforementioned step S2 to perform preliminary processing of the audio signal data. The vowel sequence recognition module performs step S31 and consonant sequence... The column recognition module performs step S32 to recognize vowel sequences and consonant sequences respectively. The database retrieval and processing module performs step S3 to retrieve and process the pre-stored data in the memory during the vowel and consonant phonetic symbol recognition steps. The whole phoneme recognition module performs step S4 to perform precise and refined recognition of whole phonemes. The word pronunciation learning and evaluation system intelligently evaluates and scores the pronunciation of the recognized words read by the user, i.e., performs step S5. In addition, the English word pronunciation learning and evaluation system proposed in this invention through audio recognition can also have a speech rate calculation module to perform step S0, which identifies each person's pronunciation and reading habits.

[0051] Through the above methods, this invention achieves refined and quantitative evaluation and correction of English word pronunciation, effectively improving learning outcomes.

Claims

1. A learning assessment method for English word pronunciation through audio recognition, characterized in that, Includes the following steps: Step S1: Construct a speech recognition system with a memory that pre-stores standard audio data for all phonetic symbols and standard phonetic sequences for each word. Step S2: Preprocess the received user pronunciation audio using the speech recognition system. Pre-emphasize, frame, and window the input raw user pronunciation audio signal x(t) to eliminate the effects of the vocal cords and lips during pronunciation and reduce spectral leakage. Extract acoustic features from each frame of the audio signal using Mel-frequency cepstral coefficients (MFCCs). The entire word, after acoustic feature extraction, is represented as a series of feature sequences O=(o1,o2,...ot...,oT). Where t is the continuous time variable of the original audio, x(t) is the amplitude of the original audio signal at time t, ot represents the acoustic feature vector of the audio in frame t, T represents the total number of frames after the word audio is segmented, and O is the time-frequency feature sequence of the entire audio; Step S3: Vowel phonetic symbol recognition and consonant phonetic symbol recognition; Step S3 includes steps S31 and S32: Step S31: Vowel phonetic symbol recognition and vowel sequence set filtering; Step S31: Vowel phonetic symbol recognition and vowel sequence set filtering; A phonetic symbol recognizer based on a Hidden Markov Model (HMM) is used to train on vowel phonetic symbols; Input the preprocessed feature sequence O, and output the vowel sequence of the pronunciation of the recognized word. The vowel sequence is stored as a set of vowel sequences. Based on the identified vowel sequence set, all words that completely match the identified vowel sequence set are retrieved from the phonetic symbol database, and this set of words is stored as set A. If the number of retrieved words is 1, step S32 is skipped, and this unique word is taken as the identified word, proceeding to step S5. If the number of retrieved words is ≥2, step S32 is performed. Step S32: Consonant phonetic symbol recognition and consonant sequence set filtering; the model is optimized based on the pronunciation features of consonant phonetic symbols to improve consonant recognition accuracy, resulting in a trained SVM model; the preprocessed feature sequence O is input, and the pronunciation of the identified word is output. Consonant sequences are identified and stored as a consonant sequence set. Based on the identified consonant sequence set, all words that completely match the identified consonant sequence set are retrieved from the phonetic symbol database and stored as set B. If the number of retrieved words is 1, this unique word is taken as the identified word, and the process proceeds to step S5. If the number of retrieved words is ≥2, set B and set A are intersected to obtain set C. If the number of words in set C is 1, this unique word is taken as the identified word, and the process proceeds to the next step S5. If the number of words in set C is ≥2, the process proceeds to step S4. Step S4: Whole phoneme recognition; Step S5: English word pronunciation learning assessment.

2. The learning assessment method for English word pronunciation through audio recognition according to claim 1, characterized in that, Step S4, full phoneme recognition, specifically includes: acoustic modeling and phoneme sequence prediction: inputting the feature sequence O into a Bi-LSTM (Bidirectional Long Short-Term Memory) network to obtain the hidden vector state ht containing contextual information for each frame; feeding the hidden vector state ht output from each frame into a fully connected layer and a Softmax layer to calculate the probability distribution P belonging to each phoneme (phonetic symbol) at time t in that frame: in, Let be the phoneme label predicted at time t, which is a random variable. It could be any of the 48 English phonetic symbols; 'a' is any of the monosyllabic phonetic symbols among the 48 English phonetic symbols; K is the total size of the set of phonetic symbols 'a', K is 40; Wa is the weight matrix corresponding to phonetic symbol 'a', ba is the bias term corresponding to phonetic symbol 'a', Wa and ba were learned from a large amount of labeled data through backpropagation algorithm in the previous model training stage; ht is the hidden state vector output by the Bi-LSTM network at time t; the phonetic symbol corresponding to the maximum probability Pt in each frame is selected as the phonetic symbol recognized at the corresponding time t frame, and the frame phonetic symbol sequence of the entire word is recognized; adjacent merging phonetic symbols in the recognized frame phonetic symbol sequence are merged through simple integration; thus, the final recognized phonetic symbol sequence is obtained, and thus the recognized word is obtained.

3. The learning assessment method for English word pronunciation through audio recognition according to claim 2, characterized in that, The method of merging adjacent merging phonetic symbols in the identified frame phonetic symbol sequence through simple integration includes: firstly, merging adjacent phonetic symbols identified between adjacent frames in the above-mentioned frame phonetic symbol sequence that identifies the entire word, that is, multiple adjacent phonetic symbols can be merged into one phonetic symbol to obtain a merged phonetic symbol sequence; secondly, based on the pronunciation duration of each phonetic symbol determined in advance according to the user's habitual pronunciation speed when learning English, and combined with the above-mentioned merged phonetic symbol sequence, intelligently integrating whether two monophthongs need to be merged into a diphthong phonetic symbol.

4. The learning assessment method for English word pronunciation through audio recognition according to claim 3, characterized in that, Before step S1, there is also step S0: the speech recognition system provides the user with the specific training English words stored in the system to read aloud, and provides the speech rate judgment of the phonetic symbols; since the speech recognition system knows in advance what words the user is reading, the speech recognition system only needs to identify the pronunciation duration of each phonetic symbol in the phonetic symbol sequence of the word pronunciation, and then store and record it to obtain the speech rate characteristics of each trainee, so as to make subsequent pronunciation practice of each trainee more targeted and accurate.

5. The learning assessment method for English word pronunciation through audio recognition according to claim 3, characterized in that, Step S5 specifically includes the following steps: Step 51: Based on the standard phonetic symbol sequence of the identified words, this standard phonetic symbol sequence has standard features corresponding to its time-frequency domain (the standard phonetic symbol sequence of words pre-stored in the memory is actually the time-frequency feature sequence of this standard phonetic symbol sequence). Assume we have obtained the user audio segment of the i-th phonetic symbol of the word read by the user; Step 52: Feature extraction and scoring, for each phonetic symbol segment, extract key acoustic features and compare them with the standard features of the standard phonetic symbol sequence to calculate the sub-score Si of the phonetic symbol; Step 53: Word score calculation, average or weighted average all the sub-scores of the phonetic symbols to obtain the total phonetic accuracy score S of the entire word.

6. The learning assessment method for English word pronunciation through audio recognition according to claim 5, characterized in that, Step S5 further includes step S54: for phonetic symbols whose sub-fraction Si is lower than a preset threshold, the system provides a key prompt and automatically plays the standard pronunciation of the phonetic symbol to guide the user to follow along and imitate, and to make corrections.

7. A system for evaluating English word pronunciation learning using the audio recognition method described in any one of claims 1-5, characterized in that, The system includes a speech recognition system and a word pronunciation learning and evaluation system. The speech recognition system includes a memory, an audio signal preprocessing module, a vowel sequence recognition module, a consonant sequence recognition module, a database retrieval and processing module, and a whole phoneme recognition module. The memory stores standard audio data for all phonetic symbols and standard phonetic sequences for each word. The audio signal preprocessing module performs preliminary processing of the audio signal data in step S2. The vowel sequence recognition module performs step S31, and the consonant sequence recognition module performs step S32 to recognize vowel and consonant sequences respectively. The database retrieval and processing module retrieves and processes the pre-stored data in the memory in step S3 (vowel and consonant phonetic symbol recognition). The whole phoneme recognition module performs precise and refined recognition of the whole phonemes in step S4. The word pronunciation learning and evaluation system intelligently evaluates and scores the pronunciation of the recognized words as read by the user, i.e., performs step S5. In addition, the English word pronunciation learning and evaluation system proposed in this invention through audio recognition may also have a speech rate calculation module to perform step S0 and recognize each person's pronunciation and reading habits.

Images