Pronunciation error detection method and apparatus, electronic device, and storage medium
By performing phoneme sequence recognition and alignment on the spoken and text texts, and combining text and speech features for pronunciation error detection, the problem of poor error detection in existing methods is solved, and more efficient pronunciation error detection and diagnosis is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- IFLYTEK CO LTD
- Filing Date
- 2022-12-29
- Publication Date
- 2026-06-05
AI Technical Summary
Existing pronunciation error detection methods suffer from poor detection performance and incomplete diagnosis. In particular, methods based on forced alignment techniques and deep neural networks trained on phoneme recognition networks are not effective at segment-level error detection.
By acquiring the spoken and text text, phoneme sequence recognition and alignment are performed to obtain the probability distribution of misreading types. Then, the pronunciation error detection is performed by combining the text phoneme sequence and speech features, including determining text features, fusion features and speech features, and performing pronunciation error classification, error type identification and misreading content diagnosis.
It improves the accuracy and reliability of pronunciation error detection, avoids missed detections, and is not limited by the complexity of segment-level modeling, providing rich error detection information.
Smart Images
Figure CN115985342B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer-aided pronunciation training technology, and in particular to a pronunciation error detection method, device, electronic device, and storage medium. Background Technology
[0002] During pronunciation training, learners are asked to read a prepared text. The CAPT (Computer-Assisted Pronunciation Training) system can detect pronunciation errors in the reading and provide appropriate feedback based on the reading and the text.
[0003] Currently, the mainstream pronunciation error detection method is based on the forced alignment technique. This method requires pre-building possible error decoding paths, but because it is difficult to exhaustively list all error decoding paths, it suffers from poor error detection and incomplete diagnosis. Summary of the Invention
[0004] This invention provides a pronunciation error detection method, apparatus, electronic device, and storage medium to address the shortcomings of poor pronunciation error detection in the prior art.
[0005] This invention provides a pronunciation error detection method, comprising:
[0006] Get the text and audio of the text to be read aloud;
[0007] The spoken text is subjected to phoneme sequence recognition to obtain a speech phoneme sequence;
[0008] Align the speech phoneme sequence with the text phoneme sequence of the read-out text to obtain the probability distribution of misreading types of each phoneme in the text phoneme sequence;
[0009] Pronunciation error detection is performed based on the text phoneme sequence, the probability distribution of misreading types, and the speech features of the read-out speech.
[0010] According to a pronunciation error detection method provided by the present invention, the pronunciation error detection is performed based on the text phoneme sequence, the probability distribution of mispronunciation types, and the speech features of the read-out speech, including:
[0011] Based on each phoneme in the text phoneme sequence, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence, text features are determined.
[0012] Based on the correlation between the text features and the speech features, the fusion features are determined;
[0013] Based on the fusion features, pronunciation error detection is performed.
[0014] According to a pronunciation error detection method provided by the present invention, the step of determining text features based on each phoneme in the text phoneme sequence, the probability distribution of mispronunciation types of each phoneme, and the position of each phoneme in the text phoneme sequence includes:
[0015] Based on each phoneme itself, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence, the phoneme features of each phoneme are determined.
[0016] Based on the correlation between the phoneme features of each phoneme, phoneme feature interaction is performed to obtain the interactive phoneme features of each phoneme as the text features.
[0017] According to a pronunciation error detection method provided by the present invention, the step of performing phoneme sequence recognition on the read-out speech to obtain a speech phoneme sequence includes:
[0018] Based on the acoustic features of each frame in the read-aloud speech and the position of each frame in the read-aloud speech, the speech features of the read-aloud speech are determined.
[0019] Phoneme recognition and sequence decoding are performed on the speech features of the read-out speech to obtain the speech phoneme sequence.
[0020] According to a pronunciation error detection method provided by the present invention, determining the speech features of the read-out speech based on the acoustic features of each frame in the read-out speech and the position of each frame in the read-out speech includes:
[0021] Based on the acoustic features of each frame in the read-aloud speech and the position of each frame in the read-aloud speech, the basic features of each frame are determined;
[0022] Based on the correlation between the basic features of each frame, basic feature interaction is performed to obtain the higher-order features of each frame as the speech features.
[0023] According to a pronunciation error detection method provided by the present invention, the step of aligning the speech phoneme sequence with the text phoneme sequence of the read-out text to obtain the probability distribution of mispronunciation types of each phoneme in the text phoneme sequence includes:
[0024] Align each speech phoneme sequence with the text phoneme sequence to obtain the phoneme error detection results corresponding to each speech phoneme sequence.
[0025] Based on the misreading types of each phoneme in the phoneme error detection results corresponding to each speech phoneme sequence, the probability distribution of misreading types of each phoneme in the text phoneme sequence is statistically obtained.
[0026] According to a pronunciation error detection method provided by the present invention, the pronunciation error detection includes:
[0027] Perform at least one of the following: classify pronunciation errors, identify pronunciation error types, and diagnose misreading content.
[0028] The present invention also provides a pronunciation error detection device, comprising:
[0029] The acquisition unit is used to acquire the text and audio of the reading aloud.
[0030] The recognition unit is used to perform phoneme sequence recognition on the read-out speech to obtain a speech phoneme sequence;
[0031] An alignment unit is used to align the speech phoneme sequence with the text phoneme sequence of the read-out text to obtain the probability distribution of misreading types of each phoneme in the text phoneme sequence.
[0032] The error detection unit is used to perform pronunciation error detection based on the text phoneme sequence, the probability distribution of the misreading type, and the speech features of the read-out speech.
[0033] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the pronunciation error detection method as described above.
[0034] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the pronunciation error detection method as described above.
[0035] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the pronunciation error detection method as described above.
[0036] The pronunciation error detection method, apparatus, electronic device, and storage medium provided by this invention obtain the probability distribution of mispronunciation types of each phoneme in the text phoneme sequence by aligning the speech phoneme sequence and the text phoneme sequence of the read text, and perform pronunciation error detection by combining the text phoneme sequence, speech features, and mispronunciation type probability distribution, which can effectively improve the error detection performance and avoid missed detection problems. Attached Figure Description
[0037] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0038] Figure 1 This is one of the flowcharts of the pronunciation error detection method provided by the present invention;
[0039] Figure 2 This is a flowchart illustrating step 140 in the pronunciation error detection method provided by the present invention;
[0040] Figure 3 This is a flowchart illustrating step 120 in the pronunciation error detection method provided by the present invention;
[0041] Figure 4 This is the second flowchart of the pronunciation error detection method provided by the present invention;
[0042] Figure 5 This is a schematic diagram of the pronunciation error detection and diagnosis model provided by the present invention;
[0043] Figure 6 This is a schematic diagram of the pronunciation error detection device provided by the present invention;
[0044] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation
[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.
[0046] In related technologies, methods for pronunciation error detection are mainly divided into two categories:
[0047] The first type of method, based on forced alignment, first generates possible pronunciation paths from the text being read aloud. Then, it decodes the path with the highest score using an acoustic model. On this path, it combines the acoustic model output to calculate phoneme confidence scores such as phoneme likelihood, likelihood ratio, or GOP (Goodness of Pronunciation), thus obtaining segment-level pronunciation detection results. However, this method requires pre-constructing possible erroneous decoding paths. Since it is difficult to exhaustively enumerate all erroneous decoding paths, it suffers from poor error detection and incomplete diagnosis.
[0048] The second type involves training a phoneme recognition network using a deep neural network and detecting pronunciation by aligning the obtained phoneme recognition results with the target phoneme sequence. However, implementing the phoneme recognition network requires modeling fine-grained phoneme information. Due to the high complexity of segment-level modeling, this method does not perform well in segment-level error detection and diagnosis.
[0049] To optimize the pronunciation error detection effect, this invention provides a pronunciation error detection method. Figure 1 This is one of the flowcharts illustrating the pronunciation error detection method provided by the present invention, such as... Figure 1 As shown, the method includes:
[0050] Step 110: Obtain the text and audio for reading aloud.
[0051] Here, the text to be read aloud is the text corresponding to the audio recording to be checked for errors. The audio recording is the voice data collected and recorded from the user's pronunciation of the audio recording, i.e., the voice recording that needs to be checked for pronunciation errors.
[0052] Step 120: Perform phoneme sequence recognition on the read-out speech to obtain a speech phoneme sequence.
[0053] Specifically, for the acquired spoken text used for pronunciation error detection, the phoneme sequence corresponding to the spoken text can be obtained through phoneme sequence recognition. Here, in order to distinguish it from the phoneme sequence corresponding to the spoken text, the phoneme sequence corresponding to the spoken text is denoted as the speech phoneme sequence, and the phoneme sequence corresponding to the spoken text is denoted as the text phoneme sequence.
[0054] The acquisition of speech phoneme sequences can be achieved using phoneme recognition algorithms, such as conventional speech recognition algorithms. In the process of speech recognition of read-aloud speech, the speech phoneme sequence of the read-aloud speech is usually obtained first, and then the text to be recognized is determined based on the speech phoneme sequence. That is, the speech phoneme sequence can be considered an intermediate result of speech recognition of read-aloud speech. For example, the speech phoneme sequence of read-aloud speech can be obtained using the common speech recognition algorithm CTC (Connectionist Temporal Classification). It is understandable that phoneme sequence recognition of read-aloud speech may yield multiple phoneme sequences. In this case, the top 5 or top 3 phoneme sequences, sorted by confidence level, can be selected as the final speech phoneme sequence.
[0055] Step 130: Align the speech phoneme sequence with the text phoneme sequence of the read-out text to obtain the probability distribution of misreading types of each phoneme in the text phoneme sequence.
[0056] Specifically, the speech phoneme sequence and the text phoneme sequence are derived from the spoken audio and the spoken text, respectively. The speech phoneme sequence reflects the pronunciation during actual reading, while the text phoneme sequence reflects the standard pronunciation. By aligning the speech phoneme sequence and the text phoneme sequence, a comparison can be made between the actual pronunciation and the standard pronunciation, thereby determining the probability distribution of mispronunciation types for each phoneme in the text phoneme sequence.
[0057] Understandably, for multiple speech phoneme sequences obtained from phoneme sequence recognition, they can be aligned and compared with the text phoneme sequence separately. By statistically analyzing the types of mispronunciations present in the alignment comparison between each speech phoneme sequence and the text phoneme sequence, the probability distribution of mispronunciation types for each phoneme in the text phoneme sequence can be determined. Here, the probability distribution of mispronunciation types for each phoneme in the text phoneme sequence reflects whether there are pronunciation errors in each phoneme in the text phoneme sequence, what specific pronunciation errors exist, and the probability of each type of pronunciation error.
[0058] Step 140: Based on the text phoneme sequence, the probability distribution of misreading types, and the speech features of the read-out speech, pronunciation error detection is performed.
[0059] Specifically, the speech features of a reading aloud can be intermediate features extracted during the process of phoneme sequence recognition of the reading aloud. It can be understood that speech features also cover the pronunciation of the reading aloud.
[0060] After obtaining the probability distribution of misreading types, pronunciation error detection can be performed by combining the text phoneme sequence that reflects standard pronunciation, the speech features that reflect actual pronunciation, and the probability distribution of error types that reflect possible pronunciation errors. In this process, the aforementioned text phoneme sequence, speech features, and misreading type probability distribution can be fused and encoded before being fed into a classifier to obtain classification results, such as whether a misreading exists, the type of misreading, and the content of the misreading diagnosis.
[0061] The method provided in this invention obtains the probability distribution of misreading types of each phoneme in the text phoneme sequence by aligning the speech phoneme sequence and the text phoneme sequence of the read text, and performs pronunciation error detection by combining the text phoneme sequence, speech features and the probability distribution of misreading types, which can effectively improve the error detection performance and avoid missed detection problems.
[0062] It is understood that the method provided in the embodiments of the present invention does not require enumerating possible error decoding paths, thus avoiding the problem of missed detection, and also does not require segment-level modeling, so it is not limited by the complexity of segment-level modeling, thereby ensuring the error detection effect.
[0063] Based on the above embodiments, Figure 2This is a flowchart illustrating step 140 of the pronunciation error detection method provided by the present invention, as follows: Figure 2 As shown, step 140 includes:
[0064] Step 141: Based on each phoneme in the text phoneme sequence, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence, determine the text features.
[0065] Specifically, each phoneme in the text phoneme sequence can reflect the standard pronunciation. Encoding each phoneme in the text phoneme sequence and fusing it with the probability distribution of mispronunciation type of each phoneme and the encoding of the position of each phoneme in the text phoneme sequence can provide rich prior information for pronunciation error detection, thereby improving the accuracy of pronunciation error detection.
[0066] The text features here can be obtained by superimposing the phoneme codes of each phoneme in the text phoneme sequence with the probability distribution of misreading types and position codes of each phoneme, or by further extracting features based on the superimposed phoneme codes, misreading type probability distributions and position codes of each phoneme. This embodiment of the invention does not specifically limit this.
[0067] Step 142: Determine the fusion features based on the correlation between the text features and the speech features.
[0068] Step 143: Based on the fusion features, perform pronunciation error detection.
[0069] Specifically, after obtaining the text features, the correlation between the text features and the speech features can be calculated. Understandably, this correlation reflects the relationship between the standard pronunciation and the actual pronunciation for each phoneme. By fusing the text features and speech features based on this correlation, a fused feature reflecting the difference between the standard pronunciation and the actual pronunciation can be obtained. Here, the correlation between the text features and speech features can be obtained through feature interaction via an attention mechanism.
[0070] After obtaining the fused features, pronunciation error detection can be performed based on these features. For example, the fused features can be input into a classifier to obtain the classification results of the pronunciation error detection based on the fused features.
[0071] The method provided in this invention fuses the phonemes themselves in the text phoneme sequence, the probability distribution of mispronunciation types of each phoneme, and the position of each phoneme in the text phoneme sequence to obtain text features containing rich prior information. Based on this, pronunciation error detection can be performed, which can effectively improve the reliability of pronunciation error detection.
[0072] Based on any of the above embodiments, step 141 includes:
[0073] Based on each phoneme itself, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence, the phoneme features of each phoneme are determined.
[0074] Based on the correlation between the phoneme features of each phoneme, phoneme feature interaction is performed to obtain the interactive phoneme features of each phoneme as the text features.
[0075] Specifically, regarding the fusion of the phoneme encoding of each phoneme in a text phoneme sequence with the probability distribution of misreading types and the positional encoding of each phoneme, we can first fuse the phoneme encoding, misreading type probability distribution, and positional encoding of a single phoneme on a unit basis, thereby obtaining the phoneme features of a single phoneme. Here, fusing the phoneme encoding, misreading type probability distribution, and positional encoding of a single phoneme can be achieved by accumulating features from these phoneme encodings, misreading type probability distributions, and positional encodings, or by concatenating features from these phoneme encodings, misreading type probability distributions, and positional encodings, or by performing further feature extraction based on feature accumulation or concatenation. This embodiment of the invention does not specifically limit this approach.
[0076] After obtaining the phoneme features of each phoneme, the correlation between these phoneme features can be calculated. Based on the correlation between the phoneme features, phoneme feature interaction can be performed, which can broaden the scope and allow phoneme features at different positions to interact with each other. The resulting interactive phoneme features, compared to the original phoneme features, also include relevant information from phonemes at other positions. After obtaining the interactive phoneme features, the interactive phoneme features of all phonemes in the text phoneme sequence can be used as the overall text features.
[0077] Based on any of the above embodiments Figure 3 This is a flowchart illustrating step 120 of the pronunciation error detection method provided by the present invention, as follows: Figure 3 As shown, step 120 includes:
[0078] Step 121: Based on the acoustic features of each frame in the read-aloud speech and the position of each frame in the read-aloud speech, determine the speech features of the read-aloud speech.
[0079] Step 122: Perform phoneme recognition and sequence decoding on the speech features of the read-out speech to obtain the speech phoneme sequence.
[0080] Specifically, the acoustic features of each frame in the read-out speech can be obtained through signal processing tools. These acoustic features can be filterbank features, Mel Frequency Cepstrum Coefficient (MFCC) features, or Perceptual Linear Predictive (PLP) features, etc.
[0081] By jointly encoding the acoustic features of each frame and the position of each frame in the reading speech, high-order acoustic and linguistic information can be extracted, thus obtaining the speech features of the reading speech.
[0082] After obtaining the speech features of the read-out speech, phoneme recognition can be performed using these features, i.e., phoneme classification. Following this, sequence decoding is performed based on the phoneme recognition results to obtain the speech phoneme sequence. Here, sequence decoding can be implemented using Beam search or other decoding methods. From the multiple phoneme sequences obtained through phoneme decoding, the top predetermined number of phoneme sequences, sorted by confidence level, can be selected as the speech phoneme sequence. For example, the top 5 or top 3 phoneme sequences can be selected as the speech phoneme sequence, thereby obtaining rich speech and semantic information.
[0083] Based on any of the above embodiments, step 121 includes:
[0084] Based on the acoustic features of each frame in the read-aloud speech and the position of each frame in the read-aloud speech, the basic features of each frame are determined;
[0085] Based on the correlation between the basic features of each frame, basic feature interaction is performed to obtain the higher-order features of each frame as the speech features.
[0086] Specifically, for the fusion of acoustic features and positional encoding of each frame in a spoken text, we can first fuse the acoustic features and positional encoding of a single frame to obtain the basic features of that frame. During this process, we can extract features from the acoustic features of a single frame to reduce the feature dimensionality and thus reduce the complexity of subsequent calculations. Then, we can overlay or concatenate the extracted acoustic features with the positional encoding of the single frame to obtain the basic features of that single frame.
[0087] After obtaining the basic features of each frame, the correlation between these basic features can be calculated. Based on this correlation, basic feature interaction can be performed to extract higher-order acoustic and linguistic information. That is, higher-order features of each frame are obtained, and the overall higher-order features of all frames are used as the speech features. Here, basic feature interaction can be implemented through an attention mechanism, such as a multi-layered cascaded attention module, which can include attention layers, convolutional layers, and deconvolutional layers.
[0088] Based on any of the above embodiments, step 130 includes:
[0089] Align each speech phoneme sequence with the text phoneme sequence to obtain the phoneme error detection results corresponding to each speech phoneme sequence.
[0090] Based on the misreading types of each phoneme in the phoneme error detection results corresponding to each speech phoneme sequence, the probability distribution of misreading types of each phoneme in the text phoneme sequence is statistically obtained.
[0091] Specifically, for multiple speech phoneme sequences obtained from phoneme sequence recognition, each speech phoneme sequence can be edited and aligned with the text phoneme sequence to obtain the phoneme detection result corresponding to each speech phoneme sequence. For any speech phoneme sequence, the phoneme detection result corresponding to the threshold can be reflected as whether each phoneme in the text speech sequence is misread, and the type of misreading.
[0092] After obtaining the phoneme detection results corresponding to each speech phoneme sequence, the misreading type of each phoneme in the phoneme error detection results corresponding to each speech phoneme sequence can be statistically analyzed, thereby obtaining the probability distribution of misreading type of each phoneme in the text phoneme sequence.
[0093] For example, assuming there are 5 speech phoneme sequences, aligning each of the 5 speech phoneme sequences with the text phoneme sequence will yield the phoneme detection results corresponding to the 5 speech phoneme sequences shown below.
[0094]
[0095] In the table, the phoneme detection results are encoded as follows: 0 indicates correct pronunciation, 1 indicates mispronunciation (dissimilar pronunciation), 2 indicates mispronunciation (similar pronunciation), 3 indicates omission, 4 indicates word-final elision, 5 indicates pre-emphasis, and 6 indicates post-emphasis. Looking at the alignment of the speech phoneme sequence 3 with the text phoneme sequence, compared to speech phoneme sequence 3, the first and third phonemes h and l in the text phoneme sequence are both correctly pronounced, while the second phoneme... Misread as a dissimilar pronunciation The fourth phoneme If it is not read out, it means that there is a word ending elision. Therefore, the phoneme detection result corresponding to the speech phoneme sequence 3 can be encoded as 0104.
[0096] After obtaining the phoneme detection results corresponding to phoneme sequences 1-5 in the speech phoneme sequence, the misreading types corresponding to each phoneme in the text phoneme sequence can be statistically analyzed in the phoneme detection results, thereby obtaining the probability distribution of misreading types for each phoneme. Taking the last phoneme in the text phoneme sequence... For example, The error types in the phoneme detection results corresponding to the five speech phoneme sequences were 0, 0, 4, 1, and 0, respectively. Statistically, it can be seen that... A probability of 0.6 corresponds to error type 0, i.e., correct pronunciation; a probability of 0.2 corresponds to error type 1, i.e., mispronunciation of a dissimilar sound; and a probability of 0.2 corresponds to error type 4, i.e., elision of the final consonant. From this, we can deduce... The probability distribution of misread types, that is, the probabilities corresponding to error type codes 0-6 respectively, are [0.6 0.2 0 0 0.2 0 0]. T .
[0097] Based on any of the above embodiments, step 140, the pronunciation error detection, includes:
[0098] Perform at least one of the following: classify pronunciation errors, identify pronunciation error types, and diagnose misreading content.
[0099] Specifically, when performing pronunciation error detection based on text phoneme sequences, mispronunciation type probability distributions, and speech features of read-aloud speech, at least one of the following can be selectively performed: pronunciation error classification, pronunciation error type identification, and mispronunciation content diagnosis. Pronunciation error classification predicts whether each phoneme is pronounced correctly or incorrectly, and can be achieved through binary classification. Pronunciation error type identification predicts the type of pronunciation error for each phoneme, for example, through a seven-class classification, where the seven categories correspond to correct pronunciation, mispronunciation as a dissimilar pronunciation, mispronunciation as a similar pronunciation, omission, word-final elision, front-added pronunciation, and back-added pronunciation. Mispronunciation content diagnosis, when predicting the existence of a mispronunciation for a phoneme, predicts the actual pronunciation phoneme of that phoneme, and can also predict the type of the actual pronunciation phoneme, such as one of 11 types: front vowel, central vowel, back vowel, open / closed diphthong, central diphthong, plosive, fricative, affricate, nasal, lateral, and semi-vowel.
[0100] It is understood that, in the process of pronunciation error detection, the embodiments of the present invention perform at least one of the following: pronunciation error classification, pronunciation error type identification, and misreading content diagnosis, thereby obtaining error detection information of different granularities, and thus providing users with richer and more comprehensive error detection information.
[0101] Based on any of the above embodiments Figure 4 This is the second flowchart of the pronunciation error detection method provided by the present invention, as shown below. Figure 4 As shown, the process first involves acquiring the text and audio for reading aloud. For the text, preprocessing is performed to convert the word sequence into a text phoneme sequence, which is then input into the subsequent pronunciation detection and diagnosis model. Specifically, the text is cleaned by removing punctuation and retaining the word sequence. Then, the word sequence is converted into a phoneme sequence using a pronunciation dictionary, and finally, the phoneme sequence is converted into a digital encoding sequence using a phoneme mapping dictionary, serving as the text phoneme sequence.
[0102] For spoken text, acoustic features can be extracted and used as input for subsequent pronunciation detection and diagnosis models.
[0103] After obtaining the text phoneme sequence and the acoustic features of the read-aloud speech, these two can be input into the pronunciation error detection and diagnosis model. The pronunciation error detection and diagnosis model performs phoneme sequence recognition based on the acoustic features to obtain the speech phoneme sequence. The speech phoneme sequence and the text phoneme sequence are then aligned to obtain the probability distribution of mispronunciation types for each phoneme in the text phoneme sequence. Finally, pronunciation error detection is performed based on the text phoneme sequence, the probability distribution of mispronunciation types, and the speech features determined based on the acoustic features.
[0104] Figure 5 This is a schematic diagram of the pronunciation error detection and diagnosis model provided by the present invention, as shown below. Figure 5 As shown, for the acoustic features of the input spoken text, basic acoustic information can first be extracted using a Convolutional Neural Network (CNN) and the feature dimensionality reduced to decrease computational complexity. Then, the features extracted by the CNN are superimposed with positional encoding and input into an encoder to extract higher-order acoustic and linguistic information, thus obtaining the speech features. Here, the encoder can contain multi-layered cascaded attention submodules, which mainly consist of attention, convolution, and deconvolution. The speech features are then processed through a linear layer and softmax for phoneme classification, and beam search is used for decoding during the forward inference stage to obtain the top 5 phoneme recognition sequences for speech phoneme sequence identification, which are then used as the speech phoneme sequences.
[0105] For the input text phoneme sequence, an encoding operation can be performed to align it with the speech phoneme sequence to obtain the probability distribution of misreading types for each phoneme in the text phoneme sequence. This misreading type probability distribution is mapped to 512 dimensions using Error Prob Embedding and then accumulated together with the phoneme embedding and position encoding (Position Embedding) before being input into the decoder. Here, phoneme embedding refers to the vector representation of each phoneme, and position embedding represents the position vector representation. The accumulation of these three provides the model with more prior information, such as error detection information and phoneme position information, which is beneficial for the model to make accurate error detection and diagnosis. In the decoder, the input first passes through two layers of masked multi-head attention modules. This module enhances the model's perspective, ensuring that input information from different positions is relevant. Subsequently, the output of the multi-head self-attention module interacts with the speech features extracted by the encoder to focus on the corresponding acoustic information of each phoneme and extract information beneficial for error detection and diagnosis. Finally, after passing through a Positionwise FFN, it is connected to three classifiers to predict error detection and diagnostic information. The first classifier classifies pronunciation correctness, outputting a binary classification, which can be implemented using a single-layer DNN. The second classifier identifies the type of pronunciation error, outputting a 7-class classification, aiming to predict more granular error detection information, such as correct pronunciation, mispronunciation as a similar pronunciation, omission, etc., which can be implemented using a single-layer DNN. The third classifier diagnoses mispronunciations, outputting an 11-class diagnostic output; that is, when the model predicts a phoneme as mispronounced, it predicts its true pronunciation phoneme type, which can be implemented using a single-layer DNN.
[0106] The method provided in this invention can acquire error detection information at different granularities, such as correct pronunciation, mispronounced as a dissimilar sound, mispronounced as a similar sound, omitted pronunciation, elision at the end of a word, pre-accumulation, and post-accumulation. Furthermore, this method can not only identify multiple types of mispronunciation but also further diagnose the content of the mispronounced phonemes, obtaining the actual pronunciation phoneme types, such as 11 types: front vowel, central vowel, back vowel, open / closed diphthong, central diphthong, plosive, fricative, affricate, nasal, lateral, and semi-vowel. This method is not limited to English spoken pronunciation error detection but is also applicable to Chinese spoken pronunciation error detection.
[0107] Based on any of the above embodiments Figure 6 This is a schematic diagram of the pronunciation error detection device provided by the present invention, as shown below. Figure 6 As shown, the device includes:
[0108] The acquisition unit 610 is used to acquire the text to be read and the audio to be read.
[0109] The recognition unit 620 is used to perform phoneme sequence recognition on the read-out speech to obtain a speech phoneme sequence;
[0110] Alignment unit 630 is used to align the speech phoneme sequence with the text phoneme sequence of the read text to obtain the probability distribution of misreading type of each phoneme in the text phoneme sequence;
[0111] The error detection unit 640 is used to perform pronunciation error detection based on the text phoneme sequence, the probability distribution of the misreading type, and the speech features of the read-out speech.
[0112] The apparatus provided in this invention obtains the probability distribution of misreading types of each phoneme in the text phoneme sequence by aligning the speech phoneme sequence and the text phoneme sequence of the read text, and performs pronunciation error detection by combining the text phoneme sequence, speech features and the probability distribution of misreading types, which can effectively improve the error detection performance and avoid missed detection problems.
[0113] Based on any of the above embodiments, the error detection unit is used for:
[0114] Based on each phoneme in the text phoneme sequence, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence, text features are determined.
[0115] Based on the correlation between the text features and the speech features, the fusion features are determined;
[0116] Based on the fusion features, pronunciation error detection is performed.
[0117] Based on any of the above embodiments, the error detection unit is used for:
[0118] Based on each phoneme itself, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence, the phoneme features of each phoneme are determined.
[0119] Based on the correlation between the phoneme features of each phoneme, phoneme feature interaction is performed to obtain the interactive phoneme features of each phoneme as the text features.
[0120] Based on any of the above embodiments, the identification unit is used for:
[0121] Based on the acoustic features of each frame in the read-aloud speech and the position of each frame in the read-aloud speech, the speech features of the read-aloud speech are determined.
[0122] Phoneme recognition and sequence decoding are performed on the speech features of the read-out speech to obtain the speech phoneme sequence.
[0123] Based on any of the above embodiments, the identification unit is used for:
[0124] Based on the acoustic features of each frame in the read-aloud speech and the position of each frame in the read-aloud speech, the basic features of each frame are determined;
[0125] Based on the correlation between the basic features of each frame, basic feature interaction is performed to obtain the higher-order features of each frame as the speech features.
[0126] Based on any of the above embodiments, the alignment unit is used for:
[0127] Align each speech phoneme sequence with the text phoneme sequence to obtain the phoneme error detection results corresponding to each speech phoneme sequence.
[0128] Based on the misreading types of each phoneme in the phoneme error detection results corresponding to each speech phoneme sequence, the probability distribution of misreading types of each phoneme in the text phoneme sequence is statistically obtained.
[0129] Based on any of the above embodiments, the error detection unit is used for:
[0130] Perform at least one of the following: classify pronunciation errors, identify pronunciation error types, and diagnose misreading content.
[0131] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include a processor 710, a communications interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communications interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 can call logical instructions in the memory 730 to execute a pronunciation error detection method, which includes: acquiring a text to be read and a speech to be read; performing phoneme sequence recognition on the speech to obtain a speech phoneme sequence; aligning the speech phoneme sequence with the text phoneme sequence of the text to obtain a probability distribution of mispronunciation types for each phoneme in the text phoneme sequence; and performing pronunciation error detection based on the text phoneme sequence, the probability distribution of mispronunciation types, and the speech features of the speech.
[0132] Furthermore, the logical instructions in the aforementioned memory 730 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0133] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the pronunciation error detection method provided by the above methods. The method includes: acquiring a text to be read and a speech to be read; performing phoneme sequence recognition on the speech to obtain a speech phoneme sequence; aligning the speech phoneme sequence with the text phoneme sequence of the text to obtain a probability distribution of mispronunciation types for each phoneme in the text phoneme sequence; and performing pronunciation error detection based on the text phoneme sequence, the probability distribution of mispronunciation types, and the speech features of the speech.
[0134] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the pronunciation error detection method provided by the above methods. The method includes: acquiring a text to be read aloud and a speech to be read aloud; performing phoneme sequence recognition on the speech to obtain a speech phoneme sequence; aligning the speech phoneme sequence with the text phoneme sequence of the text to obtain a probability distribution of mispronunciation types for each phoneme in the text phoneme sequence; and performing pronunciation error detection based on the text phoneme sequence, the probability distribution of mispronunciation types, and the speech features of the speech to be read aloud.
[0135] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.
[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A pronunciation error detection method, characterized in that, include: Get the text and audio of the text to be read aloud; The spoken text is subjected to phoneme sequence recognition to obtain a speech phoneme sequence; Align the speech phoneme sequence with the text phoneme sequence of the read-out text to obtain the probability distribution of misreading types of each phoneme in the text phoneme sequence; Pronunciation error detection is performed based on the text phoneme sequence, the probability distribution of misreading types, and the speech features of the read-out speech.
2. The pronunciation error detection method according to claim 1, characterized in that, The pronunciation error detection based on the text phoneme sequence, the misreading type probability distribution, and the speech features of the read-out speech includes: Based on each phoneme in the text phoneme sequence, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence, text features are determined. Based on the correlation between the text features and the speech features, the fusion features are determined; Based on the fusion features, pronunciation error detection is performed.
3. The pronunciation error detection method according to claim 2, characterized in that, The process of determining text features based on each phoneme in the text phoneme sequence, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence includes: Based on each phoneme itself, the probability distribution of misreading types of each phoneme, and the position of each phoneme in the text phoneme sequence, the phoneme features of each phoneme are determined. Based on the correlation between the phoneme features of each phoneme, phoneme feature interaction is performed to obtain the interactive phoneme features of each phoneme as the text features.
4. The pronunciation error detection method according to claim 1, characterized in that, The step of performing phoneme sequence recognition on the read-out speech to obtain a speech phoneme sequence includes: Based on the acoustic features of each frame in the read-aloud speech and the position of each frame in the read-aloud speech, the speech features of the read-aloud speech are determined. Phoneme recognition and sequence decoding are performed on the speech features of the read-out speech to obtain the speech phoneme sequence.
5. The pronunciation error detection method according to claim 4, characterized in that, The process of determining the speech features of the read-out speech based on the acoustic features of each frame in the read-out speech and the position of each frame in the read-out speech includes: Based on the acoustic features of each frame in the read-aloud speech and the position of each frame in the read-aloud speech, the basic features of each frame are determined; Based on the correlation between the basic features of each frame, basic feature interaction is performed to obtain the higher-order features of each frame as the speech features.
6. The pronunciation error detection method according to claim 1, characterized in that, The step of aligning the speech phoneme sequence with the text phoneme sequence of the read-out text to obtain the probability distribution of misreading types for each phoneme in the text phoneme sequence includes: Align each speech phoneme sequence with the text phoneme sequence to obtain the phoneme error detection results corresponding to each speech phoneme sequence. Based on the misreading types of each phoneme in the phoneme error detection results corresponding to each speech phoneme sequence, the probability distribution of misreading types of each phoneme in the text phoneme sequence is statistically obtained.
7. The pronunciation error detection method according to any one of claims 1 to 6, characterized in that, The pronunciation error detection includes: Perform at least one of the following: classify pronunciation errors, identify pronunciation error types, and diagnose misreading content.
8. A pronunciation error detection device, characterized in that, include: The acquisition unit is used to acquire the text and audio of the reading aloud. The recognition unit is used to perform phoneme sequence recognition on the read-out speech to obtain a speech phoneme sequence; An alignment unit is used to align the speech phoneme sequence with the text phoneme sequence of the read-out text to obtain the probability distribution of misreading types of each phoneme in the text phoneme sequence. The error detection unit is used to perform pronunciation error detection based on the text phoneme sequence, the probability distribution of the misreading type, and the speech features of the read-out speech.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the pronunciation error detection method as described in any one of claims 1 to 7.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the pronunciation error detection method as described in any one of claims 1 to 7.