Language learning system

Abstract

The present application provides a language learning system, which relates to a high-confidence audio quality detection technique. A data preprocessing system and an audio quality detection system are operated on a data processing server. The data preprocessing system includes an annotation system and an audio preprocessing system to generate an annotation and an audio data respectively according to a text and an audio of a user reading the text. The audio quality detection system scores the audio based on the annotation and the audio data. The annotation system includes an artificial intelligence model to generate the annotation in response to a condition that the text includes multi-sound words.

Description

Technical Field

[0001] This invention relates to language learning systems, and more particularly to the detection of audio quality of spoken language (Goodness of pronunciation, GOP). Background Technology

[0002] With the development of computer technology, language learning systems are becoming increasingly powerful. Users can prepare various texts for speaking practice. The system can provide feedback on the completeness, fluency, accuracy, etc., of the user's recordings, helping users improve their spoken language. This function is called Good Optical Program (GOP).

[0003] Improving the reliability of audio quality testing (GOP) and helping users improve their spoken language is a major issue in this field. Summary of the Invention

[0004] This invention proposes a language learning system with high-reliability audio quality detection (GOP).

[0005] A language learning system according to one embodiment of the present invention includes a data preprocessing system running on a data processing server and an audio quality detection system. The data preprocessing system includes an annotation system and an audio preprocessing system for generating annotations and audio data based on a text and an audio recording of a user reading the text. The audio quality detection system scores the audio based on the annotations and the audio data. The annotation system includes an artificial intelligence model that generates annotations in response to the text containing polyphonic words. The polyphonic words are words with different parts of speech and different pronunciations, or special words, including numbers or place names.

[0006] This AI model adapts to the presence of polyphonic words in the text, ensuring the annotation matches the ideal pronunciation, thereby improving the reliability of the audio quality detection system. User pronunciation errors can be accurately corrected.

[0007] In one implementation, the artificial intelligence model includes multiple audio-annotation models corresponding to different polyphonic words. In response to the text being word text and containing polyphonic words, the corresponding audio-annotation model operates to generate the annotation based on the audio.

[0008] In one implementation, the artificial intelligence model includes a sentence-labeling model. In response to the text being a sentence and containing polyphonic words, the sentence-labeling model operates to generate labels based on the sentence text.

[0009] In one embodiment, the language learning system further includes a human-computer interface, operated by a user terminal device, which allows the user to select text for reading aloud and recording, and upload the text and audio to the data processing server.

[0010] The present invention will be described in detail below with reference to specific embodiments and accompanying drawings. Attached Figure Description

[0011] Figure 1 Illustrated explanation of a language learning system (100);

[0012] Figure 2 The implementation architecture of the audio-annotation model for a specific word is illustrated using a block diagram.

[0013] Figure 3 An implementation architecture for a block diagram-annotation model 128;

[0014] Figure 4 The workflow of the illustration annotation system 112 according to one embodiment of the present invention;

[0015] Figure 5 A method for establishing a graphical language model 502 and an acoustic model 504;

[0016] Figure 6 This diagram illustrates one implementation of the audio quality inspection (GOP) system 120, in which the following is applied: Figure 5 The trained language model 502 and acoustic model 504;

[0017] Figure 7 Illustrated explanation of a language learning system 700;

[0018] Figure 8 An illustration of one implementation of an audio quality inspection (GOP) system 720 with multiple pronunciation correction capabilities; and

[0019] Figure 9 A flowchart illustrating the operation of the Audio Quality Inspection (GOP) system 720 with multi-pronunciation error correction capabilities.

[0020] The symbols in the attached diagram are briefly explained as follows:

[0021] 100: Language learning system; 102: Text; 104: Audio; 106: Application or webpage (human-computer interface); 108: Data processing server; 110: Data preprocessing system; 112: Labeling system; 114: Audio preprocessing system; 116: Labeling; 118: Audio data; 120: Audio quality detection system; 121: Scoring results; 122: Dictionary; 124: Artificial intelligence model; 126: A set of audio-labeling models; 128: Sentence-labeling model; 202: Input layer, audio input; 204: Feature extraction layer, extracting MFCCs features. 206: Normalization layer, normalizes MFCC features; 208: Deep Neural Network (DNN); 210: Fully connected layer; 212: Output layer, labels the output; 302: Functional block, constructs sentence vectors based on sentence text characteristics; 304: Functional block, determines whether word segmentation is needed; 306: Word segmentation system; 308: Prepares input vectors (and output vectors during training); 310: Embedding layer; 312: Long Short-Term Memory (LSTM) network used by the encoder; 314: Long Short-Term Memory (LSTM) network used by the decoder; 316: Fully connected layer; 31 8: Softmax activation function; S402…S420: Steps; 502: Language model; 504: Acoustic model; 506: Language model building system; 508: Acoustic model building system; 510: Training text; 512: Audio of standard pronunciation; 514: Training annotations; 516: Training audio data; 602: Forced alignment module; 604: Posterior probability calculation module; 606: Phoneme-level audio quality scoring module; 700: Language learning system; 702: Text; 704: Audio; 706: Application or webpage (human-computer interface); 708: Data processing server; 710: Data preprocessing system; 712: Annotation system; 714: Audio preprocessing system; 716: Annotation; 718: Audio data; 720: Audio quality detection system with multi-pronunciation error correction function; 802: Language model; 804: Acoustic model; 806: Speech activity detection system; 808: Recurrent neural network; 810: Forced alignment system; 812: Posterior probability calculation system; 814: Audio quality scoring system at specified level; 816: Invalid frame discrimination system; 818: Multi-pronunciation speech recognition system; S902…S912: Steps. Detailed Implementation

[0022] The following description illustrates various embodiments of the present invention, but is not intended to limit the scope of the invention. The actual scope of the invention should be defined in accordance with the claims.

[0023] Figure 1This diagram illustrates a language learning system 100. A user interface, such as an application / webpage 106, can run on a user-end device (e.g., a mobile phone, tablet, or personal computer). The user selects text 102 through this application / webpage 106 to record reading aloud, generating audio 104. The text 102 and audio 104 are uploaded to a data processing server 108 via the application / webpage 106 and processed by a data preprocessing system 110 running on the data processing server 108. The data preprocessing system 110 includes an annotation system 112 and an audio preprocessing system 114, generating annotations 116 and audio data (digital signals that a computer can process, such as speech feature sequences) 118, respectively. An audio quality detection (GOP) system 120 also runs on the data processing server 108, scoring the audio 104 based on the annotations 116 and the audio data 118. The scoring result 121 is sent back from the data processing server 108 to the application / webpage 106 for display and user reference to correct pronunciation. The scoring can display the completeness, fluency, accuracy, etc. of the audio 104 relative to the text 102.

[0024] In one implementation, English pronunciation is distinguished by 39 phonemes, details of which can be found in the published article "Use of Speech Recognition in Computer-assisted Language Learning". The ideal pronunciation of text 102 can be displayed by phonemes, i.e., annotation 116 generated by annotation system 112. Annotation 116 can be considered the benchmark for scoring by audio quality inspection (GOP) system 120, and will significantly affect the reliability of the scoring.

[0025] For example, the phoneme sequence of the text "You are my good friend" is:

[0026] [Y,UW1],[AA1,R],[M,AY1],[G,UH1,D],[F,R,EH1,N,D]

[0027] The audio data 118, converted from the user's audio 104 and labeled 116, is then fed into the audio quality inspection (GOP) system 120 for processing to score the audio 104. The resulting phoneme-level score might look like this:

[0028] [-8.263644,-9.056454], [-0.8452008,-6.122041], [-4.690601,-5.821236], [0,-2.156311,0], [-7.024531,-7.413582,-6.771776,-2.714007,-2.869504]

[0029] Using a model corresponding to the text phoneme sequence, the audio quality detection (GOP) system 120 can forcibly align the audio data 118 with the text phoneme sequence labeled 116, and then perform posterior probability calculations to generate the above score. A higher probability value (e.g., a negative value closer to zero) indicates a higher degree of matching. Scores below a certain threshold can be considered as reading errors. This phoneme-level scoring can meticulously remind the user which phonetic symbols are mispronounced and whether stress and unstress are correct. Only when labeled 116 is correct is the score reliable. In one implementation, a phoneme-level score less than -5 indicates a low degree of matching between the read audio and the stored correct phonetic symbols, i.e., a reading error. The closer the phoneme-level score is to 0, the more accurate the reading is considered.

[0030] This invention provides a highly accurate annotation system 112. In addition to operating with reference to a dictionary 122, the annotation system 112 is also designed with an artificial intelligence model 124, which enables the generated annotations 116 to adapt to various usage scenarios of polyphonic words and avoids incorrect scoring by the audio quality detection (GOP) system 120.

[0031] Polyphonic words have various forms, which can be due to different parts of speech or special words (such as numbers with different uses or place names). Taking a word as an example, it may have different pronunciations due to different parts of speech. For example, the noun and verb pronunciations of the word "contract" are different, and the stress is not the same. Regarding the word "contract", dictionary 122 can find two sets of phoneme sequences (numbers 0-2 are stress markers): "KAH0 NTR AE1 KT"; "K AA1 NTR AE2 KT". Unlike traditional annotation systems that default to the first set of phoneme sequences provided by the dictionary, artificial intelligence model 124 plays a role here, providing the correct phoneme sequence that matches the text or the user's practice context as annotation 116.

[0032] For another example, numbers may be used as dates or phone numbers, each with a special pronunciation. Taking phone numbers as an example, the emergency phone number "110" is habitually read as "yao yao ling", rather than "yi yi ling". Taking dates as an example, the date "January" is habitually read as "yi yue", not often read as "yao yue", and the date "February" is habitually read as "er yue", not often read as "liang yue". Taking money as an example, "2 yuan" is habitually read as "liang yuan", not often read as "er yuan". For different uses of corresponding numbers, the artificial intelligence model 124 will adaptively provide a suitable phoneme sequence as the annotation 116.

[0033] For another example, special words, such as place names, may also have special pronunciations, which is also a form of polysyllabic words.

[0034] For another example, taking the text "我喜欢都(拼音du)市生活,生活便利,机会较多,比如上海、成都(du)等都(dou)是比较好的大都(du)市" (I like urban life. Life is convenient and there are many opportunities. For example, Shanghai, Chengdu, etc. are all relatively good big cities) as an example, the discussion is as follows. The dictionary defaults the pronunciation of "都" as "dou", and the correct pronunciation of "du" by the user when reading the text may be judged as incorrect. The artificial intelligence model 124 of the present invention will also adaptively provide a suitable phoneme sequence as the annotation 116 for such text.

[0035] Considering the scene characteristics of polysyllabic word practice, the artificial intelligence model 124 designs a set of audio-annotation models (including multiple audio-annotation models corresponding to multiple polysyllabic words) 126 for word texts, and designs a sentence-annotation model 128 for sentence texts.

[0036] First, introduce the set of audio-annotation models 126 designed for word texts. Each polysyllabic word corresponds to an audio-annotation model. The audio-annotation models of all polysyllabic words are assembled into the set of audio-annotation models 126.

[0037] The words in word texts have no context to infer the word nature, and it is impossible to judge the ideal pronunciation from the text. The audio-annotation model proposed by the present invention determines the annotation 116 of word texts by referring to the audio 104 of user recordings to meet the current practice goals of users. For the corresponding polysyllabic word audio, the present invention selects the phoneme arrangement with the highest confidence as the annotation of the polysyllabic word, which can improve the annotation accuracy rate and further reduce the probability of mis-scoring. For example, for "都(du)", if it were a previous annotation, it might be annotated as (dou), but according to the confidence, the phoneme arrangement with the highest confidence for the word "成都" is "cheng du". Therefore, when the user correctly reads out "cheng du", it will not be misjudged as wrong like in the prior art.

[0038] For a polyphonic word, the input to the corresponding audio-annotation model is the user-recorded audio 104, and the output is the annotation 116 adapted to the user's practice goals (estimated from audio 104). Taking the polyphonic word text "contract" as an example (noun pronunciation / ′...),... / , verb pronunciation / In response, the audio-annotation model for the word "contract" operates, generating a text phoneme sequence representing the ideal pronunciation of "contract" based on the user's audio 104 (e.g., a recording of the user practicing the pronunciation of the noun "contract"), using a higher confidence level. This sequence is then used as annotation 116. In this way, the score received by the user practicing the pronunciation of "contract" is indeed based on the ideal pronunciation of "contract," allowing the user to correctly correct their pronunciation. For example, consider the polyphonic word "wind" (with multiple pronunciations, including / wInd / and / waInd / ). Traditional dictionary lookup methods often fail to find suitable annotations for the user's practice. Using this invention, the user can choose the annotation with higher confidence level as annotation 116, improving the accuracy of the annotation and enhancing the system's scoring accuracy for the user's pronunciation.

[0039] The following introduces a sentence-annotation model 128 designed for sentence text.

[0040] A sentence may contain multiple polyphonic words (including polyphonic words related to part of speech, or polyphonic words containing special characters such as numbers). The words in a sentence have context, allowing us to determine the part of speech or usage of polyphonic words and infer their ideal pronunciation. This invention employs a temporal recurrent neural network (RNN) with storage capabilities—such as a Long Short-Term Memory (LSTM) network—to implement the sentence-annotation model 128. The input to the sentence-annotation model 128 is the sentence text 102, and the output is the annotation 116 of the sentence text 102, which fully considers the appropriate pronunciation of polyphonic words in the sentence.

[0041] The implementation methods of various artificial intelligence models (126, 128) are also detailed below.

[0042] First, we will introduce the implementation methods of each audio-annotation model in 126.

[0043] Figure 2Illustrate the implementation architecture of an audio-annotation model for a specific word with a block diagram. Block 202 is the input layer; during the training phase, it receives the audio of various standard pronunciations of the specific word; after the model is established, it receives the audio 104 of the user's recording. Block 204 is the feature extraction layer, which extracts the features of the audio. For example, it extracts the features of Mel-Frequency Cepstral Coefficients (MFCCs). Block 206 is the normalization layer, which normalizes the MFCCs features. Block 208 is a deep neural network (DNN). Block 210 is a fully connected layer. Block 212 is the output layer; during the training phase, its output will be compared with the ideal annotation of the standard pronunciation to feedback and train the network parameters; after training is completed, this output is used to provide the annotation 116 that matches the audio 104.

[0044] Taking a polysyllabic word with three annotations A, B, and C as an example, after the audio 104 of the user's recording is input into the audio-annotation model corresponding to this polysyllabic word, the confidence levels of the three annotations A, B, and C will be simulated. The annotation with the highest confidence level is the annotation that matches the user's practice goal and is used as the annotation 116 for scoring.

[0045] Taking Chinese training as an example. The word "解" has multiple pronunciations. For example: 浑身解(xie, fourth tone)数; 解(jie, third tone)除; 解(jie, fourth tone)送. The word "参" has multiple pronunciations. For example: 参(can, first tone)加; 人参(shen, first tone); 参(cen, first tone)差不齐. These polysyllabic words can be annotated according to the one with the highest confidence level of the corresponding user's pronunciation. The same as the previous English "contract" example.

[0046] The implementation method of the sentence-annotation model 128 is further detailed below.

[0047] Figure 3 Illustrate an implementation architecture of the sentence-annotation model 128 with a block diagram. The sentence text needs to be processed by blocks 302 - 308 first. Block 302 constructs a sentence vector according to the characteristics of the sentence text. Block 304 determines whether to perform word segmentation. For example, Chinese needs word segmentation while English does not. Block 306 is a word segmentation system, which segments the sentence vector in response to the word segmentation requirement. Block 308 uses the sentence vector as the input vector for the subsequent network. During the training phase, block 308 will also prepare the output vector (such as the ideal annotation of the training text, and these training texts include various sentences with polysyllabic words) corresponding to the input vector to achieve the training goal. If the model is established for a language system that does not require word segmentation, the word segmentation judgment 304 and the word segmentation system 306 can be omitted.

[0048] From a linguistic perspective, a word is the smallest unit that can be spoken independently of semantics or pragmatics and carries a literal or practical meaning. For example, the Chinese character "人" (person) is a word that can exist independently of any semantics or pragmatics. This word itself represents "an animal evolved from ape that can make and use tools for labor and communicate with language." A word can also be recognized as the smallest unit that a speaker can express independently with a literal meaning. If further divided, it has no literal or practical meaning. The same is true for English. For example, "Walk" means to walk or take a walk. An English word is the smallest unit that can be spoken independently of any pragmatics or semantics and represents a unit called "practical meaning" (actual meaning, which can also be understood as action meaning) in English linguistics.

[0049] For example, in Chinese, "汽车" (car) consists of two characters but must be regarded as a single word. "汽" is just a character, not a word. "车" can be a word (usually referring to a car). "火车" (train) is a word and cannot be split into "火" (fire) and "车" (car) because the meaning will change after splitting. The same is true for English. "car" cannot be split into the letters "c", "a", and "r", and "train" cannot be split into the letters "t", "r", "a", "i", and "n". Generally, a word in Chinese can be composed of one or more characters and requires a word segmentation system. An English word usually refers to a single character.

[0050] Word segmentation is for the correct understanding of a sentence. Different word segmentations can result in different meanings. Although the basic expression unit in modern Chinese is the "word" (mostly two-character or multi-character words), due to slight differences in people's habitual grammar, it is difficult to distinguish the boundaries between words and phrases. For example, in the following examples, different word segmentation results will have different meanings. For example, in "乒乓球 / 拍卖 / 完了" (The ping-pong ball auction is over), the subject is "乒乓球" (ping-pong ball). In "乒乓 / 球拍 / 卖 / 完了" (The ping-pong racket is sold out), the subject is "乒乓球拍" (ping-pong racket). The word segmentation system 306 is necessary.

[0051] Examples of Chinese word segmentation are also discussed here. The text "小盛看到湖岸上的花草,一株不知名的小花引起了他的注意" (Xiaosheng saw the flowers and plants on the lakeshore, and an unknown little flower caught his attention) may have multiple word segmentation methods, such as: "小盛 / 看到 / 湖岸 / 上 / 的 / 花草 / ,一株 / 不知名 / 的 / 小花 / 引起 / 了 / 他的 / 注意" (Xiaosheng / saw / the lakeshore / on / the / flowers and plants / , a / unknown / one / little flower / caused / his / attention); "小盛 / 看到 / 湖 / 岸 / 上 / 的 / 花 / 草,一株 / 不 / 知名 / 的 / 小花 / 引起了 / 他的 / 注意" (Xiaosheng / saw / the lake / shore / on / the / flowers / grass, a / non- / well-known / little flower / caused his / attention); "小盛 / 看到 / 湖岸 / 上的 / 花 / 草,一株 / 不知名的 / 小花 / 引起了 / 他的 / 注意" (Xiaosheng / saw / the flowers and plants on the lakeshore / , a / unknown / little flower / caused his / attention). Different word segmentations of Chinese sentences will produce different meanings. In addition, for some special words, such as "place names", appropriate word segmentation is also needed.

[0052] Block 310 is the embedding layer. Blocks 312 and 314 are Long Short-Term Memory (LSTM) networks. The LSTM network in block 312 stores context and is used as the encoder. The LSTM network in block 314 stores context and is used as the decoder. The illustrated architecture simulates a seq2seq model, but is faster than a seq2seq layer and more accurate than a simple LSTM network. The encoding and decoding results of the LSTM networks in blocks 312 and 314 will be output as label 116 through a fully connected layer 316 and activation function 318. The normalized exponential function Softmax can be used as the activation function 318 in the model.

[0053] The sentence-labeling model 128 not only stores the context information of words in a sentence (e.g., the word before, before, after, and after the target word), but also has the ability to store the context information of the entire sentence. This allows for the effective prediction of polyphonic words in multiple sentences.

[0054] Figure 4 The workflow of the annotation system 112 according to an embodiment of the present invention is illustrated below. Step S402 receives text 102. Step S404 determines whether the text 102 is a word or a sentence. If it is a word, the process proceeds to step S406 to query dictionary 122 (or consider further factors) to determine whether this word text has only one unique annotation. If so, the process proceeds to step S408 to output it as annotation 116. In this design, special symbols such as numbers and dates, as well as words in dictionary 122 that have different pronunciations due to different parts of speech, are all considered polyphonic words without a unique annotation.

[0055] If there are multiple annotation candidates for the word text, the process proceeds to step S410, in which the audio 104 of the user reading the word text is input into the audio-annotation model corresponding to the word text 102 (one of the audio-annotation models 126 in the group), thereby generating a phoneme sequence that is close to the user's audio 104, and then outputting this phoneme sequence for annotation 116 in step S412.

[0056] If step S404 identifies the sentence text, the process proceeds to step S414 to query dictionary 122 to determine whether all words in the sentence text have only one unique annotation. If so, the process proceeds to step S416 to output the text phoneme sequence organized by querying dictionary 122 as annotation 116.

[0057] If not all words in the sentence text have a unique label, the process proceeds to step S418, inputting the sentence text into the sentence-labeling model 128 to generate a phoneme sequence, and then outputting this phoneme sequence for labeling 116 in step S420.

[0058] Taking the text "I like living in a metropolis because it's convenient and offers many opportunities. For example, Shanghai and Chengdu are good metropolises" as an example, after steps S418 and S420, "Chengdu" can be correctly identified as a place name and labeled as "Chengdu".

[0059] In one implementation, the audio quality inspection (GOP) system 120 employs language and acoustic models, not only to force the audio data 118 to align with a sequence of text phonemes, but also to calculate the posterior probability of a possible sequence of phoneme states resulting from the forced alignment. An annotation system 112 with an artificial intelligence model 124 can also help establish these language and acoustic models used within the audio quality inspection (GOP) system 120.

[0060] Figure 5 This illustrates one method for establishing the language model 502 and the acoustic model 504. Training the language model 502 requires a language model building system 506. Training the acoustic model 504 requires an acoustic model building system 508. A series of training texts 510 and audio recordings 512 of their standard pronunciations are input into the aforementioned data preprocessing system 110, where they are converted into training annotations 514 and training audio data 516 by the invention's special annotation system 112 (including the artificial intelligence model 124) and the aforementioned audio preprocessing system 114. The language model building system 506 trains the language model 502 based on the training annotations 514. The acoustic model building system 508 trains the acoustic model 504 based on the training annotations 514 and the training audio data 516. With this design, the language model building system 506 trains the language model 502 based on the highly accurate training annotations 514, resulting in a more perfect language model 502. Similarly, the acoustic model building system 508 also trains the acoustic model 504 based on the highly accurate training annotations 514, and the acoustic model 504 will also have better acoustic simulation effects.

[0061] Figure 6 This diagram illustrates one implementation of the audio quality inspection (GOP) system 120, in which the following is applied: Figure 5 The trained language model 502 and acoustic model 504 operate based on annotations 116 and audio data 118. Furthermore, the audio quality detection (GOP) system 120 includes a forced alignment module 602, a posterior probability calculation module 604, and a phoneme-level audio quality scoring module 606. The phoneme-level audio quality scoring module 606 corrects the user's pronunciation phoneme-by-phone. Other implementations may include word-level or sentence-level audio quality scoring modules, correcting the user's pronunciation word-by-word or sentence-by-sentence.

[0062] In addition, if training in Chinese, besides... Figure 3 The word segmentation design in Dictionary 122 also needs to be replaced with word-tag conversion. Such Chinese dictionaries can be custom-defined or be based on commercially available public dictionaries.

[0063] When a user reads a word with multiple pronunciations or a special word, or a sentence containing a word with multiple pronunciations or a special word, if the word with multiple pronunciations or a special word scores low, the smart terminal can further display all the marked pronunciations and explanations of the word in the human-computer interface such as the application / webpage 106 (which may be taken from the system database or searched from the Internet, and then the searched pronunciations and explanations are further displayed in the human-computer interface such as the application / webpage 106).

[0064] In some implementations, the artificial intelligence model 124 may only use the audio-annotation model 126 to process polyphonic words in the word text, without having the sentence-annotation model 128; such implementations are also within the scope of this invention.

[0065] In some implementations, the artificial intelligence model 124 may only use the sentence-annotation model 128 to process sentence text with polyphonic words, without having the audio-annotation model 126; such implementations are also within the scope of this invention.

[0066] Any technique that designs an artificial intelligence model in a labeling system to generate labels in response to text including polyphonic words (e.g., words that cannot be accurately labeled using only dictionary 122) falls within the scope of this invention. The design details of the artificial intelligence model may also have various variations, all of which should fall within the scope of this invention.

[0067] The various systems running on the data processing server 108 mentioned above include hardware and software designs, with software providing hardware implementations for high-speed computing.

[0068] In addition to adding an audio / sentence annotation model at the working end to improve the model's recognition accuracy, this invention also makes further improvements to the client terminal. In one embodiment, after the system identifies errors made by the user during reading aloud, it also clearly displays the user's own errors during reading aloud on the client interface.

[0069] When a user reads a text aloud, there may be several scenarios: a) reading normally, reading each word according to the text's requirements; b) omitting words; c) pronouncing more words than necessary; d) having background noise; e) pronouncing more phonetic symbols for individual words, such as pronouncing "good" as "goodness"; f) omitting phonetic symbols for individual words, such as pronouncing "goodness" as "good".

[0070] This invention specifically proposes a high-reliability scoring technique for multiple pronunciation scenarios such as c) and e). This invention also offers a solution for environmental noise, as in d).

[0071] Figure 7 This diagram illustrates a language learning system 700. Users can prepare text 702 for reading aloud, generating audio 704. A user interface, such as an application / webpage 706, can run on the user's device (e.g., mobile phone, tablet, personal computer). Users select text 702 through this interface to record reading aloud, generating audio 704. Text 702 and audio 704 can be uploaded to a data processing server 708 via the application / webpage 706, and processed by a data preprocessing system 710 running on the data processing server 708. The data preprocessing system 710 includes an annotation system 712 and an audio preprocessing system 714, generating annotations 716 and audio data (digital signals that a computer can process, such as speech feature sequences) 718, respectively. The data processing server 708 also runs an audio quality inspection (GOP) system 720 with multiple pronunciation correction functionality, which scores audio 704 based on annotations 716 and audio data 718. The scoring results will be sent back to the application / webpage 706 by the data processing server 708 for user reference to correct pronunciation. The score can display the audio 704 relative to the text 702 in terms of completeness, fluency, accuracy, etc.

[0072] In particular, the Audio Quality Inspection (GOP) system 720 with multiple pronunciation correction function has multiple pronunciation correction function, which will identify the user's multiple pronunciations.

[0073] Generally, the multiple pronunciations in user audio 704 cannot be found in the ideal pronunciation annotation 716 of text 702 for scoring purposes. Traditional techniques treat invalid frames that fail to match as meaningless content—for example, as silence. Traditional techniques are unable to identify multiple pronunciations in a user's reading.

[0074] For example, if the text is "good" and the user pronounces it as "goodness," omitting the syllable "ness." Similarly, if the text is "Good morning" and the user pronounces it as "Good morning, Marry," omitting the word "Marry." In traditional technologies, these extra "ness" or "Marry" pronunciations are treated as noise because there is no corresponding text label. Traditional voice quality detection (GOP) systems are unable to alert the user that they have mispronounced "ness" or "Marry." In contrast, the audio quality detection (GOP) system 720 of this invention, with its multiple pronunciation correction function, clearly detects these extra pronunciations of "ness" or "Marry" and correctly prompts the user to correct their pronunciation.

[0075] Figure 8 This diagram illustrates one implementation of an audio quality inspection (GOP) system 720 with multiple pronunciation correction capabilities, in which a language model 802 and an acoustic model 804 can be used to assist in the analysis of annotations 716 and audio data 718.

[0076] A voice activity detection (VAD) system 806 can run on an audio quality inspection (GOP) system 720 with multiple pronunciation correction capabilities. The VAD system 806 is used to filter noise and obtain valid human voice audio. The VAD system 806 uses a recurrent neural network (RNN) 808 with deep learning to filter noise. During the training phase, the RNN 808 takes human voice audio recorded in various noisy environments as input, and the training goal is to output clean human voice audio. One implementation method is to create a training set for a specific environment, such as a classroom, so that the RNN 808 can accurately filter out classroom environmental noise. The environmental noise in situation d) is thus properly addressed.

[0077] The audio quality inspection (GOP) system 720 with multiple pronunciation correction also runs a forced alignment system 810. After noise filtering, the audio data 718 is forcibly aligned with the annotations 716 of the text phoneme sequence. In this embodiment, the forced alignment system 810 operates based on the language model 802 and the acoustic model 804 to find the correspondence between each frame of the audio data 718 and the text phonemes (716). The number of audio frames corresponding to each phoneme can also be calculated by the acoustic model 804. In one embodiment, the forced alignment system 810 uses phoneme models corresponding to each phoneme in the annotations 716 in the acoustic model 804. Through these phoneme models, each frame of the audio data 718 can be determined frame by frame to determine which phonemes in the annotations 716 are closest to each other and which state of the corresponding phoneme they belong to.

[0078] Taking the CMU pronunciation dictionary as an example, each phoneme can have three stress markers: 0 indicates no stress, 1 indicates primary stress, and 2 indicates secondary stress. Each phoneme must also be indicated whether it is at the beginning (B), middle (I), or end (E) of a word, or a monosyllabic word (S). For example, the various states (phonetic symbol + stress + word position) of the phoneme "IY" and their corresponding numbers are as follows:

[0079]

[0080] Additionally, sentences can begin with the number 0, silence can be represented by SIL, and words not supported in the dictionary (OOV) can be represented by SPN.

[0081] The forced alignment system 810 is explained below. The result of forced alignment of the example text "make you sweet" and the user-recorded audio "make you sweet, I like cake" might be:

[0082] “1 1 1 1 1 1 1 1 227 227 227 227 165 165 165 165 165 165 165 165 165 165 220 220 220 220 220 220 220 220 220 220 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 320 320 320 320 320 320 320 320 320 320 320 320 320 279 279 279 279 279 279 279 279 279 279 333 333 333 333 333 333 333 209 209 209 209 209 209 209 209 209 209 209 209 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288288 1 ... 1 ...

[0083] Each number represents a phoneme corresponding to a frame.

[0084] After forced alignment (810), a "1" indicates that the text annotation 716 could not find a corresponding phoneme and is considered an invalid frame. The others are frames of pronunciation phonemes to be scored. For example, in the forced alignment results of the aforementioned text "make you sweet" and the user's recorded audio "makeyou sweet,I like cake", the first eight "1"s represent eight invalid frames, and the "227" repeated four times represents a four-frame long "M_B" phoneme ("M" pronunciation). The valid frame phoneme numbers "165"..."288" in the middle are other pronunciations, interspersed with 14 invalid frames ("1"). The last long string of 210 "1"s represents 210 invalid frames.

[0085] The audio quality inspection (GOP) system 720 with multi-pronunciation error correction function runs a posterior probability calculation system 812 and a specified level audio quality scoring system 814. In the forced alignment result, the valid frames are subjected to posterior probability calculation by the posterior probability calculation system 812, and then handed over to the specified level audio quality scoring system 814 for phoneme-by-phoneme scoring, word-by-word scoring, or sentence-by-sentence scoring.

[0086] As for the invalid frames ("1") in the forced alignment result, they will be further judged by an invalid frame discrimination system 816 running on the audio quality detection (GOP) system 720 with multiple pronunciation error correction function to determine whether these invalid frames correspond to multiple pronunciations.

[0087] The audio quality inspection (GOP) system 720 with multi-pronunciation error correction function can also run an automatic speech recognition (ASR) system 818. In response to the invalid frame discrimination system 816 identifying multiple pronunciations, the ASR system 818 further identifies the phonemes of the multiple frames with multiple pronunciations, and then analyzes the content of the multiple pronunciations.

[0088] Taking the Chinese text as an example: "I like the pace of life in cities (pinyin du), the convenience, and the many opportunities. For example, Shanghai and Chengdu are relatively good large cities (dou)." The user misread the text, emphasizing the rhythm, resulting in multiple pronunciations. Since the word "rhythm" was not present in the original document, it will receive a lower score in subsequent scoring. Furthermore, ASR (Automatic Speech Recognition) can further identify that the user was referring to the word "rhythm."

[0089] One implementation involves designing a multiple pronunciation discrimination threshold. If the number of consecutive invalid frames ("1") exceeds this threshold, the invalid frame discrimination system 816 considers this series of invalid frames to be multiple pronunciations. Conversely, it can be considered as silence. The audio quality detection (GOP) system 720 with multiple pronunciation correction function will send a notification back to the human-machine interface implemented by the application / web page 706 via the data processing server 708, prompting the user that their reading has multiple pronunciations.

[0090] In one implementation, the specified level audio quality scoring system 814 reflects the multiple pronunciations in the score, intuitively using the score to make the user aware of inaccurate readings or multiple pronunciations.

[0091] In particular, the multi-pronunciation speech recognition (ASR) 818, without text reference, converts the multiple frames considered to have multiple pronunciations into phonemes based on the acoustic model 804, and identifies what the multiple pronunciations are. The audio quality inspection (GOP) system 720 with multi-pronunciation error correction function will transmit the information about the multiple pronunciations back to the human-computer interface implemented by the application / web page 706 via the data processing server 708, directly notifying the user which phonetic symbols, characters, or words were read multiple times.

[0092] In one implementation, the multiple pronunciation discrimination threshold is set to 15. In the forced alignment result, a "1" appearing more than 15 times consecutively is no longer considered a silence. The invalid frame discrimination system 816 identifies this series of frames as multiple pronunciations. In addition, the number representing an invalid frame is not limited to "1"; the manufacturer can set a specific number to represent an invalid frame.

[0093] In one embodiment, the multiple pronunciation discrimination threshold is set based on the sampling rate of the audio / video 704.

[0094] In one implementation, the multiple pronunciation discrimination threshold is adjusted depending on the algorithm type of the speech activity detection (VAD) system 806.

[0095] The threshold for multiple pronunciations can be determined based on the specific audio recording environment or the length of the passage. If the passage is long and requires more effort to read, the reader may pause for longer periods, and the threshold for multiple pronunciations will be designed accordingly to be higher. Conversely, shorter passages can correspond to a lower threshold for multiple pronunciations. The rigor of the assessment can also be selected. A more stringent assessment standard can use a lower threshold for multiple pronunciations.

[0096] Taking the forced alignment result of the aforementioned text "make you sweet" and the user-recorded audio "make you sweet, I like cake" as an example, the long string of 210 invalid frames ("1") at the end may trigger the following calculation. The invalid frame discrimination system 816 identifies a string of 210 "1"s, determining that it far exceeds the multi-pronunciation discrimination threshold, and therefore no longer considers these multiple frames as silent, but as multi-pronunciation. The multi-pronunciation speech recognition (ASR) system 818 is activated, and based on the acoustic model 804, converts these 210 frames of audio data into the phonemes with the highest probability. The forced alignment result is thus converted as follows:

[0097] “1 1 1 1 1 1 1 1 227 227 227 227 165 165 165 165 165 165 165 165165165 220 220 220 220 220 220 220 220 220 220 220 1 1 1 1 1 1 1 1 1 1 1 1 11 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335 335335 335 335 335 335 335 320 320 320 320 320 320 320 320 320 320 320 320 320320 320 279 279 279 279 279 279 279 279 279 279 279 333 333 333 333 333 333333 333 333 333 333 209 209 209 209 209 209 209 209 209 209 209 209 209 288288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288 288288 102 102 102 102 102 102 102 102 102 102 102 102 102 102 102 102 102 102102 102 102 102 102 102 102 102 102 102 223 223 223 223 223 223 223 223 223223 223 223 223 223 223 223 101 101 101 101 101 101 101 101 101 101 101 101101 101 101 101 220 220 220 220 220 220 220 220 220 220 220 220 220 220 220220 220 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 219 165165 165 165 165 165 165 165 165220 220 220 220 220 220 220 220 220 220”

[0098] The first 8 frames of silence (the first 8 "1"s) and the middle 14 frames of silence (the middle 14 "1"s) are considered silent because they do not exceed the multiple pronunciation discrimination threshold. As for the last 210 frames of "1", they are converted into phonetic symbols "102"..."220" by the multiple pronunciation speech recognition (ASR) system 818, correctly identifying the multiple pronunciation as "I like cake". The audio quality detection (GOP) system 720 with multiple pronunciation error correction function will send a notification back to the human-machine interface implemented by the application / web page 706 via the network through the data processing server 708, prompting the user that "I like cake" was mispronounced.

[0099] Figure 9 This is a flowchart illustrating the operation of an audio quality inspection (GOP) system 720 with multiple pronunciation correction functionality. Step S902 forces the alignment of text annotations 716 and audio data 718. Step S904 determines whether the number of consecutive invalid frames exceeds the multiple pronunciation discrimination threshold. If it does, the process proceeds to step S906, which determines whether the multiple pronunciation speech recognition (ASR) system 818 is activated. If the system is set to activate the multiple pronunciation speech recognition (ASR) system 818, the process proceeds to step S908 to analyze multiple pronunciations. Step S910 scores the aligned valid pronunciations. Step S912 informs the user of the scoring results for the valid pronunciations and prompts the user about the content of the multiple pronunciations. The scoring results for the valid pronunciations may also reflect the status of multiple pronunciations.

[0100] If the multi-pronunciation speech recognition (ASR) system 818 is not started, the process skips step S908 and proceeds directly to step S910 to score the aligned valid pronunciations. Step S912 informs the user of the scoring results for the valid pronunciations, but only indicates that multi-pronunciations have occurred, without providing any explicit information about the multi-pronunciations. The scoring results for the valid pronunciations may also reflect the status of multi-pronunciations.

[0101] If step S904 determines that the number of consecutive invalid frames does not exceed the multi-pronunciation discrimination threshold, then step S906 is skipped, and step S910 is directly performed to score the aligned valid pronunciations. Step S912 informs the user of the scoring results of the valid pronunciations.

[0102] In one implementation, scoring at the word or sentence level involves directly assigning a low score to the pronunciation of the word or sentence, reflecting the presence of multiple syllables or words.

[0103] In one implementation, phoneme-level scoring directly assigns low scores to words with multiple pronunciations. For example, the text "How are you?" and the recording "How old are you?". The human-computer interface can display that the user read "How old are you?", but the scores for each phoneme of the four words are [-1.26, -1.39], [-50, -50, -50], [-0.93, -0.26], and [-1.24, -2.84]. The word "old" with multiple pronunciations is visually reminded to the user with a low score [-50, -50, -50]. Other variations of the scoring method for multiple pronunciations are also possible.

[0104] In one implementation, the system gives a low score to places where the word "old" is pronounced multiple times (at this point, the user may not yet know why the score is low), and informs the user that the word in this location is a word with multiple pronunciations, and that the word with multiple pronunciations is "old". The scoring method reflecting multiple pronunciations can also be varied in other ways, such as requiring the user to reread the text that does not contain "old", or reminding the user in other ways.

[0105] Any technique that also judges invalid frames of forced alignment results and detects multiple pronunciations falls within the scope of protection of this invention.

[0106] The various systems running on the aforementioned data processing server 708 include hardware and software designs, with software providing hardware implementations for high-speed computing.

[0107] The above description is only a preferred embodiment of the present invention, but it is not intended to limit the scope of the present invention. Any person skilled in the art can make further improvements and changes on this basis without departing from the spirit and scope of the present invention. Therefore, the scope of protection of the present invention shall be determined by the scope defined in the claims of this application.

Claims

1. A language learning system, characterized in that, include: The data preprocessing system and audio quality detection system run on the data processing server. in: The data preprocessing system includes an annotation system and an audio preprocessing system, which generate annotations and audio data based on the text and the audio of the user reading the text. The audio quality testing system scores the audio based on the annotation and the audio data; and The annotation system includes an artificial intelligence model that generates annotations in response to text containing polyphonic words. These polyphonic words are words with different parts of speech and different pronunciations, or special words, including numbers or place names. This artificial intelligence model includes multiple audio-annotation models corresponding to different polyphonic words; and In response to the situation where the text is a word text and contains polyphonic words, the corresponding audio-annotation model operates, generating the annotation based on the confidence level of the audio.

2. The language learning system according to claim 1, wherein: In response to situations where the text includes polyphonic words with different pronunciations due to different parts of speech, numbers with different pronunciations due to different uses, or place names with special pronunciations, annotations including sequences of text phonemes are generated.

3. The language learning system according to claim 1, wherein: Each audio-annotation model is trained by taking multiple standard pronunciation audios of the corresponding word as input and taking the ideal annotations of these multiple standard pronunciation audios as output.

4. The language learning system according to claim 1, wherein, The input-to-output connections of each audio-annotation model are as follows: Deep neural networks; and Fully connected layer.

5. The language learning system according to claim 4, wherein, The audio-annotation models also include: The input layer receives the audio. The audio feature acquisition layer extracts the Mel-frequency cepstral coefficient features of the audio; and The normalization layer normalizes the Mel frequency cepstral coefficient features for input into the deep neural network.

6. The language learning system according to claim 1, wherein: This artificial intelligence model includes a sentence-labeling model; and In response to the situation where the text is a sentence and the sentence contains polyphonic words, the sentence-labeling model operates to generate the label based on the sentence text.

7. The language learning system according to claim 6, wherein: The sentence-labeling model is trained using training text as input, where each training text is a sentence containing polyphonic words; and The sentence-labeling model is trained with the ideal labels of each training text as output.

8. The language learning system according to claim 7, wherein, The sentence-labeling model includes: The input layer converts sentence text, including polyphonic words, into sentence vectors, which are then processed to generate corresponding ideal annotations.

9. The language learning system according to claim 8, wherein, This sentence-annotation model also includes: The word segmentation system responds to word segmentation requirements by segmenting sentence vectors into words, and then processes them to generate corresponding ideal annotations.

10. The language learning system according to claim 6, wherein, The input to output of this sentence-labeling model is concatenated as follows: Embedded layer; Long Short-Term Memory Network as an encoder; Long Short-Term Memory Network as Decoder; and Fully connected layer.

11. The language learning system according to claim 10, wherein: The sentence-labeling model uses the Softmax activation function.

12. The language learning system according to claim 11, wherein, This sentence-annotation model also includes: The input layer converts sentence text, including polyphonic words, into sentence vectors to be input into the embedding layer.

13. The language learning system according to claim 12, wherein, This sentence-annotation model also includes: The word segmentation system responds to word segmentation requirements by segmenting the sentence vector into words before inputting them into the embedding layer.

14. The language learning system according to claim 1, wherein: The annotation system also includes a dictionary; and The annotation system queries the dictionary to determine whether the text contains polyphonic words, thus activating the artificial intelligence model.

15. The language learning system according to claim 14, wherein: The annotation system also determines whether the text is word text or sentence text; When the text is a word text, the annotation system queries the dictionary to determine whether the word text is a polyphonic word; and When the word in the text is not a polyphonic word, the annotation system uses the dictionary query results as the annotation.

16. The language learning system according to claim 15, wherein: When the text is a sentence, the annotation system queries the dictionary to determine whether the sentence contains polyphonic words; When there are no polyphonic words in the sentence text, the annotation system uses the dictionary query results as the annotation.

17. The language learning system according to claim 16, wherein, This artificial intelligence model includes: Sentence-labeling model; and In response to the situation where the text is a sentence and the sentence contains polyphonic words, the sentence-labeling model operates to generate the label based on the sentence text, including selecting labels for polyphonic words or special words with different parts of speech and pronunciations that appear in the sentence based on the context of the sentence.

18. The language learning system according to claim 1, wherein: The audio quality detection system includes a language model and an acoustic model, which are used to force the audio data to align with the annotation including the text phoneme sequence, and calculate the posterior probability of the possible phoneme state sequence generated by the forced alignment to score the audio.

19. The language learning system according to claim 18, wherein: The language model building system for training the language model and the acoustic model building system for training the acoustic model use training annotations generated by the annotation system, which includes the artificial intelligence model.

20. The language learning system according to claim 1, further comprising: The human-computer interface, run by the user terminal device, allows the user to select text for reading aloud and recording, and upload the text and audio to the data processing server.

Images