Sound correction method, sound correction device and non-transient storage medium

Abstract

A pronunciation correction method, a pronunciation correction device, and a non-transient storage medium. The pronunciation correction method includes: obtaining a word and first audio data; performing a pronunciation diagnosis operation on the first audio data based on the word to generate a pronunciation diagnosis result. The standard pronunciation of the word includes at least one standard phoneme, and the pronunciation diagnosis operation includes: aligning the first audio data with the standard pronunciation based on a first acoustic model to obtain the time boundary of each standard phoneme; determining a score for each standard phoneme based on the time boundary of each standard phoneme; performing a recognition operation on the first audio data based on a second acoustic model to obtain a decoded phoneme sequence and the time boundary of each decoded phoneme; determining a score for each decoded phoneme based on the time boundary of each decoded phoneme; determining a correspondence between each standard phoneme and each decoded phoneme; and generating a pronunciation diagnosis result based on the correspondence, the score of each standard phoneme, and the score of each decoded phoneme.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a sound correction method, a sound correction device, and a non-transient storage medium. Background Art

[0002] With the advancement of science and technology, more and more language learners are using language learning applications (APPs) to assist with their language learning. In some language learning apps, the app provider sends learning materials to the client via the internet, and users access the materials through the client to conduct their learning. In addition to learning grammar and vocabulary, improving pronunciation is also a crucial part of the language learning process. Typically, users improve their pronunciation through methods such as reading aloud and following along. However, in many cases, users have no way of knowing whether their pronunciation is accurate. Summary of the Invention

[0003] At least some embodiments of the present disclosure provide a method for correcting pronunciation. The method comprises: obtaining a word and first audio data; performing a pronunciation diagnosis operation on the first audio data based on the word to generate a pronunciation diagnosis result; wherein the standard pronunciation of the word includes at least one standard phoneme; performing the pronunciation diagnosis operation on the first audio data based on the word to generate the pronunciation diagnosis result, including: performing an alignment operation on the first audio data with the standard pronunciation based on a first acoustic model to obtain the time boundary of each standard phoneme in the standard pronunciation in the first audio data; determining the time boundary of each standard phoneme according to the audio segment determined by the time boundary of each standard phoneme, and scores of accurate phonemes; performing a recognition operation on the first audio data based on a second acoustic model to obtain a decoded phoneme sequence and a time boundary of each decoded phoneme in the decoded phoneme sequence in the first audio data, wherein the decoded phoneme sequence includes at least one decoded phoneme; determining a score of each decoded phoneme according to an audio segment determined by the time boundary of each decoded phoneme; determining a correspondence between each standard phoneme in the standard pronunciation and each decoded phoneme in the decoded phoneme sequence; and generating the pronunciation diagnosis result based on the correspondence, the score of each standard phoneme and the score of each decoded phoneme.

[0004] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the correspondence between each standard phoneme in the standard pronunciation and each decoded phoneme in the decoded phoneme sequence is determined, including: using phonemes as editing elements, performing editing distance operations on the standard pronunciation and the decoded phoneme sequence to determine the correspondence.

[0005] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the edit distance operation includes a phoneme replacement operation, and the weights of the phoneme replacement operations between different phonemes are at least not completely the same.

[0006] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation diagnosis result is generated based on the corresponding relationship, the score of each standard phoneme and the score of each decoded phoneme, including: in response to any standard phoneme having a corresponding decoded phoneme, judging whether the score of any standard phoneme is lower than a first score threshold; in response to the score of any standard phoneme being lower than the first score threshold, calculating the boundary overlap between any standard phoneme and the decoded phoneme corresponding to any standard phoneme based on the time boundary of any standard phoneme and the time boundary of the decoded phoneme corresponding to any standard phoneme; and, in response to any standard phoneme being different from the decoded phoneme corresponding to any standard phoneme and the boundary overlap between any standard phoneme and the decoded phoneme corresponding to any standard phoneme being not less than the overlap threshold, indicating in the pronunciation diagnosis result that a misreading of any standard phoneme has occurred.

[0007] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation diagnosis result is generated based on the correspondence, the scores of each standard phoneme and the scores of each decoded phoneme, and also includes: judging whether the difference between the score of the decoded phoneme corresponding to any standard phoneme and the score of any standard phoneme is not less than a second score threshold; and, in response to the difference between the score of the decoded phoneme corresponding to any standard phoneme and the score of any standard phoneme being not less than the second score threshold, indicating in the misreading situation that any standard phoneme is misread as a decoded phoneme corresponding to any standard phoneme.

[0008] For example, in the sound correction method provided in some embodiments of the present disclosure, the boundary overlap is calculated according to the following formula:

[0009]

[0010] Among them, BC represents the boundary coincidence, x1 and y1 represent the starting time boundary and ending time boundary of a standard phoneme respectively, x2 and y2 represent the starting time boundary and ending time boundary of a decoded phoneme respectively, min() is the minimum value function, and max() is the maximum value function.

[0011] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation diagnosis result is generated based on the corresponding relationship, the score of each standard phoneme and the score of each decoded phoneme, including: in response to any standard phoneme having no corresponding decoded phoneme, indicating in the pronunciation diagnosis structure that any standard phoneme has been missed.

[0012] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation diagnosis result is generated based on the correspondence, the scores of each standard phoneme and the scores of each decoded phoneme, including: in response to any decoded phoneme having no corresponding standard phoneme, indicating in the pronunciation diagnosis result that multiple readings have occurred.

[0013] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation diagnosis result is generated based on the correspondence, the scores of each standard phoneme and the scores of each decoded phoneme, and also includes: in response to the score of any decoded phoneme being not lower than a third score threshold, indicating that any decoded phoneme is read multiple times in the multiple reading situation.

[0014] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation diagnosis operation is performed on the first audio data based on the word to generate the pronunciation diagnosis result, which also includes: determining the time boundary of the vowel phoneme in the stressed syllable based on the time boundary of each standard phoneme in the standard pronunciation and the stressed syllable of the standard pronunciation; extracting the feature information of the first audio segment determined by the time boundary of the vowel phoneme in the stressed syllable; judging whether the stressed syllable is stressed through a classification model based on the feature information of the first audio segment; and, in response to the stressed syllable being judged as not being stressed, indicating in the pronunciation diagnosis result that the stressed syllable is not stressed.

[0015] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation diagnosis operation is performed on the first audio data based on the word to generate the pronunciation diagnosis result, which also includes: determining the time boundary of the vowel phoneme in the unstressed syllable based on the time boundary of each standard phoneme in the standard pronunciation and the unstressed syllable of the standard pronunciation; extracting the feature information of the second audio segment determined by the time boundary of the vowel phoneme in the unstressed syllable; judging whether the unstressed syllable is stressed through a classification model based on the feature information of the second audio segment; and, in response to the unstressed syllable being judged to be stressed, indicating in the pronunciation diagnosis result that the unstressed syllable is stressed.

[0016] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the score of each standard phoneme and the score of each decoded phoneme are determined based on a pronunciation accuracy algorithm.

[0017] For example, the pronunciation correction method provided in some embodiments of the present disclosure further includes: providing pronunciation correction guidance based on the pronunciation diagnosis result.

[0018] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation correction guidance is performed based on the pronunciation diagnosis result, including: in response to the pronunciation correction operation, displaying the standard pronunciation of the word, the pronunciation diagnosis result and text guidance, wherein the text guidance is used to guide the user to pronounce correctly.

[0019] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation correction guidance is performed based on the pronunciation diagnosis result, and also includes: when displaying the text guidance, the text guidance is played synchronously with voice.

[0020] For example, the pronunciation correction method provided in some embodiments of the present disclosure further includes: obtaining second audio data about the word and providing practice feedback for the second audio data.

[0021] For example, in the pronunciation correction method provided in some embodiments of the present disclosure, the pronunciation diagnosis result includes at least one of a syllable error and a pronunciation error, the syllable error includes at least one of a syllable number error and a stress error, and the pronunciation error includes at least one of misreading a vowel phoneme, misreading a consonant phoneme, and missing a consonant phoneme.

[0022] At least some embodiments of the present disclosure further provide a sound correction device comprising: a memory for non-temporarily storing computer-readable instructions; and a processor for executing the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, execute the sound correction method provided in any embodiment of the present disclosure.

[0023] For example, the sound correction device provided in some embodiments of the present disclosure further includes: an audio acquisition device for acquiring the first audio data.

[0024] At least some embodiments of the present disclosure further provide a non-transitory storage medium that non-temporarily stores computer-readable instructions, wherein when the non-transitory computer-readable instructions are executed by a computer, the sound correction method provided by any embodiment of the present disclosure can be executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0026] Figure 1 A flowchart of a method for correcting sound provided for at least some embodiments of the present disclosure;

[0027] Figure 2 A method corresponding to at least some embodiments of the present disclosure is provided Figure 1 An exemplary flow chart of step S200 shown in FIG.

[0028] Figure 3 Another embodiment of the present disclosure provides a method corresponding to Figure 1 An exemplary flow chart of step S200 shown in FIG.

[0029] Figure 4A A schematic diagram of a pronunciation diagnosis result displayed on a word pronunciation correction interactive interface provided in at least some embodiments of the present disclosure;

[0030] Figure 4B A schematic diagram of another pronunciation diagnosis result displayed on a word pronunciation correction interactive interface provided for at least some embodiments of the present disclosure;

[0031] Figure 4C A schematic diagram of another pronunciation diagnosis result displayed on a word pronunciation correction interactive interface provided for at least some embodiments of the present disclosure;

[0032] Figure 5A A schematic diagram of a pronunciation correction guidance interface provided by at least some embodiments of the present disclosure;

[0033] Figure 5B A schematic diagram of another pronunciation correction guidance interface provided for at least some embodiments of the present disclosure;

[0034] Figure 5C A schematic diagram of another tone correction guidance interface provided for at least some embodiments of the present disclosure;

[0035] Figure 5D A schematic diagram of yet another tone correction guidance interface provided for at least some embodiments of the present disclosure;

[0036] Figure 6 A schematic diagram of a transition interface provided for at least some embodiments of the present disclosure;

[0037] Figure 7A A schematic diagram of a practice interface provided for at least some embodiments of the present disclosure;

[0038] Figure 7B A schematic diagram of another practice interface provided for at least some embodiments of the present disclosure;

[0039] Figure 7C A schematic diagram of a feedback interface provided for at least some embodiments of the present disclosure;

[0040] Figure 7D A schematic diagram of another feedback interface provided for at least some embodiments of the present disclosure;

[0041] Figure 8 A schematic block diagram of a sound correction device provided for at least some embodiments of the present disclosure; and

[0042] Figure 9 A schematic block diagram of a non-transitory storage medium is provided for at least some embodiments of the present disclosure. DETAILED DESCRIPTION

[0043] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0044] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by persons of ordinary skill in the field to which this disclosure belongs. The words "first", "second" and similar terms used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0045] The present disclosure is described below through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of known functions and known components. When any component of an embodiment of the present disclosure appears in more than one figure, the component is represented by the same or similar reference numeral in each figure.

[0046] Traditional pronunciation assessment technology gives a 100-point score to the user's reading or amplified speech, but due to the lack of pronunciation diagnosis, the score lacks guidance for correcting the user's pronunciation. Influenced by the "preconceived notions" of their native language, users may not be able to distinguish the difference between their own pronunciation and the demonstration pronunciation; even after hearing the difference, it is often difficult to adjust the pronunciation organs to the correct position. Against this background, more detailed pronunciation diagnosis and correction technologies have begun to emerge. However, the existing language pronunciation diagnosis and correction technologies can only support limited confusion phoneme errors and cannot provide targeted pronunciation correction feedback.

[0047] At least some embodiments of the present disclosure provide a pronunciation correction method. The method includes: obtaining a word and first audio data; performing a pronunciation diagnosis operation on the first audio data based on the word to generate a pronunciation diagnosis result; wherein the standard pronunciation of the word includes at least one standard phoneme; performing a pronunciation diagnosis operation on the first audio data based on the word to generate a pronunciation diagnosis result, including: aligning the first audio data with the standard pronunciation based on a first acoustic model to obtain a time boundary of each standard phoneme in the standard pronunciation in the first audio data; determining a score for each standard phoneme based on an audio segment determined by the time boundary of each standard phoneme; performing a recognition operation on the first audio data based on a second acoustic model to obtain a decoded phoneme sequence and a time boundary of each decoded phoneme in the decoded phoneme sequence in the first audio data, wherein the decoded phoneme sequence includes at least one decoded phoneme; determining a score for each decoded phoneme based on the audio segment determined by the time boundary of each decoded phoneme; determining a correspondence between each standard phoneme in the standard pronunciation and each decoded phoneme in the decoded phoneme sequence; and generating a pronunciation diagnosis result based on the correspondence, the score of each standard phoneme, and the score of each decoded phoneme.

[0048] Some embodiments of the present disclosure also provide a sound correction device and a non-transitory storage medium corresponding to the above-mentioned sound correction method.

[0049] The pronunciation correction method provided by the embodiments of the present disclosure performs pronunciation diagnosis operations based on "dual-model two-pass decoding" (the dual model is the first acoustic model and the second acoustic model, and the two-pass decoding is the alignment operation and the recognition operation). It can obtain pronunciation diagnosis results quickly and conveniently, so that users can correct existing pronunciation problems in a targeted manner according to the pronunciation diagnosis results, thereby improving the user's language learning efficiency and having high practicality.

[0050] The following describes some embodiments and examples of the present disclosure in detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present disclosure and are not intended to limit the present disclosure.

[0051] Figure 1A flowchart of a method for correcting sound provided for at least some embodiments of the present disclosure. For example, the method for correcting sound can be applied to a computing device, which includes any electronic device with computing capabilities, such as a smart phone, a laptop computer, a tablet computer, a desktop computer, a server, etc., and the embodiments of the present disclosure are not limited to this. For example, the computing device has a central processing unit (CPU) or a graphics processing unit (GPU), and the computing device also includes a memory. The memory is, for example, a non-volatile memory (such as a read-only memory (ROM)), on which the code of the operating system is stored. For example, the memory also stores codes or instructions, and by running these codes or instructions, the method for correcting sound provided by the embodiments of the present disclosure can be implemented.

[0052] For example, Figure 1 As shown, the pronunciation correction method includes the following steps S100 to S400.

[0053] Step S100: Acquire a word and first audio data.

[0054] For example, in some embodiments, Figure 1 The pronunciation correction method shown can be executed locally, for example, by a client. In this case, the word can be any word selected by the user from various words stored on the client, or it can be a preset word provided by the client (for example, a preset word provided by a language learning application on the client), and the embodiments of the present disclosure are not limited to this. The first audio data (i.e., user audio data) can include speech collected by an audio collection module or device on the client, and the embodiments of the present disclosure include but are not limited to this. For example, the word and first audio data in step S100 can also be obtained by the client from the network.

[0055] For example, the client includes but is not limited to a smartphone, tablet computer, personal computer, personal digital assistant (PDA), wearable device, head-mounted display device, scanning pen, point reading pen, etc. For example, the audio acquisition module or device includes but is not limited to a built-in or external microphone of the client. For example, the first audio data can be pre-recorded or recorded in real time, and the embodiments of the present disclosure are not limited to this.

[0056] For example, in other embodiments, Figure 1The pronunciation correction method shown can also be performed remotely, for example, by a server. In this case, the server can receive the first audio data uploaded by the user through the client (the word in step S100 can be pre-stored on the server or uploaded to the server by the user through the client), then perform the pronunciation correction process, and return the pronunciation diagnosis results, etc. to the client for the user's reference.

[0057] For example, the word may be a word in English, French, German, Russian, Spanish, Chinese, Japanese, Korean, etc., and the embodiments of the present disclosure include but are not limited to these.

[0058] For example, in some embodiments, the standard pronunciation of the word can be searched through a pronunciation dictionary, but this is not limited to this. For example, the pronunciation dictionary can include a vocabulary set and its pronunciation that can be processed by a common speech recognition engine. For example, the standard pronunciation of the word is generally a phoneme sequence, which can include at least one phoneme (i.e., a standard phoneme). In this disclosure, for ease of explanation and distinction, the phonemes in the standard pronunciation of a word and the phonemes in the decoded phoneme sequence described later are referred to as "standard phonemes" and "decoded phonemes," respectively. It should be understood that in actual applications, the vast majority of words include multiple phonemes. When the standard pronunciation of a word includes only one standard phoneme, "each standard phoneme" and "each standard phoneme" in the standard pronunciation are used to refer to the standard phoneme; similarly, when the decoded phoneme sequence includes only one decoded phoneme, "each decoded phoneme" and "each decoded phoneme" in the decoded phoneme sequence are used to refer to the decoded phoneme. Taking English as an example, the English International Phonetic Alphabet has a total of 48 phonemes, including 20 vowel phonemes and 28 consonant phonemes.

[0059] For example, in general, the first audio data is the voice data of the user reading aloud or following the reading of a word (allowing problems such as misreading phonemes, missing phonemes, multiple readings of phonemes, and incorrect stress), so that the pronunciation correction method can accurately diagnose the user's pronunciation problems and evaluate the standardness of the user's pronunciation of the word.

[0060] Step S200: performing a pronunciation diagnosis operation on the first audio data based on the word to generate a pronunciation diagnosis result.

[0061] For example, in some embodiments, the pronunciation diagnosis operation can be used to diagnose possible problems such as misreading phonemes, missing phonemes, and over-reading phonemes to generate corresponding pronunciation diagnosis results.

[0062] Figure 2 A method corresponding to at least some embodiments of the present disclosure is provided Figure 1 The exemplary flow chart of step S200 is shown in FIG. Figure 2As shown, step S200 may include the following steps S210 to S260.

[0063] Step S210: Based on the first acoustic model, align the first audio data with the standard pronunciation to obtain the time boundary of each standard phoneme in the standard pronunciation in the first audio data.

[0064] For example, in some examples, step S210 may include: based on the first acoustic model, using a forced alignment algorithm to perform a forced alignment operation on the first audio data and the standard pronunciation to obtain the time boundary of each standard phoneme in the standard pronunciation in the first audio data.

[0065] For example, in general, the acoustic model is trained through a large amount of training data (such as recordings of speakers), and the acoustic model can be used to determine the possibility that the audio frame in the first audio data corresponds to any phoneme, thereby forcibly aligning the first audio data with the standard pronunciation of the word or performing free phoneme recognition on the first audio data at the phoneme level. For example, in some examples, the acoustic model can be a neural network-based model, and the embodiments of the present disclosure include but are not limited to this; for example, the neural network can include but is not limited to a time-delay neural network (TDNN), a recurrent neural network (RNN), a long-short term memory network (LSTM), or a bidirectional long-short term memory network (Bi-directional Long-Short Term Memory, Bi-LSTM), etc. For example, the specific technical details of the acoustic model and the forced alignment algorithm can refer to the relevant technologies in the field of natural language processing and will not be repeated here.

[0066] For example, the first acoustic model may be an acoustic model suitable for a forced alignment operation. For example, in some examples, the first acoustic model may be a TDNN model, etc., and embodiments of the present disclosure include but are not limited to this.

[0067] For example, through the above-mentioned forced alignment operation, the first audio data can be divided into at least one audio segment to correspond to at least one standard phoneme in the standard pronunciation of the above-mentioned word. For example, in some examples, the standard pronunciation of the above-mentioned word includes multiple standard phonemes, and then through the above-mentioned forced alignment operation, the first audio data can be divided into multiple audio segments corresponding one-to-one to the multiple standard phonemes. For example, in some examples, the audio segment corresponding to each standard phoneme can be identified by the time boundary (time boundary) of the audio segment in the first audio data, for example, the above-mentioned time boundary includes the starting time boundary (starting time) and the ending time boundary (ending time) of the audio segment.

[0068] Step S220: determining a score for each standard phoneme according to the audio segment determined by the time boundary of each standard phoneme.

[0069] For example, in some examples, the score of each standard phoneme can be calculated using a Goodness of Pronunciation (GOP) algorithm based on the audio segment determined by the time boundary of each standard phoneme. Embodiments of the present disclosure include but are not limited to this.

[0070] For example, in some examples, a pronunciation accuracy algorithm may include: first extracting acoustic feature information of an audio segment determined by the temporal boundaries of each standard phoneme, such as, for example, Mel-scale Frequency Cepstral Coefficients (MFCCs); then inputting the above acoustic feature information into a pre-trained phoneme evaluation model for phoneme evaluation to obtain a GOP value for each standard phoneme; and finally, determining a score for each standard phoneme based on the GOP value of each standard phoneme. For example, the specific technical details of the pronunciation accuracy algorithm can be referenced to relevant technologies in the field of speech processing and will not be elaborated here.

[0071] For example, the range of the score of the standard phoneme can be set according to actual needs, and the embodiments of the present disclosure do not limit this. For example, in some examples, the range of the score of the standard phoneme can be [0, 100], and the embodiments of the present disclosure include but are not limited to this.

[0072] Step S230: performing a recognition operation on the first audio data based on the second acoustic model to obtain a decoded phoneme sequence and a time boundary of each decoded phoneme in the decoded phoneme sequence in the first audio data.

[0073] For example, the second acoustic model may be an acoustic model suitable for free phoneme recognition operations. For example, the second acoustic model is different from the first acoustic model. For example, compared with the first acoustic model, the second acoustic model may generally be a larger model, its structure may be more sophisticated and complex, and it may be trained with a larger amount of training data. For example, in some examples, the second acoustic model may be a factorized time-delay neural network (TDNN-F) model or LSTM, etc., and embodiments of the present disclosure include but are not limited to the same.

[0074] For example, in some examples, a recognition operation can be performed on the first audio data in combination with a second acoustic model and a language model to obtain a decoded phoneme sequence and the temporal boundaries of each decoded phoneme in the decoded phoneme sequence in the first audio data. In this case, the acoustic features of the first audio data can be first extracted using the second acoustic model and converted into a candidate phoneme sequence. The final decoded phoneme sequence can then be determined from the candidate phoneme sequence using the second language model and a decoding operation. For example, the language model can be a unigram language model trained based on pronunciation phoneme sequences of a large amount of training text (e.g., words), and embodiments of the present disclosure include but are not limited to this. For example, the Viterbi algorithm can be used to perform the above-mentioned decoding operation, and embodiments of the present disclosure include but are not limited to this. For example, based on the Viterbi algorithm, an optimal decoding path can be found to determine the decoded phoneme sequence; for example, further, the temporal boundaries of each decoded phoneme can be obtained by backtracking after the Viterbi algorithm is completed. For example, the specific technical details of the language model and the Viterbi algorithm can be referred to the relevant technologies in the field of natural language processing and will not be repeated here.

[0075] Step S240: Determine a score for each decoded phoneme according to the audio segment determined by the time boundary of each decoded phoneme.

[0076] For example, in some examples, a Goodness of Pronunciation (GOP) algorithm can be used to calculate the score of each decoded phoneme based on the audio segment determined by the time boundary of each decoded phoneme. Embodiments of the present disclosure include but are not limited to this. For example, the specific details of the GOP algorithm can be referred to the relevant description in the aforementioned step S220 and will not be repeated here.

[0077] For example, in some examples, the same pronunciation accuracy algorithm (e.g., the same phoneme evaluation model) can be used to calculate the score of each standard phoneme and the score of each decoded phoneme to improve the comparability of the scores of the standard phoneme and the scores of the decoded phoneme, thereby helping to obtain more reliable pronunciation diagnosis results.

[0078] For example, the score range of the decoded phoneme is generally the same as the score range of the standard phoneme. For example, in some examples, the score range of the decoded phoneme may also be [0, 100], and embodiments of the present disclosure include but are not limited to this.

[0079] Step S250: determining the correspondence between each standard phoneme in the standard pronunciation and each decoded phoneme in the decoded phoneme sequence.

[0080] For example, in some examples, phonemes may be used as editing elements to perform an edit distance operation on the standard pronunciation and the decoded phoneme sequence to determine the correspondence between each standard phoneme in the standard pronunciation and each decoded phoneme in the decoded phoneme sequence.

[0081] The following briefly describes how to calculate the edit distance between any two strings. The edit distance of a string, also known as the Levenshtein edit distance, is the minimum number of operations required to transform a string a{i} into a string b{j} using character operations (i.e., using characters as editing elements). Character operations include: (1) deleting a character, (2) inserting a character, and (3) replacing a character.

[0082] For strings a{i} and b{j}, i represents the length of string a{i} (i.e., the number of characters included), j represents the length of string b{j}, i and j are both integers, and i ≥ 0, j ≥ 0. lev(a{i},b{j}) represents the edit distance between strings a{i} and b{j}. A Levenshtein edit distance algorithm includes the following formula:

[0083]

[0084] Among them, max() is the maximum value function, min() is the minimum value function, a{i-1} represents the string formed by the first i-1 characters of the string a{i}, b{j-1} represents the string formed by the first j-1 characters of the string b{j}, a[i] represents the i-th character (that is, the last character) in the string a{i}, and b[j] represents the j-th character (that is, the last character) in the string b{j}.

[0085] The Levenshtein edit distance algorithm expressed in the above formula includes:

[0086] (1) When the length of a string is 0 (corresponding to the case if min(i,j)=0), the edit distance is the length of the other string;

[0087] (2) When the lengths of both strings a and b are not 0 (corresponding to the case if min(i,j)≠0):

[0088] If the last characters of the two strings are the same (corresponding to the case if a[i]=b[j]), the last characters of the two strings a{i} and b{j} can be deleted to obtain two new strings a{i-1} and b{j-1}. Determining the edit distance between the strings a{i} and b{j} is converted to determining the edit distance between the new strings a{i-1} and b{j-1}.

[0089] If the last character of the two strings is different (corresponding to the case if a[i]≠b[j]), then determining the edit distance between strings a{i} and b{j} is transformed into determining the minimum of the edit distance between strings a{i-1} and b{j}, the edit distance between strings a{i} and b{j-1}, and the edit distance between strings a{i-1} and b{j-1}.

[0090] It should be understood that in the above-mentioned Levenshtein edit distance algorithm, the weight of each character operation is set to 1, that is, the edit distance corresponding to each character operation is 1. In actual applications, the weight of each character operation can be set according to actual needs.

[0091] It should be understood that the Levenshtein edit distance algorithm described above compares two strings starting from the last character (i.e., comparing whether the last characters of the two strings are identical). Practical applications include, but are not limited to, this. For example, another Levenshtein edit distance algorithm can compare two strings starting from the first character.

[0092] It should be understood that determining the Levenshtein edit distance is a dynamic programming problem, which can be calculated recursively according to the above formula. It should also be understood that during the dynamic programming process of calculating the Levenshtein edit distance, the correspondence between each character in string a{i} and each character in string b{j} can be determined (i.e., when the edit distance is minimized, if a character in string a{i} is the same as a character in string b{j}, then the two characters correspond to each other).

[0093] For example, in some examples, the Levenshtein edit distance algorithm described above can be used to perform an edit distance operation on the standard pronunciation and the decoded phoneme sequence using phonemes as edit elements to determine the correspondence between each standard phoneme in the standard pronunciation and each decoded phoneme in the decoded phoneme sequence. For example, the standard pronunciation can be regarded as a phoneme string a{i}, and the decoded phoneme sequence can be regarded as a phoneme string b{j}. The phoneme edit distance lev(a{i},b{j}) between the phoneme strings a{i} and b{j} can be expressed as:

[0094]

[0095] Among them, max() is the maximum value function, min() is the minimum value function, a{i-1} represents the phoneme string formed by the first i-1 phonemes of the phoneme string a{i}, b{j-1} represents the phoneme string formed by the first j-1 phonemes of the phoneme string b{j}, a[i] represents the i-th phoneme (i.e., the last phoneme) in the phoneme string a{i}, b[j] represents the j-th phoneme (i.e., the last phoneme) in the phoneme string b{j}, f1 represents the weight of the operation of deleting / omitting the phoneme a[i] (i.e., the phoneme edit distance contribution value of the operation), f2 represents the weight of the operation of inserting / multiple readings of the phoneme b[j] (i.e., the phoneme edit distance contribution value of the operation), and f3 represents the weight of the operation of replacing / misreading the phoneme a[i] with the phoneme b[j] (i.e., the phoneme edit distance contribution value of the operation).

[0096] It is understood that in the above-mentioned phoneme edit distance algorithm, phoneme operations (also called "edit distance operations") may include: (1) deleting a phoneme, (2) inserting a phoneme, and (3) replacing a phoneme. According to the above-mentioned phoneme edit distance formula, the phoneme edit distance algorithm includes:

[0097] (1) When the length of a phoneme string is 0 (corresponding to the case if min(i,j)=0), the phoneme edit distance is the length of the other phoneme string;

[0098] (2) When the lengths of phoneme strings a and b are both not 0 (corresponding to the case if min(i,j)≠0):

[0099] If the last phoneme of the two phoneme strings is the same (corresponding to the case if a[i]=b[j], in this case the standard phoneme a[i] has a corresponding decoded phoneme b[j]), then the last phoneme of the two phoneme strings a{i} and b{j} can be deleted to obtain two new phoneme strings a{i-1} and b{j-1}. Determining the phoneme edit distance between the phoneme strings a{i} and b{j} is converted into determining the phoneme edit distance between the new phoneme strings a{i-1} and b{j-1}.

[0100] If the last phoneme of the two phoneme strings is different (corresponding to the case if a[i]≠b[j]), then determining the phoneme edit distance between phoneme strings a{i} and b{j} is converted into determining the minimum value of the phoneme edit distance between phoneme strings a{i-1} and b{j} (corresponding to the case of deleted / omitted phoneme a[i], in which case the standard phoneme a[i] has no corresponding decoded phoneme), the phoneme edit distance between phoneme strings a{i} and b{j-1} (corresponding to the case of inserted / multiple phoneme b[j], in which case the decoded phoneme b[j] has no corresponding standard phoneme), and the phoneme edit distance between phoneme strings a{i-1} and b{j-1} (corresponding to the case of replaced / misread phoneme a[i] as phoneme b[j], in which case the standard phoneme a[i] has a corresponding decoded phoneme b[j]).

[0101] For example, in some examples, the weight f1 can be a constant value; in this case, the probability of deletion / omission of different phonemes is the same. For example, in other examples, the weight f1 is related to the type of phoneme a[i], that is, f1=f1(a[i]), that is, the weight f1 is a function value of phoneme a[i]; in this case, the probability of deletion / omission of different phonemes is at least not exactly the same, but of course, it can also be different.

[0102] For example, in some examples, the weight f2 can be a constant value; in this case, the insertion / multiple reading probabilities of different phonemes are the same. For example, in other examples, the weight f2 is related to the type of phoneme b[j], that is, f2=f2(b[j]), that is, the weight f2 is a function value of the phoneme b[j]; in this case, the insertion / multiple reading probabilities of different phonemes are at least not exactly the same, but of course, they can also be different.

[0103] For example, in some examples, the weight f3 can be a constant value; in this case, the weights of the phoneme replacement operations between different phonemes are the same, that is, the probability of phoneme replacement / misreading between different phonemes is the same. For example, in other examples, the weight f1 is related to the types of phoneme a[i] and phoneme b[j], that is, f3=f3(a[i],b[j])), that is, the weight f3 is a function value of phoneme a[i] and phoneme b[j]; in this case, the weights of the phoneme replacement operations between different phonemes are different or at least not completely the same, that is, the probability of phoneme replacement / misreading between different phonemes is at least not completely the same, and of course, they can also be different.

[0104] It should be understood that the types (constant values or function values) of the above-mentioned weights f1, f2, f3 and the corresponding specific values can be set according to actual needs, and the embodiments of the present disclosure are not limited to this. For example, the types and values (or functional relationships) of the weights f1, f2, f3 can be set according to teaching and research experience, or the pronunciation problems existing in a large amount of user pronunciation data can be counted or studied to set the types and values (or functional relationships) of the weights f1, f2, f3. For example, in a specific example, in order to simplify the pronunciation correction method provided by the embodiments of the present disclosure, the weight f1 can be set to a constant value t1 (for example, t1=1); the weight f2 can be set to a constant value t2 (for example, t2=1); the weight f3 (corresponding to if a[i]≠b[j]) can be set to a function value, wherein, when the phoneme a[i] and the phoneme b[j] are both vowel phonemes (i.e., one vowel phoneme is misread as another vowel phoneme, that is, confused vowel phonemes), the weight f3 can be set to a constant value t31 (e.g., t31=0.5), when the phoneme a[i] and the phoneme b[j] are both consonant phonemes (i.e., one consonant phoneme is misread as another consonant phoneme, that is, confused consonant phonemes), the weight f3 can be set to a constant value t32 (e.g., t32=0.5), when When phoneme a[i] and phoneme b[j] are a vowel phoneme and a consonant phoneme (i.e., a vowel phoneme is misread as a consonant phoneme or a consonant phoneme is misread as a vowel phoneme), weight f3 can be set to a constant value t33 (e.g., t33=1), wherein the probability of misreading a vowel phoneme as a consonant phoneme or misreading a consonant phoneme as a vowel phoneme is usually lower than the probability of confusing vowel phonemes and confusing consonant phonemes, and therefore, t33 is usually greater than t31 and t32. It should be noted that the values of constant values t1, t2, t31, t32, and t33 can be set according to actual needs, as long as they are greater than 0.

[0105] Step S260: Generate a pronunciation diagnosis result based on the correspondence, the score of each standard phoneme, and the score of each decoded phoneme.

[0106] For example, based on the corresponding relationship, the scores of each standard phoneme, and the scores of each decoded phoneme, it can be determined whether the first audio data has a pronunciation problem. If a pronunciation problem exists, the type of pronunciation problem can be further determined to generate a corresponding pronunciation diagnosis result. For example, the pronunciation problem here includes at least one of mispronunciation of a phoneme, omission of a phoneme, and multiple pronunciation of a phoneme.

[0107] For example, step S260 may include the following steps S261 to S263 to diagnose the possible problem of misreading phonemes and generate corresponding pronunciation diagnosis results.

[0108] Step S261: In response to any standard phoneme having a corresponding decoded phoneme, determining whether the score of the standard phoneme is lower than a first score threshold.

[0109] For example, in actual applications, the first score threshold can be reasonably set according to the value range of the score of the standard phoneme. For example, in some examples, assuming that the value range of the score of the standard phoneme is [0,100], the value range of the first score threshold can be set to, for example, [50,70]. The embodiments of the present disclosure include but are not limited to this. For example, in the above example, the first score threshold can be set to 50, 55, 60, 65, 70, etc. according to actual needs. It can be understood that if the score of any standard phoneme is not lower than the first score threshold, it can generally be considered that the pronunciation of any standard phoneme is accurate.

[0110] Step S262: In response to the score of any standard phoneme being lower than a first score threshold, a degree of boundary overlap between the any standard phoneme and the decoded phoneme corresponding to the any standard phoneme is calculated based on the time boundary of the any standard phoneme and the time boundary of the decoded phoneme corresponding to the any standard phoneme.

[0111] For example, in some examples, the boundary overlap can be calculated according to the following formula:

[0112]

[0113] Where BC represents the degree of boundary overlap, x1 and y1 represent the start and end time boundaries of a standard phoneme, respectively, x2 and y2 represent the start and end time boundaries of a decoded phoneme, respectively, min() represents the minimum value function, and max() represents the maximum value function. It will be understood that when the boundary overlap calculated according to the above formula is less than or equal to 0, it means that the time boundaries of the two phonemes do not overlap.

[0114] Step S263: In response to the fact that any standard phoneme is different from the decoded phoneme corresponding to the any standard phoneme and the boundary overlap between the any standard phoneme and the decoded phoneme corresponding to the any standard phoneme is not less than the overlap threshold, it is indicated in the pronunciation diagnosis result that a misreading of the any standard phoneme has occurred.

[0115] For example, in actual applications, the overlap threshold can be reasonably set according to the value range of the standard phoneme score. For example, in some examples, the value range of the overlap threshold can be set to, for example, [40%, 60%], and the embodiments of the present disclosure include but are not limited to this. For example, in the above example, the overlap threshold can be set to 40%, 45%, 50%, 55%, 60%, etc. according to actual needs. It can be understood that if the boundary overlap between any standard phoneme and the decoded phoneme corresponding to the any standard phoneme is not less than the overlap threshold, it can generally be considered that the correspondence between the any standard phoneme and the decoded phoneme corresponding to the any standard phoneme determined in the aforementioned step S250 is accurate.

[0116] For example, in some examples, in steps S261 to S263, if the any standard phoneme is different from the decoded phoneme corresponding to the any standard phoneme and the boundary overlap between the any standard phoneme and the decoded phoneme corresponding to the any standard phoneme is less than the overlap threshold, it generally means that there is a high probability of a mismatch in the aforementioned correspondence. Therefore, it can be indicated in the pronunciation diagnosis result that the any standard phoneme has been missed, and at the same time, it can be indicated that the decoded phoneme corresponding to the any standard phoneme has been read more. In this way, the adverse effects of the mismatch that may exist in the aforementioned correspondence on the pronunciation diagnosis result can be avoided. For example, in some examples, step S260 can also include: in response to the any standard phoneme being different from the decoded phoneme corresponding to the any standard phoneme and the boundary overlap between the any standard phoneme and the decoded phoneme corresponding to the any standard phoneme being less than the overlap threshold, it can be indicated in the pronunciation diagnosis result that the any standard phoneme has been missed, and at the same time, it can be indicated that the decoded phoneme corresponding to the any standard phoneme has been read more.

[0117] It is understood that, in steps S261 to S263, if the standard phoneme is identical to the decoded phoneme corresponding to the standard phoneme and the boundary overlap between the standard phoneme and the decoded phoneme corresponding to the standard phoneme is not less than the overlap threshold, it generally means that the pronunciation of the standard phoneme is accurate. This situation usually does not occur, otherwise it will contradict the premise of "the score of the standard phoneme is lower than the first score threshold" in step S262. In addition, in steps S261 to S263, if the standard phoneme is identical to the decoded phoneme corresponding to the standard phoneme and the boundary overlap between the standard phoneme and the decoded phoneme corresponding to the standard phoneme is less than the overlap threshold, it generally means that multiple readings have occurred around the standard phoneme, and the pronunciation of the standard phoneme is likely to be accurate. Therefore, it is generally not necessary to report the occurrence of misreading of the standard phoneme in the pronunciation diagnosis result (refer to the subsequent step S267 to indicate the occurrence of multiple readings).

[0118] It is understandable that when the pronunciation diagnosis result indicates that a misreading of any standard phoneme has occurred, the location where the misreading occurred (for example, the location of the standard phoneme in the standard pronunciation, etc.) can usually also be indicated.

[0119] For example, based on steps S261 to S263, step S260 may further include steps S264 to S265 to generate more detailed pronunciation diagnosis results for the problem of mispronunciation of phonemes.

[0120] Step S264: Determine whether the difference between the score of the decoded phoneme corresponding to the any standard phoneme and the score of the any standard phoneme is not less than a second score threshold.

[0121] For example, in practical applications, the second score threshold can be reasonably set based on the range of the standard phoneme score. For example, in some examples, assuming that the range of the standard phoneme and the decoded phoneme scores are both [0, 100], the range of the first score threshold can be set to, for example, [20, 40]. The embodiments of the present disclosure include but are not limited to this. For example, in the above example, the first score threshold can be set to 20, 25, 30, 35, 40, etc. according to actual needs.

[0122] Step S265: In response to the difference between the score of the decoded phoneme corresponding to the any standard phoneme and the score of the any standard phoneme being not less than a second score threshold, indicating in the misreading case that the any standard phoneme is misread as the decoded phoneme corresponding to the any standard phoneme.

[0123] It can be understood that, when the score of any standard phoneme is lower than the first score threshold (i.e., the pronunciation of any standard phoneme is likely to be inaccurate), if the difference between the score of the decoded phoneme corresponding to any standard phoneme and the score of any standard phoneme is not less than the second score threshold, then the decoded phoneme identified in step S230 can generally be considered to be accurate. Therefore, it can be specifically indicated in the misreading situation that any standard phoneme is misread as the decoded phoneme corresponding to any standard phoneme, so that the user can be more clearly aware of the pronunciation problems he or she has. On the other hand, if the difference between the score of the decoded phoneme corresponding to any standard phoneme and the score of any standard phoneme is less than the second score threshold, then it can generally be considered that the decoded phoneme identified in step S230 may be inaccurate. Therefore, it can be indicated in the pronunciation diagnosis result that only a misreading situation has occurred for any standard phoneme, without specifically indicating that any standard phoneme is misread as the decoded phoneme corresponding to any standard phoneme, so as to avoid misleading the user.

[0124] For example, step S260 may include the following step S266 to diagnose the possible problem of missing phonemes and generate corresponding pronunciation diagnosis results.

[0125] Step S266: In response to any standard phoneme not having a corresponding decoded phoneme, indicating in the pronunciation diagnosis structure that a reading omission has occurred for any standard phoneme.

[0126] It is understood that when the pronunciation diagnosis structure indicates that a missed pronunciation of any standard phoneme has occurred, the position where the missed pronunciation occurred (e.g., the position of the standard phoneme in the standard pronunciation, etc.) can also be indicated. It is also understood that if any standard phoneme has no corresponding decoded phoneme, the score of the standard phoneme will usually be very low (e.g., far below the first score threshold); therefore, in this case, it is generally not necessary to simultaneously or further consider the impact of the score of any standard phoneme.

[0127] For example, step S260 may further include the following step S267 to diagnose the possible problem of multiple-read phonemes and generate corresponding pronunciation diagnosis results.

[0128] Step S267: In response to any decoded phoneme not having a corresponding standard phoneme, it is indicated in the pronunciation diagnosis result that multiple readings have occurred.

[0129] It is understandable that when the pronunciation diagnosis structure indicates that multiple readings have occurred, the location where the multiple readings have occurred can usually also be indicated (for example, the location of the decoded phoneme relative to a certain standard phoneme, etc.).

[0130] For example, based on step S267, step S260 may further include the following step S268 to generate a more detailed pronunciation diagnosis result for the problem of multiple phonemes.

[0131] Step S268: In response to the score of any decoded phoneme being not lower than a third score threshold, indicating that any decoded phoneme is read multiple times in the multiple reading situation.

[0132] For example, in practical applications, the third score threshold can be reasonably set based on the value range of the decoded phoneme score. For example, in some examples, assuming that the value range of the decoded phoneme score is [0, 100], the value range of the third score threshold can be set to, for example, [50, 70]. The embodiments of the present disclosure include but are not limited to this. For example, in the above example, the third score threshold can be set to 50, 55, 60, 65, 70, etc. according to actual needs. For example, the third score threshold can be the same as or different from the first score threshold.

[0133] It can be understood that if the score of any decoded phoneme is not lower than the third score threshold, it can be generally considered that the decoded phoneme identified in step S230 is accurate. Therefore, it can be specifically indicated in the multiple reading situation that the decoded phoneme is read more, so that the user can be more clearly aware of the pronunciation problems he or she has. On the other hand, if the score of any decoded phoneme is lower than the third score threshold, it can be generally considered that the decoded phoneme identified in step S230 may be inaccurate. Therefore, it can be indicated in the pronunciation diagnosis result that the multiple reading situation has occurred (and the location where the multiple reading situation has occurred), without specifically indicating which phoneme has been read more, so as to avoid misleading the user.

[0134] For example, in some embodiments, the pronunciation diagnosis operation can also be used to diagnose problems such as accent errors to generate corresponding pronunciation diagnosis results. For example, the accent error problem here includes at least one of not accentuating a stressed syllable and accentuating an unstressed syllable.

[0135] Figure 3 Another embodiment of the present disclosure provides a method corresponding to Figure 1 The exemplary flow chart of step S200 is shown in FIG. Figure 3 As shown, in Figure 2 Based on the steps S210 to S260 shown in FIG. Figure 3 (The specific contents of steps S210 to S260 are omitted in the figure), step S200 may also include the following steps S271 to S274 to diagnose the possible stress error problem of not stressing the stressed syllables when the standard pronunciation of the word includes stressed syllables, and generate corresponding pronunciation diagnosis results.

[0136] Step S271: determining the time boundary of the vowel phoneme in the stressed syllable based on the time boundary of each standard phoneme in the standard pronunciation and the stressed syllable in the standard pronunciation;

[0137] Step S272: extracting feature information of the first audio segment determined by the time boundary of the vowel phoneme in the stressed syllable;

[0138] Step S273: determining whether the stressed syllable is stressed using a classification model based on the feature information of the first audio segment; and

[0139] Step S274: In response to the stressed syllable being determined to be unstressed, indicating in the pronunciation diagnosis result that the stressed syllable is unstressed.

[0140] For example, in some examples, in step S271, for a stressed syllable in standard pronunciation, the vowel phoneme in the stressed syllable can be determined first (the vowel phoneme is one of the standard phonemes in the standard pronunciation). Thus, the time boundary of the vowel phoneme can be determined from the time boundaries of the standard phonemes in the standard pronunciation.

[0141] For example, in some examples, in step S272, the feature information may include at least one of energy (e.g., including a normalized energy value), fundamental frequency (e.g., including a normalized fundamental frequency value), short-time average zero-crossing rate, mel-frequency cepstral coefficients, first-order mel-frequency cepstral coefficients, and second-order mel-frequency cepstral coefficients. For example, methods for extracting feature information such as energy, fundamental frequency, short-time average zero-crossing rate, mel-frequency cepstral coefficients, first-order mel-frequency cepstral coefficients, and second-order mel-frequency cepstral coefficients can refer to relevant technologies in the field of natural language processing and are not further described here. For example, in a specific example, the feature information includes the normalized energy value and normalized fundamental frequency value of the vowel phoneme. For example, the normalized energy value of the vowel phoneme can be expressed as the ratio of the average energy value of the first audio segment to the average energy value of the first audio data; similarly, the normalized fundamental frequency value can be expressed as the ratio of the average fundamental frequency value of the first audio segment to the average fundamental frequency value of the first audio data. It should be noted that embodiments of the present disclosure include but are not limited to this.

[0142] For example, in some examples, in step S273, the classification model can be a two-classification model, for example, it can include any one of a support vector machine (SVM) classifier, a Softmax classifier, etc. It is understandable that the classification model can be obtained through machine learning. In the machine learning process, feature information of sample audio segments corresponding to vowel phonemes in a large number of sample syllables (including stressed syllables and unstressed syllables) can be extracted as input to the classification model, and the classification model can be trained based on the stress of these sample syllables in the sample audio data (for example, it can be manually marked) so that the trained classification model can predict whether the vowel phonemes in a certain syllable are stressed. For example, the training process and details of the classification model can refer to the relevant technologies in the field of machine learning and will not be repeated here.

[0143] For example, Figure 3 As shown, based on the aforementioned steps S210 to S260 (and steps S271 to S274), step S200 may further include the following steps S281 to S284 to diagnose possible stress errors in stressing unstressed syllables when the standard pronunciation of a word includes unstressed syllables, and generate corresponding pronunciation diagnosis results.

[0144] Step S281: determining the time boundary of the vowel phoneme in the unstressed syllable based on the time boundary of each standard phoneme in the standard pronunciation and the unstressed syllable in the standard pronunciation;

[0145] Step S282: extracting feature information of the second audio segment determined by the time boundary of the vowel phoneme in the unstressed syllable;

[0146] Step S283: Based on the feature information of the second audio segment, determine whether the unstressed syllable is stressed using a classification model; and

[0147] Step S284: In response to the unstressed syllable being determined to be stressed, indicating in the pronunciation diagnosis result that the unstressed syllable is stressed.

[0148] For example, in some examples, in step S281, for an unstressed syllable in standard pronunciation, the vowel phoneme in the unstressed syllable can be first determined (the vowel phoneme is one of the standard phonemes in the standard pronunciation). Thus, the time boundary of the vowel phoneme can be determined from the time boundaries of the standard phonemes in the standard pronunciation.

[0149] For example, in some examples, the type of feature information extracted in step S282 may be the same as the type of feature information extracted in step S272; in this case, the details of feature information extraction in step S282 can refer to the relevant description of feature information extraction in step S272, and will not be repeated here. It should be noted that the embodiments of the present disclosure include but are not limited to this.

[0150] For example, in some examples, the classification model used in step S283 may be the same classification model as the classification model used in step S273; in this case, the details of the classification model in step S283 may refer to the relevant description of the classification model in the aforementioned step S272, which will not be repeated here. It should be noted that the embodiments of the present disclosure include but are not limited to this. For example, in some embodiments, different classification models may be used in steps S273 and S283, respectively. For example, the classification model used in step S273 may be a classification model specifically used to predict whether a stressed syllable is stressed (i.e., whether a vowel phoneme in a stressed syllable is stressed), and the classification model used in step S283 may be a classification model specifically used to predict whether a non-stressed syllable is stressed (i.e., whether a vowel phoneme in a non-stressed syllable is stressed); in this case, the type of feature information extracted in step S282 may be the same as or different from the type of feature information extracted in step S272.

[0151] It can be understood that in the pronunciation correction method provided in some embodiments, before executing steps S271 to S274 and / or steps S281 to S284, it can be determined whether the standard pronunciation of the word includes stressed syllables and / or unstressed syllables, and then, if the standard pronunciation of the word includes stressed syllables and / or unstressed syllables, steps S271 to S274 and / or steps S281 to S284 are executed accordingly.

[0152] For example, in the pronunciation correction methods provided in some embodiments, pronunciation problems in the pronunciation diagnosis results can be integrated and classified into syllable errors and / or pronunciation errors, etc. according to the phonetic rules and phonological rules of the language.

[0153] For example, syllable errors mainly refer to problems with pronunciation rhythm, which are usually manifested as errors in the rhythm and / or stress of the syllables of a word. For example, syllable errors can include at least one of an error in the number of syllables and an error in stress.

[0154] For example, syllable number errors mainly refer to the addition or omission of syllables in a word during pronunciation, which usually includes at least one of the following situations: reading more vowel phonemes or missing vowel phonemes. Syllable number errors are usually closely related to the transfer of the mother tongue. For example, English learners whose mother tongue is Chinese are prone to making syllable addition errors in pronunciation. Because Chinese syllables are mostly composed of initial consonants and finals (similar to consonants and vowels in English), in English, if a syllable ends with a consonant similar to a Chinese initial consonant, such as / p / , / b / , / d / , / t / , etc., it is easy to subconsciously add / at the end. / to match the Chinese pronunciation rules, resulting in the incorrect pronunciation of a syllable. It is understandable that the situation of reading an extra consonant phoneme rarely occurs alone. It usually occurs when a syllable is read extra (that is, a consonant phoneme and a vowel phoneme are read extra at the same time, and the consonant phoneme and vowel phoneme form a syllable). Of course, this situation also belongs to the error of the number of syllables.

[0155] For example, stress error refers to the stress being placed in the wrong syllable position during pronunciation, which generally includes at least one of the following situations: not stressing a stressed syllable and stressing an unstressed syllable.

[0156] For example, mispronunciation primarily refers to confusion in the pronunciation of one or more syllables or phonemes within a word. This typically manifests as poor pronunciation, making it difficult to understand. It can also manifest as the direct pronunciation of another word, altering the meaning. For example, mispronunciation can include at least one of misreading vowels, misreading consonants, and missing consonants.

[0157] For example, mispronunciation of vowel phonemes (also known as "confusing vowel phonemes") mainly refers to the confusion between vowel phonemes that have certain similarities in pronunciation. It includes not only the confusion of loose and tense vowels, which are easy for English learners to confuse, but also includes the confusion of / e / and / a / that are often made by English learners whose native language is Chinese due to the influence of Chinese and / or dialects. I / confusing situations, etc.

[0158] For example, misreading consonant phonemes (also called "confusing consonant phonemes") mainly refers to the confusion between consonant phonemes that have certain similarities in pronunciation. It not only includes the confusion between voiced and unvoiced consonants that are easy for English learners to confuse, but also includes the confusion between / n / and / l / that often occurs among English learners whose native language is Chinese due to the influence of Chinese and / or dialects.

[0159] For example, missed consonant phonemes usually include the omission of consonants at the end of syllables.

[0160] It is understandable that in actual applications, the pronunciation problems in the pronunciation diagnosis results may include one or more syllable errors, one or more pronunciation errors, or both syllable errors and pronunciation errors, etc. The embodiments of the present disclosure do not limit this.

[0161] For example, the pronunciation diagnosis result generated in step S200 may be presented on an interactive interface of a language learning application, for example, for the user to read and browse.

[0162] Figure 4A A schematic diagram of a pronunciation diagnosis result displayed on a word pronunciation correction interactive interface provided by at least some embodiments of the present disclosure, Figure 4B A schematic diagram of another pronunciation diagnosis result displayed on a word pronunciation correction interactive interface provided in at least some embodiments of the present disclosure is provided. Figure 4C A schematic diagram of another pronunciation diagnosis result displayed on a word pronunciation correction interactive interface provided for at least some embodiments of the present disclosure.

[0163] For example, Figures 4A-4C As shown, the word pronunciation correction interactive interface (as shown in the black solid line box in the figure) can display at least one of the following: word spellings divided by syllables (as shown in the figure, word spellings are divided by small dots), word parts of speech (as shown in the figure, such as "adj." for adjectives and "n." for nouns), word phonetic symbols divided by syllables (as shown in the figure, word phonetic symbols are divided by short dashes), and an indication of the number of pronunciation problems (as shown in the figure, "pronunciation problems at x", Figures 4A-4B In the case of x=2, Figure 4C, x=1), a "standard pronunciation" button, a "your pronunciation" button, and a pronunciation diagnosis result. For example, when a user clicks the "standard pronunciation" button on the interactive interface, a demonstration recording corresponding to the standard pronunciation of the word can be played. For example, when a user clicks the "your pronunciation" button on the interactive interface, the user audio data (i.e., the first audio data) can be played.

[0164] For example, Figures 4A-4C As shown, the pronunciation diagnosis result may include whether there are syllable errors and / or pronunciation errors, and if there are syllable errors and / or pronunciation errors, what specific syllable errors and / or pronunciation errors exist.

[0165] For example, Figure 4A The syllable errors (specifically, the number of syllables) and pronunciation errors (specifically, vowel confusion errors, i.e., misreading vowel phonemes) that exist are reported exemplarily. Figure 4A As shown in , when reporting a syllable number error, the number of syllables in the word's phonetic symbol, the number of syllables in the user's pronunciation, and the difference between the two syllable numbers (more / less) can be indicated. Figure 4A As shown, when reporting a vowel confusion error, the syllable position containing the incorrect phoneme, the correct phoneme, the classification of the correct phoneme (monophony / diphthong), the incorrect phoneme pronounced by the user, the classification of the incorrect phoneme (monophony / diphthong), etc. can be indicated.

[0166] For example, Figure 4B The syllable errors (specifically, syllable stress errors, i.e., stress errors) and pronunciation errors (specifically, consonant omission errors, i.e., omission of consonant phonemes) that exist are reported exemplarily. Figure 4B As shown in , when reporting syllable stress errors, you can indicate the syllable position where the word's original stress is located, the syllable position where the user's pronunciation stress is located, and whether it is consistent with the word's original stress. For example, Figure 4B As shown, when reporting a consonant omission error, the position of the syllable containing the omitted consonant (which syllable), the position of the syllable where the omitted consonant is located (beginning / end / middle), the omitted consonant, etc. can be indicated.

[0167] For example, Figure 4C The absence of syllable errors and the presence of pronunciation problems (particularly, consonant confusion errors, i.e., mispronunciation of consonant phonemes) are reported exemplarily. Figure 4C As shown in FIG, when reporting that there are no syllable errors, it can be explained through symbols (e.g., an icon including a check mark) and text (e.g., no syllable problems found). For example, when reporting a consonant confusion error, the syllable position (syllable number), correct phoneme, category of the correct phoneme (consonant), incorrect phoneme, etc. containing the consonant confusion error can be reported.

[0168] It can be understood that the number x in the indication of the number of pronunciation problems ("pronunciation problems at x") is equal to the sum of the numbers of specific pronunciation problems in the pronunciation diagnosis results.

[0169] It is understandable that in practical applications, Figures 4A-4C The portion of the indicator arrow in the interactive interface does not exist, that is, the portion of the indicator arrow is not displayed in the interactive interface.

[0170] Understandably, one can refer to Figures 4A-4C The reporting method in Figures 4A-4C The other pronunciation diagnosis results other than those appearing in the report are not described here. Figures 4A-4C The word pronunciation correction interaction interfaces shown are all exemplary, and the embodiments of the present disclosure do not limit the reporting method of the pronunciation diagnosis results.

[0171] For example, Figures 4A-4C As shown, a "Start Pronunciation Correction" button can also be displayed on the word pronunciation correction interactive interface. When the user clicks the "Start Pronunciation Correction" button, the pronunciation correction guidance interface can be entered. For example, in some examples, such as Figures 4A-4C As shown, each specific pronunciation problem is provided with a corresponding "Start Pronunciation Correction" button. When the user clicks the "Start Pronunciation Correction" button, the user can enter the pronunciation correction guidance interface for the specific pronunciation problem; of course, the user can also enter the pronunciation correction guidance interface for all specific pronunciation problems. For example, in other examples, there is only one "Start Pronunciation Correction" button in the word pronunciation correction interactive interface. When the user clicks the "Start Pronunciation Correction" button, the user can enter the pronunciation correction guidance interface for all specific pronunciation problems.

[0172] It should be noted that the embodiments of the present disclosure do not limit the layout of the content displayed on the word pronunciation correction interactive interface.

[0173] It should be noted that, taking English as an example, the above-mentioned specific pronunciation problems basically cover all errors made by English learners whose native language is Chinese. At the same time, the pronunciation diagnosis operation in the pronunciation correction method provided by the embodiment of the present disclosure can achieve an error detection rate of more than 90%.

[0174] Step S300: Providing pronunciation correction guidance based on the pronunciation diagnosis result.

[0175] For example, in some embodiments, step S300 may include: in response to a pronunciation correction operation (for example, the user clicks the aforementioned "Start Pronunciation Correction" button), displaying the standard pronunciation of the word (for example, the phonetic symbols of the word divided into syllables, etc.), pronunciation diagnosis results, and text guidance, wherein the text guidance is used to guide the user to pronounce correctly.

[0176] Figure 5AA schematic diagram of a tone correction guidance interface provided for at least some embodiments of the present disclosure. Figure 5A The interface shown is a pronunciation correction guidance interface for incorrect number of syllables. Figure 5A As shown, the pronunciation correction guidance interface may include: display of word phonetic symbols divided by syllables (as shown in the figure, word phonetic symbols are divided by dashes), visual display of correct syllables (number, weight), display of phonetic symbols pronounced by the user, visual display of incorrect syllables pronounced by the user (number, weight), and text guidance. For example, Figure 5A As shown, in the visual display of syllables, the number of circles (of course, other graphic shapes are also possible) can represent the number of syllables; at the same time, the weight of the syllables can also be represented by the size of the circles. For example, a larger circle indicates that the corresponding syllable is a stressed syllable, and a smaller circle indicates that the corresponding syllable is an unstressed syllable. Figure 5A As shown, in the display of phonetic symbols, the problem syllable can be highlighted (for example, the brightness and / or color of the problem syllable is changed relative to the normal syllable), and further, the display color of the problem phoneme in the problem syllable can be changed (for example, the normal phoneme is displayed in black, and the problem phoneme is displayed in red, but not limited thereto); for example, correspondingly, the graphical representation of the problem syllable (for example, Figure 5A For example, Figure 5A As shown, in the pronunciation correction guidance interface for incorrect number of syllables, the text guidance can indicate the number of syllables contained in the word phonetic symbols, the number of syllables contained in the user's pronunciation, the consonants of the problem syllables pronounced by the user, the vowels after the consonants of the problem syllables pronounced by the user, and guidance methods for users to correct errors in the number of syllables.

[0177] Figure 5B A schematic diagram of another pronunciation correction guidance interface provided for at least some embodiments of the present disclosure. Figure 5B The interface shown is a pronunciation correction guidance interface for accent errors. Figure 5B As shown, the pronunciation correction guidance interface may include: display of word phonetic symbols divided by syllables (as shown in the figure, word phonetic symbols are divided by dashes), visual display of correct syllables (number, weight), visual display of incorrect syllables pronounced by the user (number, weight), and text guidance. Figure 5B As shown in FIG, in the display of phonetic symbols, the problem syllables that require the user's focus can be highlighted (for example, the brightness and / or color of the problem syllables are changed relative to the normal syllables). Figure 5BAs shown, in the visual display of syllables, the number of circles (of course, other graphic shapes are also possible) can represent the number of syllables; at the same time, the weight of the syllables can also be represented by the size of the circles. For example, a larger circle indicates that the corresponding syllable is a stressed syllable, and a smaller circle indicates that the corresponding syllable is an unstressed syllable. Figure 5B As shown, in the visual display of syllables (for example, the visual display of the wrong syllable pronounced by the user), a graphical representation of the problem syllable (for example, Figure 5B For example, Figure 5B As shown, in the pronunciation correction guidance interface for accent errors, the text guidance can indicate the number of syllables contained in the word phonetic symbols, a description of the user's pronunciation problem, and guidance methods for the user to correct accent errors.

[0178] Figure 5C A schematic diagram of yet another tone correction guidance interface provided for at least some embodiments of the present disclosure. Figure 5C The interface shown is a pronunciation guidance interface for consonant confusion errors. Figure 5C As shown, the pronunciation correction guidance interface may include: word spellings divided by syllables (as shown in the figure, word spellings are divided by small dots), word phonetic symbols divided by syllables (as shown in the figure, word phonetic symbols are divided by short dashes), the user's actual pronunciation phonemes and text guidance, etc. For example, Figure 5C As shown in , in the spelling display, the problem letters can be highlighted. Figure 5C As shown in , in the display of phonetic symbols, the problem phonemes can be highlighted. Figure 5C As shown, in the pronunciation correction guidance interface for consonant confusion errors, the text guidance can indicate the guidance method for the user to correct the consonant confusion errors (for example, the correct pronunciation method of the consonants), etc.

[0179] Figure 5D A schematic diagram of yet another tone correction guidance interface provided for at least some embodiments of the present disclosure. Figure 5D The interface shown is a guidance interface for correcting missed consonant pronunciation errors. Figure 5D As shown, the pronunciation correction guidance interface may include: displaying word spellings according to syllable segmentation (as shown in the figure, word spellings are segmented by small dots), displaying word phonetic symbols according to syllable segmentation (as shown in the figure, word phonetic symbols are segmented by short dashes), phonemes that the user has missed, and text guidance. For example, Figure 5D As shown in , in the spelling display, the problem letters can be highlighted. Figure 5D As shown in , in the display of phonetic symbols, the problem phonemes can be highlighted. Figure 5DAs shown, in the pronunciation correction guidance interface for missed consonant reading errors, the text guidance can indicate the guidance method for the user to correct the missed consonant reading error (for example, the correct pronunciation method for the missed consonant) and the like.

[0180] For example, in some embodiments, Figures 5A-5D As shown, step S300 may also include: when displaying the text guidance, using voice synchronization to play the text guidance (see Figures 5A-5D The text in the upper left corner says "Playing the pronunciation correction guide...").

[0181] Understandably, one can refer to Figures 5A-5D The correction guidance method in Figures 5A-5D The pronunciation problems other than those in the pronunciation problems are not explained here. Figures 5A-5D The pronunciation correction guidance interface provides pronunciation correction guidance for a specific pronunciation problem. However, in actual application, the pronunciation correction guidance interface can provide pronunciation correction guidance for multiple specific pronunciation problems at the same time.

[0182] It should be noted that Figures 5A-5D The illustrated pronunciation correction guidance interfaces are exemplary, and the embodiments of the present disclosure do not limit the pronunciation correction guidance method. It should also be noted that the embodiments of the present disclosure do not limit the layout of the content displayed on the pronunciation correction guidance interface.

[0183] For example, in some embodiments, after the user clicks the aforementioned "Start Pronunciation Correction" button, the user can directly switch from the word pronunciation correction interactive interface to the pronunciation correction guidance interface. For example, in other embodiments, after the user clicks the aforementioned "Start Pronunciation Correction" button, the user can first switch from the word pronunciation correction interactive interface to the transition interface, and then switch from the transition interface to the pronunciation correction guidance interface.

[0184] Figure 6 A schematic diagram of a transition interface provided for at least some embodiments of the present disclosure. Figure 6 As shown, the transition interface may include: word spellings divided by syllables (as shown in the figure, word spellings are divided by small dots), word parts of speech (as shown in the figure, "adj." for adjectives), word phonetic symbols divided by syllables (as shown in the figure, word phonetic symbols are divided by dashes), visual display of correct syllables (number, weight), "demonstration recording" control and "your pronunciation" control, etc. For example, Figure 6As shown, in the visual display of syllables, the number of circles (of course, other graphic shapes are also possible) can be used to represent the number of syllables; at the same time, the weight of the syllables can also be represented by the size of the circles. For example, a larger circle indicates that the corresponding syllable is a stressed syllable, and a smaller circle indicates that the corresponding syllable is an unstressed syllable. For example, when the user clicks the “Demonstration Recording” control on the transition interface, a demonstration recording corresponding to the standard pronunciation of the word can be played. For example, when the user clicks the “Your Pronunciation” control on the transition interface, the user audio data (i.e., the first audio data mentioned above) can be played. For example, when the transition interface is displayed, a language learning application can compare the user audio data and the demonstration recording in the background (see Figure 6 The text prompt in the upper left corner reads "Comparing pronunciation...") and prepares the content to be presented on the pronunciation correction guidance interface; when these contents are ready, switch to the pronunciation correction guidance interface.

[0185] Step S400: Acquire second audio data about the word and provide feedback on the second audio data.

[0186] For example, after completing the pronunciation correction guidance, the user may be provided with a pronunciation practice opportunity to test the effect of the pronunciation correction. For example, in some embodiments, the user's practice audio data (i.e., the second audio data) may be collected by the audio collection module or device of the client; then, referring to the aforementioned step S200, a pronunciation diagnosis operation is performed on the user's practice audio data (in this case, the user's practice audio data is regarded as the first audio data); if there is no pronunciation problem in the pronunciation diagnosis result of the user's practice audio data, feedback on the correct pronunciation is provided to the user; if there is a pronunciation problem in the pronunciation diagnosis result of the user's practice audio data, reference is made to the aforementioned steps S300 and S400 to continue the pronunciation correction guidance and provide practice feedback.

[0187] Figure 7A A schematic diagram of a practice interface provided for at least some embodiments of the present disclosure. For example, Figure 7A The interface shown is for errors in the number of syllables (e.g. Figure 5A The number of syllables shown in the example is incorrect). Figure 7A As shown, the practice interface may include: display of word spellings divided according to syllables (as shown in the figure, word spellings are divided by small dots), display of word phonetic symbols divided according to syllables (as shown in the figure, word phonetic symbols are divided by dashes), visual display of correct syllables (quantity, weight), a "demonstration recording" control (as shown in the sound control icon in the figure) and a "start recording" button, etc.

[0188] Figure 7B A schematic diagram of another practice interface provided for at least some embodiments of the present disclosure. For example, Figure 7BThe interface shown is for consonant confusion errors (e.g. Figure 5C For example, Figure 7B As shown, the practice interface may include: display of word spellings divided according to syllables (as shown in the figure, word spellings are divided by small dots), display of word phonetic symbols divided according to syllables (as shown in the figure, word phonetic symbols are divided by short dashes), a "demonstration recording" control (as shown in the sound control icon in the figure) and a "start recording" button, etc.

[0189] Understandably, one can refer to Figures 7A-7B The practice interface shown is Figures 7A-7B Corresponding practice interfaces are provided for pronunciation problems other than those involved, which will not be discussed here.

[0190] It should be noted that Figures 7A-7B The practice interfaces shown are all exemplary, and the embodiments of the present disclosure do not limit the content displayed on the practice interface and the layout of the content on the practice interface.

[0191] For example, in some embodiments, after entering the practice interface, the demonstration recording and the follow-up voice prompts can be automatically played (see Figures 7A-7B Then, in response to the user clicking (e.g., long pressing) the "Start Recording" button, the audio data of the user's practice is collected by the audio collection module or device. It should be noted that the embodiments of the present disclosure do not limit the order in which the demonstration recording and the follow-up voice prompt are played. For example, before clicking (e.g., long pressing) the "Start Recording" button, the user can click the "Demonstration Recording" control to listen to the demonstration recording repeatedly.

[0192] For example, if there are pronunciation problems in the pronunciation diagnosis results of the user's practice audio data, the pronunciation diagnosis results can be displayed on the corresponding word correction interactive interface (for example, you can refer to Figures 4A-4C On this basis, the related operations of step S300 and step S400 may be further performed.

[0193] For example, if there is no pronunciation problem in the pronunciation diagnosis result of the audio data practiced by the user, a feedback interface indicating that the pronunciation is correct may be provided to the user.

[0194] Figure 7C A schematic diagram of a feedback interface provided for at least some embodiments of the present disclosure. For example, Figure 7C The interface shown is for accent errors (e.g. Figure 5B For example, Figure 7CAs shown, the feedback interface may include: word spellings divided by syllables (as shown in the figure, word spellings are divided by small dots), word phonetic symbols divided by syllables (as shown in the figure, word phonetic symbols are divided by dashes), visual display of correct syllables (number, weight), problem correction prompts (as shown in the figure, including a check mark icon and the text "Your pronunciation syllable stress is correct") and a "Done" button. For example, Figure 7C As shown in , in the spelling display, the stressed syllables can be highlighted to emphasize them. Figure 7C As shown, in the display of phonetic symbols, stressed syllables can be highlighted to emphasize them.

[0195] Figure 7D A schematic diagram of another feedback interface provided for at least some embodiments of the present disclosure. For example, Figure 7D The interface shown is for a consonant confusion error (e.g., Figure 5C For example, Figure 7D As shown in the figure, the feedback interface may include: display of word spellings divided by syllables (as shown in the figure, word spellings are divided by small dots), display of word phonetic symbols divided by syllables (as shown in the figure, word phonetic symbols are divided by dashes), problem correction prompts (as shown in the figure, including a check mark icon and the text "Your pronunciation no longer confuses / v / and / w / !") and a "Done" button. For example, Figure 7D As shown in , in the spelling display, the problem letters can be highlighted to emphasize the effect. Figure 7D As shown, in the display of phonetic symbols, the problem phonemes can be highlighted to emphasize them.

[0196] For example, in some embodiments, the feedback interface may also include feedback text (such as Figures 7C-7D The text "Great" in the upper left corner is shown to encourage the user. For example, the above feedback copy can also be played synchronously with voice.

[0197] For example, in response to the user clicking (eg, long pressing) the "Done" button, the feedback interface may be closed, ending the practice of the current word.

[0198] It should be noted that, in the embodiments of the present disclosure, the process of the above-mentioned pronunciation correction method may include more or fewer operations, and these operations may be performed sequentially or in parallel. Although the process of the pronunciation correction method described above includes multiple operations that appear in a specific order, it should be clearly understood that the order of multiple operations is not limited. The pronunciation correction method described above can be performed once or multiple times according to predetermined conditions.

[0199] The pronunciation correction method provided by the embodiments of the present disclosure uses a "dual-model, two-pass decoding" method to perform pronunciation diagnosis operations. This method can quickly and easily obtain pronunciation diagnosis results, allowing users to specifically correct existing pronunciation problems based on the pronunciation diagnosis results, thereby improving the user's language learning efficiency and having high practicality. Furthermore, the pronunciation correction method provided by the embodiments of the present disclosure can also identify confusion, addition, deletion errors, and stress errors of single or multiple phonemes, and can provide correction feedback from both syllable and pronunciation dimensions, guiding users from correcting incorrect pronunciation to demonstrating pronunciation.

[0200] At least some embodiments of the present disclosure also provide a sound correction device. Figure 8 A schematic block diagram of a sound correction device provided in at least some embodiments of the present disclosure. Figure 8 As shown, the sound correction device 100 includes a memory 110 and a processor 120 .

[0201] For example, the memory 110 is used to non-temporarily store computer-readable instructions, and the processor 120 is used to execute the computer-readable instructions. When the computer-readable instructions are executed by the processor 120, the sound correction method provided by any embodiment of the present disclosure is executed.

[0202] For example, the memory 110 and the processor 120 may communicate with each other directly or indirectly. Figure 8 As shown, the sound correction device 100 may further include a system bus 130, and the memory 110 and the processor 120 may communicate with each other through the system bus 130. For example, the processor 120 may access the memory 110 through the system bus 130. For example, in other examples, the memory 110 and the processor 120 and other components may communicate through a network connection. The network may include a wireless network, a wired network, and / or any combination of a wireless network and a wired network. The network may include a local area network, the Internet, a telecommunications network, an Internet of Things based on the Internet and / or a telecommunications network, and / or any combination of the above networks. For example, a wired network may communicate using twisted pair, coaxial cable or optical fiber transmission, and a wireless network may communicate using 3G / 4G / 5G mobile communication networks, Bluetooth, Zigbee or WiFi. The present disclosure does not limit the type and function of the network.

[0203] For example, the processor 120 can control other components in the audio correction device to perform desired functions. The processor 120 can be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). The central processing unit (CPU) can be an X86 or ARM architecture. The GPU can be directly integrated into the motherboard or built into the motherboard's north bridge chip. The GPU can also be built into the central processing unit (CPU).

[0204] For example, the memory 110 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, etc.

[0205] For example, the memory 110 may store one or more computer instructions, and the processor 120 may execute the computer instructions to implement various functions. The computer-readable storage medium may also store various applications and various data, such as words, first audio data, first acoustic model, second acoustic model, pronunciation diagnosis results, second audio data, and various data used and / or generated by the application.

[0206] For example, when some computer instructions stored in the memory 210 are executed by the processor 220 , one or more steps in the sound correction method described above can be performed.

[0207] For example, Figure 8As shown, the sound correction device 100 can also include an input interface 140 that allows external devices to communicate with the sound correction device 100. For example, the input interface 140 can be used to receive instructions from external computer equipment, from users, etc. The sound correction device 100 can also include an output interface 150 that interconnects the sound correction device 100 and one or more external devices. For example, the sound correction device 100 can output sound correction results, etc., through the output interface 150. The external device that communicates with the sound correction device 100 through the input interface 140 and the output interface 150 can be included in the environment of providing any type of user interface that the user can interact with. Examples of user interface types include graphical user interfaces, natural user interfaces, etc. For example, graphical user interfaces can accept input from users using input devices such as keyboards, mice, remote controllers, etc., and provide output on output devices such as displays. In addition, natural user interfaces can enable users to interact with the sound correction device 100 in a manner that does not require the constraints imposed by input devices such as keyboards, mice, remote controllers, etc. In contrast, natural user interfaces may rely on speech recognition, touch and stylus recognition, gesture recognition on and near the screen, mid-air gestures, head and eye tracking, voice and semantics, vision, touch, gestures, and machine intelligence, among others.

[0208] For example, in some embodiments, the sound correction device 100 may further include an audio collection device ( Figure 8 For example, the audio collection device may be the audio collection module or device described in the embodiment of the aforementioned sound correction method, including but not limited to a built-in or external microphone of the client.

[0209] In addition, although the tone correction device 100 Figure 8 Although shown as a single system, it is understood that the pronunciation correction device 100 may also be a distributed system and may also be arranged as a cloud facility (including a public cloud or a private cloud). Thus, for example, several devices may communicate via a network connection and may jointly perform the tasks described as being performed by the pronunciation correction device 100. For example, in some embodiments, a word and first audio data may be obtained through a client and uploaded to a server; after the server performs a pronunciation diagnosis operation, the pronunciation diagnosis result is returned to the client to be provided to the user, and the server may further provide pronunciation correction guidance operations; thereafter, second audio data may be obtained through the client and uploaded to the server, and the server performs a pronunciation diagnosis operation on the second audio data and provides feedback.

[0210] For example, for a detailed description of the implementation process of the pronunciation correction method, reference may be made to the relevant description in the embodiment of the pronunciation correction method described above, and repeated parts will not be repeated here.

[0211] For example, in some examples, the pronunciation correction device may include but is not limited to a smartphone, a tablet computer, a personal computer, a personal digital assistant (PDA), a wearable device, a head-mounted display device, a scanning pen, a point reading pen, a server, etc.

[0212] It should be noted that the sound correction device provided in the embodiments of the present disclosure is exemplary and non-restrictive. According to actual application needs, the sound correction device may also include other conventional components or structures. For example, in order to realize the necessary functions of the sound correction device, those skilled in the art may set other conventional components or structures according to the specific application scenarios, and the embodiments of the present disclosure do not limit this.

[0213] The technical effects of the sound correction device provided by the embodiments of the present disclosure can be referred to the corresponding description of the sound correction method in the above embodiments, which will not be repeated here.

[0214] At least some embodiments of the present disclosure also provide a non-transitory storage medium. Figure 9 A schematic block diagram of a non-transitory storage medium provided in some embodiments of the present disclosure. Figure 9 As shown, the non-transitory storage medium 200 non-transitory stores computer-readable instructions 201 , and when the non-transitory computer-readable instructions 201 are executed by a computer (including a processor), the sound correction method provided by any embodiment of the present disclosure can be executed.

[0215] For example, one or more computer-readable instructions may be stored on the non-transitory storage medium 200. Some of the computer-readable instructions stored on the non-transitory storage medium 200 may be, for example, instructions for implementing one or more steps in the above-mentioned pronunciation correction method.

[0216] For example, the non-transitory storage medium may include a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a compact disc read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage media.

[0217] The technical effects of the non-transitory storage medium provided by the embodiments of the present disclosure can be referred to the corresponding description of the sound correction method in the above embodiments, which will not be repeated here.

[0218] Regarding this disclosure, the following points need to be explained:

[0219] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to conventional designs.

[0220] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0221] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A method for correcting pronunciation, comprising: Obtaining a word and first audio data; Based on the word, performing a pronunciation diagnosis operation on the first audio data to generate a pronunciation diagnosis result; Wherein, the standard pronunciation of the word includes at least one standard phoneme; Based on the word, performing the pronunciation diagnosis operation on the first audio data to generate the pronunciation diagnosis result includes: Based on the first acoustic model, aligning the first audio data with the standard pronunciation to obtain a time boundary of each standard phoneme in the standard pronunciation in the first audio data; Determining a score for each standard phoneme according to an audio segment determined by a time boundary of each standard phoneme; performing a recognition operation on the first audio data based on a second acoustic model to obtain a decoded phoneme sequence and a time boundary of each decoded phoneme in the decoded phoneme sequence in the first audio data, wherein the decoded phoneme sequence includes at least one decoded phoneme; determining a score for each decoded phoneme according to an audio segment determined by a time boundary of each decoded phoneme; Determining a correspondence between each standard phoneme in the standard pronunciation and each decoded phoneme in the decoded phoneme sequence; and generating the pronunciation diagnosis result based on the correspondence, the scores of the respective standard phonemes, and the scores of the respective decoded phonemes; Generating the pronunciation diagnosis result based on the corresponding relationship, the score of each standard phoneme, and the score of each decoded phoneme includes: In response to any standard phoneme having a corresponding decoded phoneme, determining whether the score of the any standard phoneme is lower than a first score threshold; In response to the score of the any standard phoneme being lower than the first score threshold, calculating a degree of boundary coincidence between the any standard phoneme and the decoded phoneme corresponding to the any standard phoneme based on a time boundary of the any standard phoneme and a time boundary of the decoded phoneme corresponding to the any standard phoneme; and In response to the fact that any of the standard phonemes is different from the decoded phonemes corresponding to the any of the standard phonemes and the boundary overlap between the any of the standard phonemes and the decoded phonemes corresponding to the any of the standard phonemes is not less than the overlap threshold, it is indicated in the pronunciation diagnosis result that a misreading of the any of the standard phonemes has occurred.

2. The method for correcting sound according to claim 1, wherein: Determining the correspondence between each standard phoneme in the standard pronunciation and each decoded phoneme in the decoded phoneme sequence includes: Using phonemes as editing elements, an edit distance operation is performed on the standard pronunciation and the decoded phoneme sequence to determine the corresponding relationship.

3. The method for correcting sound according to claim 2, wherein: The edit distance operation includes a phoneme replacement operation, and the weights of the phoneme replacement operations between different phonemes are at least not completely the same.

4. The method for correcting sound according to claim 1, wherein: Generating the pronunciation diagnosis result based on the corresponding relationship, the score of each standard phoneme and the score of each decoded phoneme further includes: determining whether a difference between a score of a decoded phoneme corresponding to any one of the standard phonemes and a score of any one of the standard phonemes is not less than a second score threshold; and In response to the difference between the score of the decoded phoneme corresponding to the any standard phoneme and the score of the any standard phoneme being not less than the second score threshold, it is indicated in the misreading case that the any standard phoneme is misread as the decoded phoneme corresponding to the any standard phoneme.

5. The method for correcting sound according to claim 1, wherein: The boundary overlap is calculated according to the following formula: Among them, BC represents the boundary coincidence, x1 and y1 represent the starting time boundary and ending time boundary of a standard phoneme respectively, x2 and y2 represent the starting time boundary and ending time boundary of a decoded phoneme respectively, min() is the minimum value function, and max() is the maximum value function.

6. The method for correcting tone according to any one of claims 1 to 3, wherein: Generating the pronunciation diagnosis result based on the corresponding relationship, the score of each standard phoneme, and the score of each decoded phoneme includes: In response to any standard phoneme not having a corresponding decoded phoneme, it is indicated in the pronunciation diagnosis result that a reading omission occurs for the any standard phoneme.

7. The method for correcting sound according to any one of claims 1 to 3, wherein: Generating the pronunciation diagnosis result based on the corresponding relationship, the score of each standard phoneme, and the score of each decoded phoneme includes: In response to any decoded phoneme not having a corresponding standard phoneme, it is indicated in the pronunciation diagnosis result that multiple readings have occurred.

8. The method for correcting tone according to claim 7, wherein: Generating the pronunciation diagnosis result based on the corresponding relationship, the score of each standard phoneme and the score of each decoded phoneme further includes: In response to the score of the any one decoded phoneme being not lower than a third score threshold, it is indicated in the multiple reading case that the any one decoded phoneme is multiple read.

9. The method for correcting sound according to any one of claims 1 to 3, wherein: Based on the word, performing the pronunciation diagnosis operation on the first audio data to generate the pronunciation diagnosis result, further comprising: Determining the time boundary of the vowel phoneme in the stressed syllable based on the time boundary of each standard phoneme in the standard pronunciation and the stressed syllable of the standard pronunciation; extracting feature information of a first audio segment determined by a time boundary of the vowel phoneme in the stressed syllable; Based on the feature information of the first audio segment, determining whether the stressed syllable is stressed by using a classification model; and In response to the stressed syllable being determined to be unstressed, indicating that the stressed syllable is unstressed is provided in the pronunciation diagnosis result.

10. The method for correcting sound according to any one of claims 1 to 3, wherein: Based on the word, performing the pronunciation diagnosis operation on the first audio data to generate the pronunciation diagnosis result, further comprising: Determining the time boundary of the vowel phoneme in the unstressed syllable based on the time boundary of each standard phoneme in the standard pronunciation and the unstressed syllable in the standard pronunciation; extracting feature information of a second audio segment determined by a time boundary of the vowel phoneme in the unstressed syllable; Based on the feature information of the second audio segment, determining whether the unstressed syllable is stressed by using a classification model; and In response to the unstressed syllable being determined to be stressed, it is indicated in the pronunciation diagnosis result that the unstressed syllable is stressed.

11. The method for correcting sound according to any one of claims 1 to 3, wherein: Based on a pronunciation accuracy algorithm, a score of each standard phoneme and a score of each decoded phoneme are determined.

12. The method for correcting sound according to any one of claims 1 to 3, further comprising: Pronunciation correction guidance is provided based on the pronunciation diagnosis result.

13. The method for correcting sound according to claim 12, wherein: Providing the pronunciation correction guidance according to the pronunciation diagnosis result includes: In response to the pronunciation correction operation, the standard pronunciation of the word, the pronunciation diagnosis result and text guidance are displayed, wherein the text guidance is used to guide the user to pronounce the word correctly.

14. The method for correcting sound according to claim 13, wherein: Providing the pronunciation correction guidance according to the pronunciation diagnosis result further includes: When presenting the text guidance, the text guidance is played synchronously with voice.

15. The method for correcting sound according to claim 12, further comprising: Second audio data about the word is acquired, and practice feedback for the second audio data is provided.

16. The method for correcting tone according to any one of claims 1 to 3, wherein: The pronunciation diagnosis result includes at least one of syllable error and pronunciation error, The syllable error includes at least one of a syllable number error and a stress error. The pronunciation errors include at least one of misreading vowel phonemes, misreading consonant phonemes and missing consonant phonemes.

17. A tone correction device comprising: a memory for non-transitory storage of computer-readable instructions; as well as A processor is configured to execute the computer-readable instructions, wherein the computer-readable instructions, when executed by the processor, execute the sound correction method according to any one of claims 1 to 16.

18. The sound correction device according to claim 17, further comprising: An audio collection device is used to collect the first audio data.

19. A non-transitory storage medium that non-transitory stores computer-readable instructions, wherein: When the non-transitory computer-readable instructions are executed by a computer, the sound correction method according to any one of claims 1 to 16 can be performed.

Images