Voice evaluation method, device, equipment and storage medium

Abstract

The present invention discloses a speech evaluation method, apparatus, device, and storage medium. The speech evaluation comprises: determining the pronunciation of a phoneme to be tested, wherein the pronunciation conditions include correct pronunciation and incorrect pronunciation; and for each pronunciation condition, determining the GOP value corresponding to the phoneme to be tested using a GOP calculation scheme corresponding to the pronunciation condition. The technical solution of this embodiment distinguishes different pronunciation conditions, using different GOP calculation schemes for different pronunciation conditions, ensuring that the GOP value is not affected when the pronunciation is correct; and ensuring that an accurate GOP score can be obtained even when the pronunciation is incorrect, thereby reducing the impact of incorrect pronunciation on the speech evaluation results and improving the accuracy of the speech evaluation results.

Description

Technical Field

[0001] The present invention relates to a data processing technology, and more particularly to a speech evaluation method, apparatus, device, and storage medium. Background Art

[0002] Pronunciation quality assessment technology is a sub-discipline of computer-assisted language learning (CALL). Pronunciation quality assessment technology requires efficiently and accurately pointing out learners' pronunciation errors, providing objective evaluation at the phoneme level, and helping learners correct pronunciation errors.

[0003] Goodness of Pronunciation (GOP) is a commonly used feature in pronunciation assessment. It measures pronunciation at the phoneme level. Existing assessment models often use triphones as modeling units. When a phoneme in a sentence is mispronounced, the triphone structure causes the mispronunciation to affect the GOP values of the two preceding and following phonemes, thereby reducing the speech assessment results. Summary of the Invention

[0004] The embodiments of the present invention provide a speech evaluation method, apparatus, device and storage medium, which reduce the impact of incorrect pronunciation on speech evaluation results and improve the accuracy of speech evaluation results.

[0005] In a first aspect, an embodiment of the present invention provides a speech evaluation method, comprising:

[0006] Determining the pronunciation of the phoneme to be tested, wherein the pronunciation includes correct pronunciation and incorrect pronunciation;

[0007] For each pronunciation situation, the GOP value corresponding to the phoneme to be measured is determined using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation.

[0008] In one embodiment, determining the pronunciation of the phoneme to be tested includes:

[0009] During the phoneme duration, determining frame by frame whether the phoneme corresponding to the maximum posterior probability of the state of the acoustic model is the same as the phoneme to be tested;

[0010] If the phoneme corresponding to the maximum a posteriori probability of the state of the acoustic model is the same as the phoneme to be tested, it is determined that the pronunciation of the phoneme to be tested is correct;

[0011] If the phoneme corresponding to the maximum a posteriori probability of the state of the acoustic model is different from the phoneme to be tested, it is determined that the phoneme to be tested is mispronounced.

[0012] In one embodiment, determining the pronunciation of the phoneme to be tested includes:

[0013] Determine the cumulative state posterior probability corresponding to each phoneme state within the phoneme duration;

[0014] Determine the first phoneme corresponding to the maximum cumulative state posterior probability among all phoneme states;

[0015] If the first phoneme is identical to the phoneme to be tested, determining that the phoneme to be tested is a correctly pronounced phoneme;

[0016] If the first phoneme is different from the phoneme to be tested, it is determined that the phoneme in the audio to be tested is a mispronounced phoneme.

[0017] In one embodiment, for each pronunciation situation, determining the GOP value corresponding to the phoneme using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation includes:

[0018] When the pronunciation is correct, the GOP value corresponding to the phoneme to be tested is determined by summing the state posterior probabilities of all three phonemes centered on the phoneme to be tested;

[0019] When the pronunciation condition is a pronunciation error, the state posterior probability value of a single triphone centered on the phoneme to be tested is used to determine the GOP value corresponding to the phoneme to be tested.

[0020] In one embodiment, for each pronunciation situation, determining the GOP value corresponding to the phoneme using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation includes:

[0021] When the pronunciation is correct, the GOP value corresponding to the phoneme to be measured is calculated using the following first GOP formula;

[0022] Among them, the first GOP formula is:

[0023]

[0024] Among them, t e is the starting time of the phoneme to be tested, t s is the end time of the phoneme to be tested, p is the phoneme to be tested, s p is a triphone state with the phoneme to be tested being p, S is a triphone state set, s is a triphone state belonging to S, o t is the speech feature of the t-th frame, P(s p |o t )、P(s|o t ) is the posterior probability output by the acoustic model.

[0025] In one embodiment, for each pronunciation situation, determining the GOP value corresponding to the phoneme using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation includes:

[0026] When the pronunciation condition is a pronunciation error, the GOP value corresponding to the phoneme to be measured is calculated by the following second GOP formula;

[0027] Among them, the second GOP formula is:

[0028]

[0029] Among them, t e is the starting time of the phoneme to be tested, t s is the end time of the phoneme to be tested, p is the phoneme to be tested, S is a set of three phoneme states, s is a three phoneme state belonging to S, o t is the speech feature of the t-th frame, P(p|o t )、P(s|o t ) is the posterior probability output by the acoustic model.

[0030] In one embodiment, after determining the GOP value corresponding to the phoneme to be tested by using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation, the method further includes:

[0031] When the GOP value corresponding to the phoneme to be tested is greater than or equal to a preset GOP threshold, determining that the pronunciation of the phoneme to be tested is qualified;

[0032] When the GOP value corresponding to the phoneme to be measured is less than a preset GOP threshold, it is determined that the pronunciation of the phoneme to be measured is unqualified.

[0033] In a second aspect, an embodiment of the present invention further provides a speech evaluation device, comprising:

[0034] A pronunciation status determination module, configured to determine the pronunciation status of the phoneme to be tested, wherein the pronunciation status includes correct pronunciation and incorrect pronunciation;

[0035] The GOP determination module is used to determine the GOP value corresponding to the phoneme to be measured by using the pronunciation goodness GOP calculation scheme corresponding to each pronunciation situation.

[0036] In a third aspect, an embodiment of the present invention further provides a speech evaluation device, comprising:

[0037] one or more processors;

[0038] a memory for storing one or more programs;

[0039] The one or more programs are executed by the one or more processors, so that the one or more processors implement the speech evaluation method provided in the first aspect above.

[0040] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having one or more computer programs stored thereon, which, when executed by a processor, implements the speech evaluation method provided in the first aspect above.

[0041] In the speech assessment methods, devices, equipment, and storage media provided in the above-mentioned embodiments, the speech assessment includes: determining the pronunciation of the phoneme to be tested, where pronunciation conditions include correct pronunciation and incorrect pronunciation; and, for each pronunciation condition, determining the GOP value corresponding to the phoneme to be tested using a GOP calculation scheme corresponding to the pronunciation condition. The technical solution of this embodiment distinguishes between different pronunciation conditions, using different GOP calculation schemes for different pronunciation conditions, ensuring that the GOP value is not affected when the pronunciation is correct; and ensuring that an accurate GOP score is also generated when the pronunciation is incorrect, thereby reducing the impact of incorrect pronunciation on the speech assessment results and improving the accuracy of the speech assessment results. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a diagram of the three phonemes of the word park;

[0043] Figure 2a This is an example diagram of an application scenario provided by an embodiment of the present invention;

[0044] Figure 2b is another example diagram of an application scenario provided by an embodiment of the present invention;

[0045] Figure 3 is a flow chart of a speech evaluation method provided by an embodiment of the present invention;

[0046] Figure 4 It is a schematic diagram of the phoneme corresponding to the maximum posterior probability of the frame state and the phoneme to be tested;

[0047] Figure 5 A schematic diagram of the structure of a speech evaluation device provided in an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the hardware structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0050] Existing speech evaluation technologies often use triphones as modeling units. When a phoneme in a sentence is mispronounced, the triphone structure makes the error in the current phoneme affect the GOP values of the two phonemes before and after it. Figure 1 Take the word park as an example to illustrate. The corresponding phoneme p aa rk has the following three-phoneme sequences: sil-p+aa, p-aa+r, aa-r+k, and r-k+sil. Sil represents a silent phoneme. If aa is pronounced as er, the posterior probability of the three-phoneme p-aa+r will become lower. The posterior probabilities of sil-p+aa and aa-r+k will also become lower, while the posterior probabilities of sil-p+er and er-r+k will become higher, which will affect the forced alignment results.

[0051] Therefore, based on the above problems, the embodiments of the present invention provide a speech evaluation method, apparatus, device and storage medium. By adopting different GOP calculation schemes for correctly pronounced phonemes and incorrectly pronounced phonemes, it is ensured that the GOP value will not be affected when the pronunciation is correct; and it is ensured that an accurate GOP score can be obtained even when the pronunciation is incorrect, thereby reducing the impact of incorrect pronunciation on the speech evaluation results and improving the accuracy of the speech evaluation results.

[0052] This solution can be used in areas including, but not limited to, pronunciation error detection and diagnosis in speech assessment. For example, online or offline speech assessment systems can provide pronunciation error detection for language learners based on the accuracy of the learner's pronunciation assessment results. For example, native Chinese speakers learning a foreign language can be used.

[0053] Figure 2a This is an example diagram of an application scenario provided by an embodiment of the present invention. Figure 2a As shown, server 102 is used to execute the speech evaluation method described in any of the method embodiments of this application. Client 101 receives speech from a user through an input device. Server 102 interacts with client 101 to obtain the speech. After executing the speech evaluation method, server 102 outputs the pronunciation evaluation results to client 101, which is then notified to the learner via the output device of client 101. Furthermore, client 101 provides the correct pronunciation to the learner to help them correct their pronunciation.

[0054] The input device may be an input device built into the computer device, such as a built-in voice input device, or an external input device connected to the computer device via a communication line, such as a microphone. Furthermore, the output device may be an output device built into the computer device, such as a touch screen display, or an external output device connected to the computer device via a communication line, such as a projector or digital TV.

[0055] It should be noted that in this embodiment, the client 101 is described using a computer as an example. The computer device may specifically be a computer device including a processor, a memory, an input device, and an output device, such as a laptop computer, a desktop computer, a tablet computer, a smart terminal, a learning machine, an early childhood education machine, a smart wearable device, etc.

[0056] Alternatively, when the client 101 has a certain data processing capability, that is, when the client 101 has a processor and a memory, the client 101 can independently serve as the execution subject of the pronunciation evaluation method described in any embodiment of the present application, such as Figure 2b As an example. Figure 2b In the process, the learner presses the microphone, and the built-in voice collection device of the mobile phone can collect the voice of the user in real time. After executing the above-mentioned voice evaluation method, the evaluation result of the tested voice is displayed to the learner through the display screen.

[0057] The pronunciation evaluation method provided by the present invention is explained below with reference to specific embodiments.

[0058] Figure 3 This is a flow chart of a speech assessment method provided by an embodiment of the present invention. This method is suitable for detecting whether a learner's pronunciation is correct. The speech assessment method can be performed by a speech assessment device, which can be implemented in hardware and / or software. The speech assessment device can be composed of two or more physical entities, or a single physical entity, and is generally integrated into a computer device.

[0059] like Figure 3 As shown, the speech evaluation method provided by the embodiment of the present invention mainly includes the following steps:

[0060] S21. Determine the pronunciation of the phoneme to be tested, wherein the pronunciation includes correct pronunciation and incorrect pronunciation.

[0061] Among them, the phoneme refers to the smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one action constitutes one phoneme. Generally speaking, a phoneme is a more detailed unit of a word (Chinese: character). For example, the phonemes of "one" are the three phonemes "w", "ah", and "n", and the phonemes of "一" are the two phonemes "yh" and "i_1" (each phoneme can be used as an independent pronunciation unit). It can be simply understood that the phonemes of Chinese are the initials and finals of the pronunciation of the character, and the pronunciation of an English word can be understood as the phonetic transcription of the word; in speech recognition, a phoneme is trained by a large amount of speech data. The simplest way to understand it is to find many people to pronounce "w" (woo), and then extract the features of these audio signals to train the phoneme of "w". A phoneme is composed of multiple states. For example, a phoneme is composed of three states, and at least one frame of duration is allocated to each of these three states. The duration when the phoneme is pronounced is greater than three frames. Among them, the phoneme to be measured can be understood as one or more phonemes divided from the speech to be measured.

[0062] Among them, the speech to be measured can be understood as the speech collected by the client from the learner for the pronunciation text. Further, it can be the sound directly collected by the learner through the microphone, or the sound of the learner obtained by retrieving the pre-recorded voice. The learner refers to the person who needs to train their own pronunciation according to the pronunciation text.

[0063] In practical applications, when the learner reads a pronunciation text, the corresponding speech of the pronunciation text will be generated. The electronic device first obtains this speech and determines whether the learner's pronunciation is correct by evaluating this speech signal. Exemplarily, the above pronunciation text can specifically be at least one word, or even at least one phoneme.

[0064] Further, taking a learning machine as an example for illustration, when the learner reads the pronunciation text on the display interface of the learning machine, the learning machine collects the speech signal through a sound pickup device such as a microphone to obtain the speech to be measured, and divides the speech to be measured into one or more phonemes through a phoneme division method, and then evaluates and scores each phoneme.

[0065] In this embodiment, a method for determining the phoneme to be measured is provided. Based on the pronunciation text and the speech to be measured, the speech to be measured is decomposed to obtain the phonemes and boundary information included in the speech to be measured, and the phoneme state sequence is composed of the states corresponding to these phonemes. That is to say, the phoneme state sequence contains the phonemes to be measured corresponding to the speech to be measured.

[0066] In one implementation, the pronunciation text and the speech to be measured are aligned. Further, using the pre-trained acoustic model, the acoustic score of each frame in the speech to be measured is calculated, and then the Viterbi algorithm is used to search for the optimal path in the alignment network to obtain the phoneme state sequence and boundary information of the speech to be measured.

[0067] The Viterbi algorithm is a widely used dynamic programming algorithm in machine learning. It is used to find the most likely hidden state sequence—the Viterbi path—that generates a sequence of observed events, particularly in the context of Markov information sources and hidden Markov models. The term "Viterbi algorithm" is also used to refer to a dynamic programming algorithm that finds the most likely explanation for an observation. In the embodiments of the present invention, the Viterbi algorithm is used to search for the optimal path in an alignment network to obtain a phoneme state sequence.

[0068] The acoustic model can be constructed using a Deep Neural Network (DNN)-Hidden Markov Model (HMM) approach, effectively a DNN-HMM acoustic model. The speech signal to be detected is input frame by frame into the DNN-HMM acoustic model, which outputs the corresponding state posterior probabilities for each frame. These are converted into acoustic scores, and the Viterbi algorithm is used to search for the optimal path to obtain the first phoneme state sequence and boundary information. The Viterbi algorithm's path search objective is to find an optimal path within the WFST alignment network that matches the speech feature sequence.

[0069] Furthermore, after the phoneme to be tested is determined according to the above method, the pronunciation of the phoneme to be tested is determined, that is, whether the phoneme to be tested is pronounced correctly or incorrectly.

[0070] In one embodiment, within the duration of any phoneme to be tested, a determination is made frame by frame whether the phoneme corresponding to the maximum a posteriori probability of the state of the acoustic model is identical to the phoneme to be tested. If so, the pronunciation of the phoneme to be tested is determined to be correct; if not, the pronunciation of the phoneme to be tested is determined to be incorrect.

[0071] In another embodiment, during the duration of the phoneme to be tested, for each phoneme state, the state output posterior probability is accumulated, and the phoneme corresponding to the maximum state output probability is found. When the phoneme is the same as the phoneme to be tested, it is determined that the pronunciation of the phoneme to be tested is correct; if they are different, it is determined that the pronunciation of the phoneme to be tested is incorrect.

[0072] It should be noted that there are various ways and criteria for determining the pronunciation of the phoneme to be tested, and the above-mentioned ways are merely exemplary and not limiting.

[0073] S22 . For each pronunciation situation, determine the GOP value corresponding to the phoneme to be measured using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation.

[0074] Among them, GOP is a commonly used feature in pronunciation evaluation, and the pronunciation goodness is the pronunciation evaluation result at the phoneme level.

[0075] In this embodiment, when the phoneme being measured is pronounced correctly, the first method is used to determine the GOP value corresponding to the phoneme being measured. When the phoneme being measured is pronounced incorrectly, the second method is used to determine the GOP value corresponding to the phoneme being measured. This solves the problem in the prior art of using the same GOP calculation method regardless of whether the phoneme being measured is pronounced correctly, resulting in low pronunciation assessment accuracy.

[0076] The speech evaluation method provided in this embodiment includes: determining the pronunciation of a phoneme to be tested, where pronunciation conditions include correct pronunciation and incorrect pronunciation; and, for each pronunciation condition, determining the GOP value corresponding to the phoneme to be tested using a GOP calculation scheme corresponding to the pronunciation condition. The technical solution of this embodiment distinguishes between different pronunciation conditions, using different GOP calculation schemes for each pronunciation condition, ensuring that the GOP value is not affected when pronunciation is correct; and also ensuring that an accurate GOP score is obtained even when pronunciation is incorrect, thereby reducing the impact of incorrect pronunciation on the speech evaluation results and improving the accuracy of the speech evaluation results.

[0077] Furthermore, after determining the GOP value corresponding to the phoneme to be tested by using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation, it also includes: when the GOP value corresponding to the phoneme to be tested is greater than or equal to a preset GOP threshold, determining that the pronunciation of the phoneme to be tested is qualified; when the GOP value corresponding to the phoneme to be tested is less than the preset GOP threshold, determining that the pronunciation of the phoneme to be tested is unqualified.

[0078] Specifically, the preset GOP threshold may be set according to actual conditions. In this embodiment, the preset threshold is not specifically limited.

[0079] In one embodiment, when the GOP value corresponding to the phoneme to be measured is greater than or equal to a first threshold value, the evaluation result of the phoneme to be measured is determined to be excellent; when the GOP value corresponding to the phoneme to be measured is less than the first threshold value and greater than or equal to a second threshold value, the evaluation result of the phoneme to be measured is determined to be good; when the GOP value corresponding to the phoneme to be measured is less than the second threshold value and greater than or equal to a third threshold value, the evaluation result of the phoneme to be measured is determined to be medium; when the GOP value corresponding to the phoneme to be measured is less than the third threshold value, the evaluation result of the phoneme speech to be measured is determined to be poor.

[0080] Among them, the first threshold is greater than the second threshold and the second threshold is greater than the third threshold. The first threshold, the second threshold and the third threshold can be set according to actual conditions. In this embodiment, the threshold is not specifically limited.

[0081] It should be noted that the above embodiment only uses one threshold and three thresholds as examples for simple explanation. In specific applications, different thresholds can be set according to different situations to divide the GOP value into multiple levels, such as star-level classification. The specific classification method is not repeated in this embodiment.

[0082] In one embodiment, the GOP value can be directly converted into a fraction or percentage value, that is, the evaluation results are displayed in the form of fractions and percentages, which makes the evaluation results more detailed and enables learners to have a clear understanding of their pronunciation.

[0083] Specifically, the highest GOP value corresponds to 100 points, and the lowest GOP value corresponds to 0 points; then the GOP values between 0-100 are divided into 99 parts, corresponding to 1 point to 99 points respectively, and finally the correspondence between the GOP value and the score is stored.

[0084] Furthermore, after determining the GOP value of the phoneme to be measured, a query is performed in the correspondence between the GOP value and the score, and the score corresponding to the GOP value of the phoneme to be measured is used to determine the evaluation result of the phoneme to be measured. The specific query method is not repeated in this embodiment.

[0085] It should be noted that after determining the evaluation results of the phonemes to be tested, the evaluation results need to be sent to an output device for display so that learners can have a correct understanding of their pronunciation. The output device can be a display device built into the computer device, such as a touch screen display, or an external playback device connected to the computer device via a communication line, such as a projector or digital TV. It can also be an audio playback device built into the computer device, such as a built-in speaker, or an external playback device connected to the computer device via a communication line, such as headphones or external speakers.

[0086] Furthermore, if the evaluation result is unqualified, or after receiving the template voice playback instruction input by the learner, the template voice is played through the playback device or playback equipment to provide the learner with correct pronunciation for the learner to perform imitation training.

[0087] On the basis of the above embodiment, a method for determining the pronunciation of a phoneme to be tested is provided, wherein determining the pronunciation of the phoneme to be tested includes: within the duration of the phoneme, judging frame by frame whether the phoneme corresponding to the state maximum a posteriori probability of the acoustic model is the same as the phoneme to be tested; if the phoneme corresponding to the state maximum a posteriori probability of the acoustic model is the same as the phoneme to be tested, determining that the pronunciation of the phoneme to be tested is correct; if the phoneme corresponding to the state maximum a posteriori probability of the acoustic model is different from the phoneme to be tested, determining that the pronunciation of the phoneme to be tested is incorrect.

[0088] In this embodiment, the phoneme duration can be understood as the time corresponding to the pronunciation of a phoneme. Typically, the time corresponding to the pronunciation of a phoneme is greater than the duration of three frames. The acoustic model can be constructed using a DNN-HMM, that is, the acoustic model is a DNN-HMM acoustic model. The speech signal to be tested is input into the DNN-HMM acoustic model frame by frame, and the corresponding state posterior probability for each frame is output.

[0089] Posterior probability is a fundamental concept in information theory. After receiving a message, the probability that the receiving end knows that the message was sent is called the posterior probability. The calculation of the posterior probability is based on the prior probability. The posterior probability can be calculated using the Bayesian formula using the prior probability and the likelihood function. This embodiment does not limit the calculation method of the posterior probability.

[0090] Furthermore, in Bayesian statistics, "maximum a posteriori probability" is the maximum value of the posterior probability distribution. Using maximum a posteriori probability, we can obtain point estimates for quantities that are not directly observable in experimental data. It is closely related to the classic method of maximum likelihood estimation, but it uses an augmented optimization objective that further considers the prior probability distribution of the quantity being estimated. Therefore, maximum a posteriori probability estimation can be viewed as a regularized maximum likelihood estimation.

[0091] Among them, a phoneme is generally composed of three states. Taking the English word park as an example, the corresponding phoneme of the word park is p aa rk, and the corresponding three-phoneme sequences are sil-p+aa, p-aa+r, aa-r+k, and r-k+sil. The acoustic model is used to determine the phoneme corresponding to the maximum a posteriori probability of the state in the three-phoneme sequence p-aa+r. If the phoneme corresponding to the maximum a posteriori probability of each frame within the continuous frame length of the phoneme is not the phoneme aa, and the phoneme to be tested is aa, then it is determined that the pronunciation of the phoneme to be tested is incorrect. If the phoneme corresponding to the maximum a posteriori probability of the state within the continuous frame length of the phoneme exists phoneme aa, and the phoneme to be tested is aa, then it is determined that the pronunciation of the phoneme to be tested is correct.

[0092] Based on the above embodiment, a method for determining the pronunciation of a phoneme to be tested is provided, wherein determining the pronunciation of the phoneme to be tested includes: determining the state posterior probability corresponding to each phoneme state within the phoneme duration; determining the first phoneme corresponding to the maximum state posterior probability among all phoneme states; if the first phoneme is the same as the phoneme to be tested, determining that the phoneme to be tested is a correctly pronounced phoneme; if the first phoneme is different from the phoneme to be tested, determining that the phoneme in the audio to be tested is an incorrectly pronounced phoneme.

[0093] Among them, each phoneme state can be understood as each phoneme state in the three-phoneme sequence to which the phoneme to be tested belongs, and the cumulative state posterior probability can be understood as the sum of the state posterior probabilities corresponding to a certain phoneme in the three-phoneme sequence divided based on a word. For example: Taking the English word park as an example, the corresponding phoneme p aa rk of the word park corresponds to the three-phoneme sequence sil-p+aa, p-aa+r, aa-r+k, r-k+sil. Taking p-aa+r as an example, the state posterior probabilities of all three phones A-aa+B with phoneme aa as the central phoneme are calculated, namely A-aa+B0, A-aa+B1, and A-aa+B2, and the sum of all state posterior probabilities is taken as the cumulative state posterior probability of phoneme aa.

[0094] The posterior probability of each phoneme state can be determined by the above-mentioned acoustic model, or calculated by the Bayesian formula. The specific calculation method will not be described in detail in this embodiment.

[0095] Furthermore, after determining the accumulated state posterior probability corresponding to each phoneme state, the phoneme with the largest state posterior probability among the three phonemes is determined as the first phoneme. If the first phoneme is the same as the phoneme to be tested, it is determined that the phoneme to be tested is a correctly pronounced phoneme; if the first phoneme is different from the phoneme to be tested, it is determined that the phoneme in the audio to be tested is an incorrectly pronounced phoneme.

[0096] In one embodiment, a method for determining a GOP value is provided, wherein, for each pronunciation situation, a GOP calculation scheme corresponding to the pronunciation situation is used to determine the GOP value corresponding to the phoneme, including: when the pronunciation situation is correct, the GOP value corresponding to the phoneme to be tested is determined by the sum of the state posterior probabilities of all three phonemes centered on the phoneme to be tested; when the pronunciation situation is incorrect, the GOP value corresponding to the phoneme to be tested is determined by the state posterior probability value of a single three phonemes centered on the phoneme to be tested.

[0097] In this embodiment, when the phoneme to be tested is pronounced correctly, the sum of the state posterior probabilities of all three phonemes centered on the phoneme to be tested is the numerator, and the maximum value of the frame posterior probability within the duration of the phoneme to be tested is the denominator to calculate the GOP value of the phoneme.

[0098] Specifically, when the pronunciation is correct, the GOP value corresponding to the phoneme to be measured is calculated using the following first GOP formula;

[0099] Among them, the first GOP formula is:

[0100]

[0101] Among them, t e is the starting time of the phoneme to be tested, t sis the end time of the phoneme to be tested, p is the phoneme to be tested, s p is a triphone state with the phoneme to be tested being p, S is a triphone state set, s is a triphone state belonging to S, o t is the speech feature of the t-th frame, P(s p |o t )、P(s|o t ) is the posterior probability of the acoustic model output. Among them, max s∈S P(s|o t ) refers to the maximum state posterior probability of the frame corresponding to the phoneme, and its position in the DNN output state is as follows Figure 4 shown.

[0102] When the phoneme to be tested is pronounced incorrectly, the sum of the state posterior probability values of the single three phonemes centered on the phoneme to be tested is used as the numerator, and the maximum value of the frame posterior probability within the duration of the phoneme to be tested is used as the denominator to calculate the GOP value of the phoneme.

[0103] Specifically, when the pronunciation condition is a pronunciation error, the GOP value corresponding to the phoneme to be measured is calculated by the following second GOP formula;

[0104] Among them, the second GOP formula is:

[0105]

[0106] Among them, t e is the starting time of the phoneme to be tested, t s is the end time of the phoneme to be tested, p is the phoneme to be tested, S is a set of three phoneme states, s is a three phoneme state belonging to S, o t is the speech feature of the t-th frame, P(p|o t )、P(s|o t ) is the posterior probability of the acoustic model output. t ) refers to the state posterior probability after forced alignment corresponding to the phoneme to be tested, and its position in the DNN output state is as follows Figure 4 shown.

[0107] In this embodiment, different GOP calculation schemes are used for correctly pronounced phonemes and incorrectly pronounced phonemes to ensure that the GOP value will not be affected when the pronunciation is correct; and to ensure that accurate GOP scores can be obtained even when the pronunciation is incorrect, thereby reducing the impact of incorrect pronunciation on the speech evaluation results and improving the accuracy of the speech evaluation results.

[0108] Figure 5This is a schematic diagram of the structure of a speech evaluation device provided by an embodiment of the present invention. This device is suitable for detecting whether a learner's pronunciation is correct. The speech evaluation device can be implemented by hardware and / or software. The speech evaluation device can be composed of two or more physical entities or a single physical entity and is generally integrated into a computer device.

[0109] like Figure 5 As shown, the speech evaluation device provided by the embodiment of the present invention mainly includes a pronunciation condition determination module 51 and a GOP determination module 52.

[0110] The pronunciation status determination module 51 is used to determine the pronunciation status of the phoneme to be tested, wherein the pronunciation status includes correct pronunciation and incorrect pronunciation;

[0111] The GOP determination module 52 is configured to determine, for each pronunciation situation, a GOP value corresponding to the phoneme to be measured by using a pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation.

[0112] The speech evaluation device provided in an embodiment of the present invention performs the following operations: determining the pronunciation of a phoneme to be tested, where pronunciation conditions include correct pronunciation and incorrect pronunciation; and for each pronunciation condition, determining the corresponding GOP value of the phoneme to be tested using a corresponding pronunciation goodness of pronunciation (GOP) calculation scheme. The technical solution of this embodiment distinguishes between different pronunciation conditions, using different GOP calculation schemes for each condition, ensuring that the GOP value is not affected when pronunciation is correct; and ensuring that an accurate GOP score is also generated when pronunciation is incorrect, thereby reducing the impact of incorrect pronunciation on the speech evaluation results and improving the accuracy of the speech evaluation results.

[0113] In one embodiment, the pronunciation condition determination module 51 includes:

[0114] A first judging unit is configured to judge, frame by frame within the duration of a phoneme, whether the phoneme corresponding to the maximum a posteriori probability of the state of the acoustic model is the same as the phoneme to be tested;

[0115] a pronunciation correctness determination unit, configured to determine that the pronunciation of the phoneme to be tested is correct if the phoneme corresponding to the maximum a posteriori probability of the state of the acoustic model is the same as the phoneme to be tested;

[0116] The pronunciation error determination unit is used to determine that the phoneme to be tested is mispronounced if the phoneme corresponding to the maximum a posteriori probability of the state of the acoustic model is different from the phoneme to be tested.

[0117] In one embodiment, the pronunciation condition determination module 51 includes:

[0118] An accumulated state posterior probability determination unit, configured to determine the accumulated state posterior probability corresponding to each phoneme state within the phoneme duration;

[0119] A first phoneme determination unit, configured to determine a first phoneme corresponding to a maximum accumulated state posterior probability among all phoneme states;

[0120] The correct pronunciation determination unit is further configured to determine that the phoneme to be tested is a correctly pronounced phoneme if the first phoneme is identical to the phoneme to be tested;

[0121] The mispronunciation determination unit is further configured to determine that the phoneme in the audio to be tested is a mispronounced phoneme if the first phoneme is different from the phoneme to be tested.

[0122] In one embodiment, the GOP determination module 52 includes:

[0123] a first GOP determination unit, configured to determine, when the pronunciation is correct, a GOP value corresponding to the phoneme to be measured based on the sum of state posterior probabilities of all three phonemes centered on the phoneme to be measured;

[0124] The second GOP determining unit is configured to determine, when the pronunciation condition is a pronunciation error, a GOP value corresponding to the phoneme to be tested based on a state posterior probability value of a single triphone centered on the phoneme to be tested.

[0125] In one embodiment, the first GOP determining unit is specifically configured to calculate the GOP value corresponding to the phoneme to be measured by using the following first GOP formula when the pronunciation condition is correct;

[0126] Among them, the first GOP formula is:

[0127]

[0128] Among them, t e is the starting time of the phoneme to be tested, t s is the end time of the phoneme to be tested, p is the phoneme to be tested, s p is a triphone state with the phoneme to be tested being p, S is a triphone state set, s is a triphone state belonging to S, o t is the speech feature of the t-th frame, P(s p |o t )、P(s|o t ) is the posterior probability output by the acoustic model.

[0129] In one embodiment, the second GOP determining unit is specifically configured to calculate the GOP value corresponding to the phoneme to be measured by using the following second GOP formula when the pronunciation condition is a pronunciation error;

[0130] Among them, the second GOP formula is:

[0131]

[0132] Among them, t e is the starting time of the phoneme to be tested, t s is the end time of the phoneme to be tested, p is the phoneme to be tested, S is a set of three phoneme states, s is a three phoneme state belonging to S, o t is the speech feature of the t-th frame, P(p|o t )、P(s|o t ) is the posterior probability output by the acoustic model.

[0133] In one embodiment, the apparatus further comprises:

[0134] The pronunciation qualified determination module is used to determine that the pronunciation of the phoneme to be tested is qualified when the GOP value corresponding to the phoneme to be tested is greater than or equal to the preset GOP threshold; and to determine that the pronunciation of the phoneme to be tested is unqualified when the GOP value corresponding to the phoneme to be tested is less than the preset GOP threshold.

[0135] Figure 6 A schematic diagram of the hardware structure of a device provided in an embodiment of the present invention is shown in FIG. Figure 6 As shown, the device includes a processor 601, a memory 602, an input device 603 and an output device 604; the number of processors 601 in the device can be one or more. Figure 6 In the embodiment, a processor 601 is used as an example; the processor 601, the memory 602, the input device 603 and the output device 604 in the device can be connected by a bus or other means. Figure 6 The bus connection is taken as an example.

[0136] The memory 602 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the speech evaluation method in the embodiment of the present invention (for example, the attached Figure 5 The modules in the speech evaluation device shown include: a pronunciation determination module 51 and a GOP determination module 52. The processor 601 executes the software programs, instructions, and modules stored in the memory 602 to perform various functional applications and data processing of the device, thereby implementing the above-mentioned speech evaluation method.

[0137] The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the device, etc. In addition, the memory 602 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 602 may further include a memory remotely located relative to the processor 601, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0138] Furthermore, when one or more programs included in the above device are executed by the one or more processors 601, the program performs the following operations:

[0139] Determining the pronunciation of the phoneme to be tested, wherein the pronunciation includes correct pronunciation and incorrect pronunciation;

[0140] For each pronunciation situation, the GOP value corresponding to the phoneme to be measured is determined using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation.

[0141] The input device 603 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 604 may include a display device such as a display screen.

[0142] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processing device, the speech evaluation method provided by an embodiment of the present invention is implemented. The method includes:

[0143] Determining the pronunciation of the phoneme to be tested, wherein the pronunciation includes correct pronunciation and incorrect pronunciation;

[0144] For each pronunciation situation, the GOP value corresponding to the phoneme to be measured is determined using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation.

[0145] Of course, the computer-executable instructions provided in the storage medium of the embodiment of the present invention are not limited to the operations of the method described above, but can also perform the related operations of the speech evaluation method provided in any embodiment of the present invention.

[0146] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0147] It is worth noting that in the embodiment of the above-mentioned speech evaluation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0148] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A speech evaluation method, characterized in that: include: Determining the pronunciation of the phoneme to be tested, wherein the pronunciation includes correct pronunciation and incorrect pronunciation; For each pronunciation situation, the GOP value corresponding to the phoneme to be tested is determined using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation; The method of determining the GOP value corresponding to the phoneme by using the GOP calculation scheme corresponding to the pronunciation situation for each pronunciation situation includes: When the pronunciation is correct, the GOP value corresponding to the phoneme to be tested is determined by summing the state posterior probabilities of all three phonemes centered on the phoneme to be tested; When the pronunciation condition is a pronunciation error, the state posterior probability value of a single triphone centered on the phoneme to be tested is used to determine the GOP value corresponding to the phoneme to be tested.

2. The method according to claim 1, characterized in that Determine the pronunciation of the phoneme to be tested, including: During the phoneme duration, determining frame by frame whether the phoneme corresponding to the maximum posterior probability of the state of the acoustic model is the same as the phoneme to be tested; If the phoneme corresponding to the maximum a posteriori probability of the state of the acoustic model is the same as the phoneme to be tested, it is determined that the pronunciation of the phoneme to be tested is correct; If the phoneme corresponding to the maximum a posteriori probability of the state of the acoustic model is different from the phoneme to be tested, it is determined that the phoneme to be tested is mispronounced.

3. The method according to claim 1, characterized in that Determine the pronunciation of the phoneme to be tested, including: Determine the cumulative state posterior probability corresponding to each phoneme state within the phoneme duration; Determine the first phoneme corresponding to the maximum cumulative state posterior probability among all phoneme states; If the first phoneme is identical to the phoneme to be tested, determining that the phoneme to be tested is a correctly pronounced phoneme; If the first phoneme is different from the phoneme to be detected, it is determined that the phoneme to be detected is a mispronounced phoneme.

4. The method according to claim 1, wherein For each pronunciation situation, a GOP calculation scheme corresponding to the pronunciation situation is used to determine a GOP value corresponding to the phoneme, including: When the pronunciation is correct, the GOP value corresponding to the phoneme to be measured is calculated using the following first GOP formula; Among them, the first GOP formula is: Among them, t e is the starting time of the phoneme to be tested, t s is the end time of the phoneme to be tested, p is the phoneme to be tested, s p is a triphone state with the phoneme to be tested being p, S is a triphone state set, s is a triphone state belonging to S, o t is the speech feature of the t-th frame, P(s p |o t )、P(s|o t ) is the posterior probability output by the acoustic model.

5. The method according to claim 1, wherein For each pronunciation situation, a pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation is used to determine the GOP value corresponding to the phoneme, including: When the pronunciation condition is a pronunciation error, the GOP value corresponding to the phoneme to be measured is calculated by the following second GOP formula; Among them, the second GOP formula is: Among them, t e is the starting time of the phoneme to be tested, t s is the end time of the phoneme to be tested, p is the phoneme to be tested, S is a set of three phoneme states, s is a three phoneme state belonging to S, o t is the speech feature of the t-th frame, P(p|o t )、P(s|o t ) is the posterior probability output by the acoustic model.

6. The method according to claim 1, characterized in that After determining the GOP value corresponding to the phoneme to be measured by using the pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation, the method further includes: When the GOP value corresponding to the phoneme to be tested is greater than or equal to a preset GOP threshold, determining that the pronunciation of the phoneme to be tested is qualified; When the GOP value corresponding to the phoneme to be measured is less than a preset GOP threshold, it is determined that the pronunciation of the phoneme to be measured is unqualified.

7. A speech evaluation device, characterized in that: include: A pronunciation status determination module, configured to determine the pronunciation status of the phoneme to be tested, wherein the pronunciation status includes correct pronunciation and incorrect pronunciation; A GOP determination module is configured to determine, for each pronunciation situation, a GOP value corresponding to the phoneme to be tested using a pronunciation goodness GOP calculation scheme corresponding to the pronunciation situation; Wherein, the GOP determination module includes: a first GOP determination unit and a second GOP determination unit; The first GOP determining unit is configured to determine a GOP value corresponding to the phoneme to be measured based on the sum of state posterior probabilities of all three phonemes centered on the phoneme to be measured when the pronunciation condition is correct; The second GOP determining unit is configured to determine, when the pronunciation condition is a pronunciation error, a GOP value corresponding to the phoneme to be tested based on a state posterior probability value of a single triphone centered on the phoneme to be tested.

8. A voice evaluation device, characterized in that: include: one or more processors; a memory for storing one or more programs; The one or more programs are executed by the one or more processors, so that the one or more processors implement the speech evaluation method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the speech evaluation method according to any one of claims 1 to 6 is implemented.

Images