Voice evaluation method, device, equipment and storage medium

Abstract

Embodiments of the present invention disclose a speech evaluation method, apparatus, device, and storage medium. The speech evaluation method includes determining a first pronunciation-goodness GOP sequence corresponding to the test speech and a second GOP sequence corresponding to a template speech, determining the KL divergence of the relative entropy between the first and second GOP sequences, and determining an evaluation result for the test speech based on the KL divergence. The technical solution of this embodiment utilizes the fact that GOP features are probability distributions. The KL divergence is used to measure the similarity between the template pronunciation GOP sequences, thereby improving the accuracy of the evaluation results.

Description

Technical Field

[0001] The present invention relates to a data processing technology, and more particularly to a speech evaluation method, apparatus, device, and storage medium. Background Art

[0002] Pronunciation Quality Assessment (PQA) is a sub-discipline of computer-assisted language learning (CALL). It aims to efficiently and accurately identify learners' pronunciation errors, provide objective evaluations at the phoneme level, and help learners correct them.

[0003] Goodness of Pronunciation (GOP) is a commonly used feature in pronunciation evaluation, but it can only obtain pronunciation evaluation results at the phoneme level. For direct evaluation of entire sentences, existing evaluation methods mainly use weighted phoneme-level GOP as sentence-level GOP, thereby indirectly obtaining sentence-level pronunciation evaluation results.

[0004] The above-mentioned whole sentence evaluation method does not address the impact of differences in the distribution of GOP values of different phonemes, which will cause inaccurate whole sentence speech evaluation. Summary of the Invention

[0005] The embodiments of the present invention provide a speech evaluation method, apparatus, device and storage medium to improve the accuracy of whole-sentence speech evaluation results.

[0006] In a first aspect, an embodiment of the present invention provides a speech evaluation method, comprising:

[0007] Determine a first pronunciation goodness GOP sequence corresponding to the test speech and a second GOP sequence corresponding to the template speech;

[0008] determining a relative entropy KL divergence between the first GOP sequence and the second GOP sequence;

[0009] An evaluation result of the speech to be tested is determined based on the KL divergence.

[0010] In one embodiment, determining a first GOP sequence corresponding to the speech to be tested and a second GOP sequence corresponding to the template speech includes:

[0011] Acquire a first phoneme state sequence and first boundary information of the speech to be tested;

[0012] Determine a first GOP sequence corresponding to the speech to be tested based on the first phoneme state sequence and the first boundary information;

[0013] Acquire a second phoneme state sequence and second boundary information of the template speech;

[0014] A second GOP sequence corresponding to the template speech is determined based on the second phoneme state sequence and the second boundary information.

[0015] In one embodiment, determining a first GOP sequence corresponding to the speech to be tested based on the first phoneme state sequence and the first boundary information, and determining a second GOP sequence corresponding to the template speech based on the second phoneme state sequence and the second boundary information includes:

[0016] determining, based on the first phoneme state sequence and the first boundary information, a first GOP value corresponding to each non-silent phoneme in the first phoneme state sequence;

[0017] Arranging first GOP values corresponding to a plurality of non-silent phonemes in a first preset order to obtain a first GOP sequence;

[0018] determining, based on the second phoneme state sequence and the second boundary information, a second GOP value corresponding to each non-silent phoneme in the second phoneme state sequence;

[0019] The second GOP values corresponding to the plurality of non-silent phonemes are arranged in a second preset order to obtain a second GOP sequence.

[0020] In one embodiment, after determining the second GOP sequence corresponding to the template speech based on the second phoneme state sequence and the second boundary information, the method further includes:

[0021] For each phoneme state, comparing a GOP value in the first GOP sequence with a GOP value in the second GOP sequence;

[0022] If the GOP value in the second GOP sequence is smaller than the GOP value in the first GOP sequence, the GOP value in the second GOP sequence is replaced with the GOP value in the first GOP sequence to obtain a new second GOP sequence.

[0023] In one embodiment, determining the relative entropy KL divergence between the first GOP sequence and the second GOP sequence includes:

[0024] The KL divergence between the first GOP sequence and the second GOP sequence is calculated using a KL divergence formula.

[0025] In one embodiment, the KL divergence formula is:

[0026]

[0027] Among them, D KL(P||Q) is the KL divergence between the first GOP sequence and the second GOP sequence, P(i) is the i-th GOP value in the second GOP sequence, and Q(i) is the i-th GOP value in the first GOP sequence.

[0028] In one embodiment, the KL divergence determines the evaluation result of the speech to be tested, including:

[0029] When the KL divergence is greater than or equal to a preset threshold, determining that the evaluation result of the speech to be tested is correct pronunciation;

[0030] When the KL divergence is less than the preset threshold, it is determined that the evaluation result of the speech to be tested is a pronunciation error.

[0031] In a second aspect, an embodiment of the present invention further provides a speech evaluation device, characterized in that it includes:

[0032] A GOP sequence determination module is used to determine a first pronunciation goodness GOP sequence corresponding to the speech to be tested and a second GOP sequence corresponding to the template speech;

[0033] A KL divergence determination module, configured to determine a relative entropy KL divergence between the first GOP sequence and the second GOP sequence;

[0034] An evaluation result determination module is used to determine the evaluation result of the speech to be tested based on the KL divergence.

[0035] In a third aspect, an embodiment of the present invention further provides a speech evaluation device, comprising:

[0036] one or more processors;

[0037] a memory for storing one or more programs;

[0038] The one or more programs are executed by the one or more processors, so that the one or more processors implement the speech evaluation method provided in the first aspect above.

[0039] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having one or more computer programs stored thereon, which, when executed by a processor, implements the speech evaluation method provided in the first aspect above.

[0040] The speech evaluation method, apparatus, device, and storage medium provided in the above-mentioned embodiments include determining a first pronunciation-goodness GOP sequence corresponding to the test speech and a second GOP sequence corresponding to the template speech, determining the KL divergence of relative entropy between the first GOP sequence and the second GOP sequence, and determining an evaluation result for the test speech based on the KL divergence. The technical solution of this embodiment utilizes the fact that GOP features are probability distributions. The KL divergence is used to measure the similarity between the template pronunciation GOP sequence and the template pronunciation GOP sequence, thereby improving the accuracy of the evaluation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] FIG1a is an example diagram of an application scenario provided by an embodiment of the present invention;

[0042] FIG1b is another example diagram of an application scenario provided by an embodiment of the present invention

[0043] Figure 2 is a flow chart of a speech evaluation method provided by an embodiment of the present invention;

[0044] Figure 3 is a flow chart of another voice evaluation method provided by an embodiment of the present invention;

[0045] Figure 4 This is a flow chart of another voice evaluation method provided by an embodiment of the present invention;

[0046] Figure 5 A schematic diagram of the structure of a speech evaluation device provided in an embodiment of the present invention;

[0047] Figure 6 A schematic diagram of the hardware structure of a device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all structures.

[0049] Existing speech measurement technology research typically only produces phoneme-level pronunciation evaluation results. For direct sentence evaluation, existing methods typically use weighted phoneme-level GOP values as sentence-level GOP values, indirectly obtaining sentence-level pronunciation evaluation results. These whole-sentence evaluation methods fail to address the impact of differences in the distribution of GOP values for different phonemes, resulting in inaccurate whole-sentence speech evaluation.

[0050] Therefore, to address the above issues, embodiments of the present invention provide a speech evaluation method, apparatus, device, and storage medium. These methods calculate the KL divergence between the GOP sequence of phonemes in a template pronunciation and the GOP sequence of phonemes in the speech to be tested, thereby obtaining sentence-level pronunciation evaluation results. The KL divergence is then used to classify the pronunciation into multiple levels, enabling sentence-level pronunciation evaluation.

[0051] This solution can be used in areas including, but not limited to, pronunciation error detection and diagnosis in speech assessment. For example, online or offline speech assessment systems can provide language learners with pronunciation error detection for entire sentences, enabling sentence-level pronunciation assessment. For example, native Chinese speakers learning a foreign language can use this solution.

[0052] FIG1a is an example diagram of an application scenario provided by an embodiment of the present invention. As shown in FIG1a, server 102 is used to execute the pronunciation evaluation method described in any method embodiment of the present application. Client 101 receives a speech to be tested issued by a user via an input device. Server 102 interacts with client 101 to obtain the speech to be tested. After executing the above-mentioned pronunciation evaluation method, server 102 outputs the pronunciation evaluation result to client 101, which is notified to the learner via the output device of client 101. Furthermore, client 101 provides the correct pronunciation to the learner to help them correct their pronunciation.

[0053] The input device may be an input device built into the computer device, such as a built-in voice input device, or an external input device connected to the computer device via a communication line, such as a microphone. Furthermore, the output device may be an output device built into the computer device, such as a touch screen display, or an external output device connected to the computer device via a communication line, such as a projector or digital TV.

[0054] It should be noted that in this embodiment, the client 101 is described using a computer as an example. The computer device may specifically be a computer device including a processor, a memory, an input device, and an output device, such as a laptop computer, a desktop computer, a tablet computer, a smart terminal, a learning machine, an early childhood education machine, a smart wearable device, etc.

[0055] Alternatively, when the client 101 has a certain data processing capability, that is, when the client 101 has a processor and a memory, the client 101 can independently serve as the execution subject of the pronunciation evaluation method described in any embodiment of the present application, such as Figure 1b As an example. Figure 1b In the process, the learner presses the microphone, and the built-in voice collection device of the mobile phone can collect the test voice issued by the user in real time, and after executing the above pronunciation evaluation method, the evaluation result of the test voice is displayed to the learner through the display screen.

[0056] The pronunciation evaluation method provided by the present invention is explained below with reference to specific embodiments.

[0057] Figure 2 This is a flow chart of a speech assessment method provided by an embodiment of the present invention. This method is suitable for detecting whether a learner's pronunciation is correct. The speech assessment method can be performed by a speech assessment device, which can be implemented in hardware and / or software. The speech assessment device can be composed of two or more physical entities, or a single physical entity, and is generally integrated into a computer device.

[0058] like Figure 2 As shown, the speech evaluation method provided by the embodiment of the present invention mainly includes the following steps:

[0059] S11 , determining a first GOP sequence of good pronunciation corresponding to the speech to be tested and a second GOP sequence corresponding to the template speech.

[0060] In this embodiment, the speech to be tested can be understood as the learner's speech collected in response to the pronunciation text. Furthermore, it can be the learner's voice collected directly through a microphone, or it can be the learner's voice recorded in advance. The learner refers to someone who needs to train their pronunciation based on the pronunciation text. The template speech refers to the standard pronunciation of the speech that the learner needs to train. It is recorded according to the standard pronunciation of various languages. For example, Chinese is recorded according to the national standard pronunciation of Mandarin, and English can be recorded according to standard American pronunciation or British pronunciation standards. The above template pronunciation can also be understood as the correct pronunciation, which is used to help learners correct incorrect pronunciation and prompt correct pronunciation.

[0061] It should be noted that the above-mentioned test speech and template speech are speech for the same pronunciation text.

[0062] In actual applications, when a learner reads a text, a speech signal corresponding to the text is generated. The electronic device first obtains the speech signal and detects the speech signal to determine whether the learner's pronunciation is correct.

[0063] For example, the text can be specifically at least one word, or even at least one phoneme. Among them, the phoneme is the smallest speech unit divided according to the natural properties of the speech, and is analyzed based on the pronunciation action in the syllable, and one action constitutes a phoneme. A phoneme is composed of multiple states. For example, a phoneme is composed of three states, and each of these three states is assigned at least one frame of duration. The corresponding duration of the phoneme being read out must be greater than the duration of three frames. The above text is the pronunciation text described in the embodiment of the present application, and the speech signal is the speech to be detected.

[0064] Taking the learning machine as an example, when the learner reads the pronunciation text on the display interface of the learning machine, the learning machine collects voice signals through microphones and other sound pickup devices to obtain the voice to be tested.

[0065] In this embodiment, the first GOP sequence refers to analyzing the speech to be tested to obtain a series of GOP values, and the second GOP sequence refers to analyzing the template speech to obtain a series of GOP values.

[0066] Furthermore, the order of the GOP values in the first GOP sequence is determined by the first phoneme state sequence of the speech to be tested. The order of the GOP values in the second GOP sequence is determined by the second phoneme state sequence of the template speech. The length of the first GOP sequence is equal to the length of the second GOP sequence.

[0067] In one embodiment, a method for determining a first GOP sequence is provided. Specifically, based on a pronunciation text and a test speech, the test speech is decomposed to obtain phonemes and boundary information contained in the test speech. The states corresponding to these phonemes form a first phoneme state sequence. The GOP value corresponding to each phoneme is calculated, and the GOP values corresponding to the phonemes are arranged sequentially according to the order of the phonemes in the first phoneme state sequence to obtain the first GOP sequence.

[0068] In one embodiment, a method for determining a second GOP sequence is provided. Specifically, based on pronunciation text and standard speech, the standard speech is decomposed to obtain phonemes and boundary information contained in the standard speech. The states corresponding to these phonemes form a second phoneme state sequence. The GOP value corresponding to each phoneme is calculated, and the GOP values corresponding to the phonemes are arranged in order according to the order of the phonemes in the second phoneme state sequence to obtain the second GOP sequence.

[0069] It should be noted that, for the first GOP sequence corresponding to the speech to be tested, since the speech uttered by the learner for the same pronunciation text may be different each time, it is necessary to execute the solution for determining the first GOP sequence each time the speech signal to be tested is collected.

[0070] In one embodiment, for the second GOP sequence corresponding to the template speech, a scheme of obtaining the template speech corresponding to the test speech and determining the second GOP sequence corresponding to the template speech may be executed each time the test speech signal is collected.

[0071] In two embodiments, for the second GOP sequence corresponding to the template speech, since the standard pronunciation of a pronunciation text is the same, after determining the second GOP sequence corresponding to the template speech, the correspondence between the second GOP sequence and the pronunciation text is stored. After obtaining the speech to be tested corresponding to the pronunciation text and the first GOP sequence corresponding to the speech to be tested, the second GOP sequence corresponding to the pronunciation text is directly obtained in the memory or storage unit, and then the second GOP sequence corresponding to the template speech is determined.

[0072] S12: Determine a relative entropy KL divergence between the first GOP sequence and the second GOP sequence.

[0073] Relative entropy, also known as the Kullback-Leibler divergence or information divergence, is a measure of the asymmetry between two probability distributions. The KL divergence is used to measure the degree of difference between two probability distributions, P and Q, where P represents the true or observed distribution of the data and Q represents the theoretical, estimated, or approximate model distribution of the data. The KL divergence, also known as the KL distance, is non-negative.

[0074] Specifically, KL divergence is used to measure the degree of difference between the first GOP sequence and the second GOP sequence. In this embodiment, the specific method for calculating KL divergence is not limited. The GOP values in the first GOP sequence and the second GOP sequence can be directly substituted into the KL divergence formula to obtain the KL divergence; alternatively, the KL divergence of the first GOP sequence and the second GOP sequence can be obtained using a mathematical derivation method. This embodiment only illustrates the calculation method of KL divergence, without limiting it. In this embodiment, only the KL divergence method of measuring the degree of difference between two sequences is described, without limiting it. Other methods such as Euclidean distance can also be used.

[0075] In one embodiment, determining the relative entropy KL divergence between the first GOP sequence and the second GOP sequence includes: calculating the KL divergence between the first GOP sequence and the second GOP sequence using a KL divergence formula.

[0076] In this embodiment, the KL divergence formula may be a KL conclusion formula or a KL derivation formula, which is not limited in this embodiment.

[0077] S13. Determine an evaluation result of the speech to be tested based on the KL divergence.

[0078] In this embodiment, the evaluation result may be correct pronunciation or incorrect pronunciation, or may be divided into multiple levels such as excellent, good, medium, and poor, or may be a score or percentage value, which is not limited in this embodiment.

[0079] In one embodiment, the KL divergence determines the evaluation result of the speech to be tested, including: when the KL divergence is greater than or equal to a preset threshold, determining that the evaluation result of the speech to be tested is correct pronunciation; when the KL divergence is less than the preset threshold, determining that the evaluation result of the speech to be tested is incorrect pronunciation.

[0080] Specifically, the preset threshold value can be set according to actual conditions. In this embodiment, the preset threshold value is not specifically limited.

[0081] In one embodiment, the KL divergence determines the evaluation result of the speech to be tested, including: when the KL divergence is greater than or equal to a first threshold, determining that the evaluation result of the speech to be tested is excellent; when the KL divergence is less than the first threshold and greater than or equal to a second threshold, determining that the evaluation result of the speech to be tested is good; when the KL divergence is less than the second threshold and greater than or equal to a third threshold, determining that the evaluation result of the speech to be tested is medium; when the KL divergence is less than the third threshold, determining that the evaluation result of the speech to be tested is poor.

[0082] Among them, the first threshold is greater than the second threshold and the second threshold is greater than the third threshold. The first threshold, the second threshold and the third threshold can be set according to actual conditions. In this embodiment, the threshold is not specifically limited.

[0083] It should be noted that the above embodiment only uses 1 threshold and 3 thresholds as examples for simple explanation. In specific applications, different thresholds can be set according to different situations to divide the evaluation results into multiple levels. The specific level division method will not be repeated in this embodiment.

[0084] In one embodiment, the KL divergence can be directly converted into a fraction or percentage value, that is, the evaluation results are displayed in the form of fractions and percentages, which makes the evaluation results more detailed and enables learners to have a clear understanding of their pronunciation.

[0085] Specifically, when the phonemes of the test speech and the standard speech are exactly the same, the KL divergence corresponds to 100 points, and when the phonemes of the test speech and the standard speech are completely different, the KL divergence corresponds to 0 points; then the KL divergence between 0-100 is divided into 99 parts, corresponding to 1 point to 99 points respectively, and finally the correspondence between the KL divergence and the score is stored.

[0086] Furthermore, after determining the KL divergence of the speech to be tested, a query is performed in the correspondence between KL divergence and scores, and the score corresponding to the KL divergence of the speech to be tested is used to determine the evaluation result of the speech to be tested. The specific query method is not further described in this embodiment.

[0087] It should be noted that after determining the evaluation results of the speech to be tested, the evaluation results need to be sent to an output device for display so that the learner can have a correct understanding of their pronunciation. The output device can be a display device built into the computer device, such as a touch screen display; it can also be an external playback device connected to the computer device via a communication line, such as a projector, digital TV, etc. It can also be an audio playback device built into the computer device, such as a built-in speaker; it can also be an external playback device connected to the computer device via a communication line, such as headphones, external speakers, etc.

[0088] Furthermore, if the evaluation result is unqualified, or after receiving the template voice playback instruction input by the learner, the template voice is played through the playback device or playback equipment to provide the learner with correct pronunciation for the learner to perform imitation training.

[0089] The speech evaluation method provided by an embodiment of the present invention includes determining a first pronunciation-goodness GOP sequence corresponding to the test speech and a second GOP sequence corresponding to the template speech, determining the KL divergence of the relative entropy between the first and second GOP sequences, and determining an evaluation result for the test speech based on the KL divergence. This embodiment utilizes the fact that GOP features are probability distributions and measures the similarity between the template GOP sequences using the KL divergence, thereby improving the accuracy of the evaluation results.

[0090] Figure 3 Another speech evaluation method provided by the embodiment of the present invention is Figure 3 As shown, the speech evaluation method provided by the embodiment of the present invention mainly includes the following steps:

[0091] S21. Obtain a first phoneme state sequence and first boundary information of the speech to be tested.

[0092] In this embodiment, the pronunciation text and the test speech are aligned. Furthermore, a pre-trained acoustic model is used to calculate the acoustic score for each frame of the test speech. The Viterbi algorithm is then used to search for the optimal path in the alignment network to obtain the first phoneme state sequence and first boundary information of the test speech.

[0093] The Viterbi algorithm is a widely used dynamic programming algorithm in machine learning. It is used to find the most likely hidden state sequence—the Viterbi path—that produces the observed event sequence, particularly in the context of Markov information sources and hidden Markov models. The terms "Viterbi path" and "Viterbi algorithm" are also used to describe dynamic programming algorithms that find the most likely explanation for observed results. In the embodiments of the present invention, the Viterbi algorithm is used to search for the optimal path in the alignment network to obtain the first phoneme state sequence.

[0094] The acoustic model can be constructed using a Deep Neural Network (DNN)-Hidden Markov Model (HMM) approach, effectively a DNN-HMM acoustic model. The speech signal to be detected is input frame by frame into the DNN-HMM acoustic model, which outputs the corresponding state posterior probabilities for each frame. These are converted into acoustic scores, and the Viterbi algorithm is used to search for the optimal path to obtain the first phoneme state sequence and boundary information. The Viterbi algorithm's path search objective is to find an optimal path within the WFST alignment network that matches the speech feature sequence.

[0095] S22: Determine a first GOP sequence corresponding to the speech to be tested based on the first phoneme state sequence and the first boundary information.

[0096] In this embodiment, the alignment results reveal the first phoneme state sequence and the first boundary. A GOP score is calculated using the phoneme length as the unit based on the GOP calculation formula. The numerator in the GOP calculation formula is the phoneme sequence likelihood value obtained by forced alignment, and the denominator is the sequence likelihood value obtained by free decoding of the phonemes. Free decoding refers to a decoding process based on a recurrent phoneme network.

[0097] The calculation formula of GOP is:

[0098]

[0099] Where T is the duration of a phoneme, O represents the speech feature sequence corresponding to the speech signal within the duration of phoneme p, Q is the set of phonemes, P(o|p) is the observed probability of phoneme p, and P(p) is the prior probability of phoneme p. Only the GOP values of non-silent phonemes are calculated to obtain the GOP score for each non-silent phoneme in the phoneme sequence.

[0100] In one embodiment, based on the first phoneme state sequence and the first boundary information, the first GOP value corresponding to each non-silent phoneme in the first phoneme state sequence is determined; the first GOP values corresponding to multiple non-silent phonemes are arranged in a first preset order to obtain a first GOP sequence.

[0101] The first preset order refers to the order of each phoneme in the speech to be tested in the first phoneme state sequence.

[0102] S23. Obtain a second phoneme state sequence and second boundary information of the template speech.

[0103] S24: Determine a second GOP sequence corresponding to the template speech based on the second phoneme state sequence and the second boundary information.

[0104] In this embodiment, the second phoneme state sequence and the second boundary information of the template speech are obtained, and the second GOP sequence corresponding to the template speech is determined based on the second phoneme state sequence and the second boundary information. The specific method is the same as the method of the above steps S21 and S24. For details, please refer to the description in the above embodiment, which will not be repeated in this embodiment.

[0105] In one embodiment, based on the second phoneme state sequence and the second boundary information, the second GOP value corresponding to each non-silent phoneme in the second phoneme state sequence is determined; the second GOP values corresponding to multiple non-silent phonemes are arranged in a second preset order to obtain a second GOP sequence.

[0106] The second preset order refers to the order of each phoneme in the template speech in the second phoneme state sequence.

[0107] S25 . For each phoneme state, compare the GOP value in the first GOP sequence with the GOP value in the second GOP sequence.

[0108] S26. If the GOP value in the second GOP sequence is smaller than the GOP value in the first GOP sequence, the GOP value in the second GOP sequence is replaced with the GOP value in the first GOP sequence to obtain a new second GOP sequence.

[0109] In this embodiment, the second GOP sequence corresponding to the template speech and the first GOP sequence corresponding to the test speech have already been obtained in S21-S24. Considering that the acoustic model output probability may be inaccurate, the GOP value of the phoneme in the second GOP sequence is compared phoneme by phoneme to see if it is less than the GOP value of the phoneme in the first GOP sequence. If so, the GOP value of the phoneme in the second GOP sequence is temporarily replaced with the GOP value of the corresponding phoneme in the first GOP sequence, while the GOP value of the phoneme in the first GOP sequence remains unchanged. If not, no operation is performed, thus ensuring the stability and reliability of the scoring.

[0110] Exemplarily, the second GOP sequence corresponding to the template speech includes, in sequence: GOP value a1 corresponding to phoneme A, GOP value b1 corresponding to phoneme B, and GOP value c1 corresponding to phoneme C. The first GOP sequence corresponding to the test speech includes, in sequence: GOP value a2 corresponding to phoneme A, GOP value b2 corresponding to phoneme B, and GOP value c2 corresponding to phoneme C. Phoneme-by-phoneme comparison refers to comparing GOP value a1 corresponding to phoneme A with GOP value a2 corresponding to phoneme A, comparing GOP value b1 corresponding to phoneme B with GOP value b2 corresponding to phoneme B, and comparing GOP value c1 corresponding to phoneme C with GOP value c2 corresponding to phoneme C. If a1 is less than a2, a1 in the second GOP sequence is replaced with a2. If a1 is greater than or equal to a2, no operation is performed on the GOP values in the second GOP sequence.

[0111] Furthermore, the second GOP sequence is a1, b1, and c1, and the first GOP sequence is a2, b2, and c2. If a1 is less than a2, b1 is not less than b2, and c1 is not less than c2, then the second GOP sequence after comparison is a2, b1, and c1, and the first GOP sequence is a2, b2, and c2.

[0112] S27. Calculate the KL divergence between the first GOP sequence and the second GOP sequence using a KL divergence formula.

[0113] In this embodiment, the KL divergence of the two updated GOP sequences of equal length is calculated. KL divergence, also known as relative entropy, is a method used to measure the degree of difference between two probability distributions P and Q, where P represents the actual or observed distribution of the data, and Q represents the theoretical, estimated, or approximate model distribution of the data.

[0114] Among them, the KL divergence formula is:

[0115]

[0116] Among them, D KL (P||Q) is the KL divergence between the first GOP sequence and the second GOP sequence, P(i) is the i-th GOP value in the first GOP sequence, and Q(i) is the i-th GOP value in the second GOP sequence.

[0117] S28. Determine an evaluation result of the speech to be tested based on the KL divergence.

[0118] This embodiment utilizes a sentence speech evaluation method based on KL divergence. Because the GOP distributions of different phonemes vary significantly, the impact of these differences in GOP values can be effectively mitigated by considering the degree of similarity between the test sample and the template, rather than simply taking a weighted average of the phoneme evaluation results. This method leverages the fact that GOP features are probabilistically distributed, and measures the similarity with the template pronunciation GOP sequence using KL divergence, ensuring the accuracy of the evaluation results.

[0119] In an applied example, Figure 4 This is a flow chart of another voice evaluation method provided by an embodiment of the present invention. Figure 4 As shown, the voice evaluation method provided in this embodiment mainly includes:

[0120] S31. Force the alignment of text and audio to obtain phoneme state sequence and boundary information.

[0121] In this embodiment, the audio includes the template audio and the audio to be tested. Accordingly, the phoneme state sequence includes a first phoneme state sequence corresponding to the audio to be tested and a second phoneme state sequence corresponding to the template audio. The boundary information includes first boundary information corresponding to the audio to be tested and second boundary information corresponding to the template audio.

[0122] S32: Calculate the GOP value of the non-silent phoneme.

[0123] In this embodiment, the template speech and the speech to be tested are subjected to the same calculation process to obtain the template GOP sequence and the GOP sequence to be tested. The specific calculation process can refer to the description in the above embodiment and is not limited in this embodiment.

[0124] S33. Determine phoneme by phoneme whether the template GOP value is smaller than the GOP value to be tested. If so, execute step S34; if not, execute step S35.

[0125] S34: Replace the template GOP value with the GOP value to be tested, and execute step S35.

[0126] S35 , calculating the KL divergence between the template GOP sequence and the GOP sequence to be tested.

[0127] The specific calculation method of KL divergence can refer to the description in the above embodiment and is not limited in this embodiment.

[0128] S36: Determine whether the KL divergence is greater than a preset threshold. If so, execute S37; if not, execute S38.

[0129] S37. The sentences are pronounced correctly.

[0130] S38. The sentence is pronounced incorrectly.

[0131] In this embodiment, the characteristic that the GOP feature is a probability distribution is utilized, and the similarity with the template pronunciation GOP sequence is measured using KL divergence to ensure the accuracy of the evaluation result.

[0132] Figure 5 This is a schematic diagram of the structure of a speech evaluation device provided by an embodiment of the present invention. This device is suitable for detecting whether a learner's pronunciation is correct. The speech evaluation device can be implemented by hardware and / or software. The speech evaluation device can be composed of two or more physical entities or a single physical entity and is generally integrated into a computer device.

[0133] like Figure 2 As shown, the speech evaluation device provided by the embodiment of the present invention mainly includes a GOP sequence determination module 51, a KL distance determination module 52 and an evaluation result determination module 53.

[0134] The GOP sequence determining module 51 is used to determine a first GOP sequence of good pronunciation corresponding to the speech to be tested and a second GOP sequence corresponding to the template speech;

[0135] A KL divergence determination module 52 is configured to determine a relative entropy KL divergence between the first GOP sequence and the second GOP sequence;

[0136] The evaluation result determination module 53 is configured to determine the evaluation result of the speech to be tested based on the KL divergence.

[0137] The speech evaluation device provided in an embodiment of the present invention is configured to perform the following operations: determining a first pronunciation-goodness GOP sequence corresponding to the test speech and a second GOP sequence corresponding to the template speech, determining the KL divergence of the relative entropy between the first and second GOP sequences, and determining an evaluation result for the test speech based on the KL divergence. This embodiment utilizes the fact that GOP features are probability distributions, and measures the similarity between the template GOP sequences using the KL divergence, thereby improving the accuracy of the evaluation results.

[0138] Furthermore, the GOP sequence determination module 51 includes: a first GOP sequence determination unit and a second GOP sequence determination unit;

[0139] A first GOP sequence determining unit is configured to obtain a first phoneme state sequence and first boundary information of the speech to be tested; and determine a first GOP sequence corresponding to the speech to be tested based on the first phoneme state sequence and the first boundary information;

[0140] The second GOP sequence determining unit is configured to obtain a second phoneme state sequence and second boundary information of the template speech; and determine a second GOP sequence corresponding to the template speech based on the second phoneme state sequence and the second boundary information.

[0141] Furthermore, the first GOP sequence determining unit is specifically configured to determine a first GOP value corresponding to each non-silent phoneme in the first phoneme state sequence based on the first phoneme state sequence and the first boundary information; and arrange the first GOP values corresponding to the plurality of non-silent phonemes in a first preset order to obtain a first GOP sequence;

[0142] The second GOP sequence determination unit is specifically used to determine the second GOP value corresponding to each non-silent phoneme in the second phoneme state sequence based on the second phoneme state sequence and the second boundary information; and arrange the second GOP values corresponding to multiple non-silent phonemes in a second preset order to obtain a second GOP sequence.

[0143] Furthermore, the second GOP sequence determining unit is further configured to compare, for each phoneme state, a GOP value in the first GOP sequence and a GOP value in the second GOP sequence;

[0144] If the GOP value in the second GOP sequence is smaller than the GOP value in the first GOP sequence, the GOP value in the second GOP sequence is replaced with the GOP value in the first GOP sequence to obtain a new second GOP sequence.

[0145] Furthermore, the KL divergence determining module 52 is specifically configured to determine the relative entropy KL divergence between the first GOP sequence and the second GOP sequence, including: calculating the KL divergence between the first GOP sequence and the second GOP sequence using a KL divergence formula.

[0146] Among them, the KL divergence formula is:

[0147]

[0148] Among them, D KL (P||Q) is the KL divergence between the first GOP sequence and the second GOP sequence, P(i) is the i-th GOP value in the first GOP sequence, and Q(i) is the i-th GOP value in the second GOP sequence.

[0149] Furthermore, the evaluation result determination module 53 is specifically used to determine that the evaluation result of the speech to be tested is correct pronunciation when the KL divergence is greater than or equal to a preset threshold; and to determine that the evaluation result of the speech to be tested is incorrect pronunciation when the KL divergence is less than the preset threshold.

[0150] The speech evaluation device provided in the embodiment of the present invention can execute the speech evaluation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0151] Figure 6 A schematic diagram of the hardware structure of a device provided in an embodiment of the present invention is shown in FIG. Figure 6 As shown, the device includes a processor 601, a memory 602, an input device 603 and an output device 604; the number of processors 601 in the device can be one or more. Figure 6 In the embodiment, a processor 601 is used as an example; the processor 601, the memory 602, the input device 603 and the output device 604 in the device can be connected by a bus or other means. Figure 6 The bus connection is taken as an example.

[0152] The memory 602 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the speech evaluation method in the embodiment of the present invention (for example, the attached Figure 5 The modules in the speech evaluation device shown include a GOP sequence determination module 51, a KL distance determination module 52, and an evaluation result determination module 53. The processor 601 executes the software programs, instructions, and modules stored in the memory 602 to perform various functional applications and data processing of the device, thereby implementing the above-mentioned speech evaluation method.

[0153] The memory 602 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created based on the use of the device, etc. In addition, the memory 602 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 602 may further include a memory remotely located relative to the processor 601, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0154] Furthermore, when one or more programs included in the above device are executed by the one or more processors 601, the program performs the following operations:

[0155] Determine a first pronunciation goodness GOP sequence corresponding to the test speech and a second GOP sequence corresponding to the template speech;

[0156] determining a relative entropy KL divergence between the first GOP sequence and the second GOP sequence;

[0157] An evaluation result of the speech to be tested is determined based on the KL divergence.

[0158] The input device 603 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 604 may include a display device such as a display screen.

[0159] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processing device, the speech evaluation method provided by an embodiment of the present invention is implemented. The method includes:

[0160] Determine a first pronunciation goodness GOP sequence corresponding to the test speech and a second GOP sequence corresponding to the template speech;

[0161] determining a relative entropy KL divergence between the first GOP sequence and the second GOP sequence;

[0162] An evaluation result of the speech to be tested is determined based on the KL divergence.

[0163] Of course, the computer-executable instructions provided in the storage medium of the embodiment of the present invention are not limited to the operations of the method described above, but can also perform the related operations of the speech evaluation method provided in any embodiment of the present invention.

[0164] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0165] It is worth noting that in the embodiment of the above-mentioned speech evaluation device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0166] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A speech evaluation method, characterized in that: include: Determine a first pronunciation goodness GOP sequence corresponding to the test speech and a second GOP sequence corresponding to the template speech; Determining a relative entropy KL divergence between the first GOP sequence and the second GOP sequence; wherein the KL divergence is used to measure the degree of difference between the first GOP sequence and the second GOP sequence; Determining an evaluation result of the speech to be tested based on the KL divergence; Determining a first GOP sequence corresponding to the speech to be tested and a second GOP sequence corresponding to the template speech includes: Acquire a first phoneme state sequence and first boundary information of the speech to be tested; Determine a first GOP sequence corresponding to the speech to be tested based on the first phoneme state sequence and the first boundary information; Acquire a second phoneme state sequence and second boundary information of the template speech; determining a second GOP sequence corresponding to the template speech based on the second phoneme state sequence and the second boundary information; After determining a second GOP sequence corresponding to the template speech based on the second phoneme state sequence and the second boundary information, the method further includes: For each phoneme state, comparing a GOP value in the first GOP sequence with a GOP value in the second GOP sequence; If the GOP value in the second GOP sequence is smaller than the GOP value in the first GOP sequence, the GOP value in the second GOP sequence is replaced with the GOP value in the first GOP sequence to obtain a new second GOP sequence.

2. The method according to claim 1, characterized in that Determining a first GOP sequence corresponding to the speech to be tested based on the first phoneme state sequence and the first boundary information, and determining a second GOP sequence corresponding to the template speech based on the second phoneme state sequence and the second boundary information, including: determining, based on the first phoneme state sequence and the first boundary information, a first GOP value corresponding to each non-silent phoneme in the first phoneme state sequence; Arranging first GOP values corresponding to a plurality of non-silent phonemes in a first preset order to obtain a first GOP sequence; determining, based on the second phoneme state sequence and the second boundary information, a second GOP value corresponding to each non-silent phoneme in the second phoneme state sequence; The second GOP values corresponding to the plurality of non-silent phonemes are arranged in a second preset order to obtain a second GOP sequence.

3. The method according to claim 1, characterized in that Determining a relative entropy KL divergence between the first GOP sequence and the second GOP sequence includes: The KL divergence between the first GOP sequence and the second GOP sequence is calculated using a KL divergence formula.

4. The method according to claim 1, wherein The KL divergence formula is: Among them, D KL (P||Q) is the KL divergence between the first GOP sequence and the second GOP sequence, P(i) is the i-th GOP value in the second GOP sequence, and Q(i) is the i-th GOP value in the first GOP sequence.

5. The method according to claim 1, wherein The KL divergence determines the evaluation result of the speech to be tested, including: when the KL divergence is greater than or equal to a preset threshold, determining that the evaluation result of the speech to be tested is correct pronunciation; when the KL divergence is less than the preset threshold, determining that the evaluation result of the speech to be tested is incorrect pronunciation.

6. A speech evaluation device, characterized in that: include: A GOP sequence determination module is used to determine a first pronunciation goodness GOP sequence corresponding to the speech to be tested and a second GOP sequence corresponding to the template speech; a KL divergence determination module, configured to determine a relative entropy KL divergence between the first GOP sequence and the second GOP sequence; wherein the KL divergence is used to measure the degree of difference between the first GOP sequence and the second GOP sequence; An evaluation result determination module, configured to determine an evaluation result of the speech to be tested based on the KL divergence; The GOP sequence determination module includes: a first GOP sequence determination unit and a second GOP sequence determination unit; The first GOP sequence determining unit is configured to obtain a first phoneme state sequence and first boundary information of the speech to be tested; and determine a first GOP sequence corresponding to the speech to be tested based on the first phoneme state sequence and the first boundary information; The second GOP sequence determining unit is configured to obtain a second phoneme state sequence and second boundary information of the template speech; and determine a second GOP sequence corresponding to the template speech based on the second phoneme state sequence and the second boundary information; The second GOP sequence determining unit is further configured to compare the GOP value in the first GOP sequence with the GOP value in the second GOP sequence for each phoneme state; If the GOP value in the second GOP sequence is smaller than the GOP value in the first GOP sequence, the GOP value in the second GOP sequence is replaced with the GOP value in the first GOP sequence to obtain a new second GOP sequence.

7. A speech evaluation device, characterized in that: include: one or more processors; a memory for storing one or more programs; The one or more programs are executed by the one or more processors, so that the one or more processors implement the speech evaluation method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the speech evaluation method according to any one of claims 1 to 5 is implemented.

Images