A method, device, equipment and storage medium for evaluating pronunciation fluency

Abstract

The embodiment of the present application discloses a pronunciation fluency evaluation method, device, equipment and storage medium. The technical solution provided by the embodiment of the present application performs speech recognition on the speech to be evaluated to obtain a test text containing test pause information, and removes the test pause information in the test text to obtain a pause-free text, constructs an inter-word pause path network based on the pause-free text, and determines the optimal path from the inter-word pause path network based on the probability weights corresponding to the direct branch path and the pause branch path in the inter-word pause path network, and determines the reference text containing reference pause information based on the optimal path, and can perform pronunciation fluency evaluation based on the test text and the reference text to obtain the pronunciation fluency evaluation result corresponding to the speech to be evaluated, the reference pause information reflects the pause habits existing in fluent speech, is more suitable for the characteristics of pronunciation pauses, and effectively improves the evaluation effect of pronunciation fluency.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a pronunciation fluency assessment method, apparatus, device and storage medium. Background Art

[0002] Pronunciation fluency assessment is a sub-area of ​​computer-assisted language learning. Pronunciation fluency assessment requires the efficient and accurate restoration of learners' actual pronunciation fluency, providing a fluency evaluation at the level of rhythmic phrases, and helping learners evaluate their oral proficiency.

[0003] The fluency assessment of open-ended questions uses prosodic dimension features, mainly including prosodic phrase pause features, stressed syllable features, boundary tone features, and rhythm features. Among them, the phrase pause feature combines both the speech speed feature and the mastery of grammar, and is an important feature in fluency assessment. However, the current assessment of pronunciation fluency generally starts from the grammatical semantics of the text, ignoring the pause habits that may exist in fluent speech, and cannot adapt to the characteristics of pronunciation pauses, resulting in unsatisfactory results in pronunciation fluency assessment. Summary of the invention

[0004] The embodiments of the present application provide a pronunciation fluency evaluation method, apparatus, device and storage medium to improve the evaluation effect of pronunciation fluency.

[0005] In a first aspect, an embodiment of the present application provides a method for evaluating pronunciation fluency, comprising:

[0006] Acquire the speech to be evaluated, and perform speech recognition on the speech to be evaluated to obtain a test text containing test pause information;

[0007] The test pause information in the test text is removed to obtain a pause-free text, and an inter-word pause path network is constructed based on the pause-free text, wherein the inter-word pause path network records direct branch paths between text words and pause branch paths containing inter-word pauses, as well as probability weights corresponding to the direct branch paths and the pause branch paths, wherein the probability weights are obtained by analyzing the pause-free text using a trained inter-word pause prediction model;

[0008] Determine an optimal path in the inter-word pause path network according to the probability weights, and determine a reference text containing reference pause information based on the optimal path, wherein the optimal path consists of a direct branch path and a pause branch path;

[0009] Pronunciation fluency evaluation is performed based on the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated.

[0010] In a second aspect, an embodiment of the present application provides a pronunciation fluency evaluation device, including a text extraction module, a network construction module, a path determination module and a pronunciation evaluation module, wherein:

[0011] The text extraction module is used to obtain the speech to be evaluated, and perform speech recognition on the speech to be evaluated to obtain a test text containing test pause information;

[0012] The network construction module is used to remove the test pause information in the test text to obtain a pause-free text, and to construct an inter-word pause path network based on the pause-free text, wherein the inter-word pause path network records direct branch paths between text words and pause branch paths containing inter-word pauses, as well as probability weights corresponding to the direct branch paths and the pause branch paths, wherein the probability weights are obtained by analyzing the pause-free text using a trained inter-word pause prediction model;

[0013] The path determination module is used to determine an optimal path in the inter-word pause path network according to the probability weight, and determine a reference text containing reference pause information based on the optimal path, wherein the optimal path consists of a direct branch path and a pause branch path;

[0014] The pronunciation evaluation module is used to perform pronunciation fluency evaluation based on the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated.

[0015] In a third aspect, an embodiment of the present application provides a pronunciation fluency evaluation device, including: a memory and one or more processors;

[0016] The memory is used to store one or more programs;

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the pronunciation fluency evaluation method as described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to perform the pronunciation fluency evaluation method as described in the first aspect.

[0019] The embodiment of the present application performs speech recognition on the speech to be evaluated to obtain a test text containing test pause information, and removes the test pause information in the test text to obtain a pause-free text, constructs an inter-word pause path network based on the pause-free text, and determines the optimal path from the inter-word pause path network based on the probability weights corresponding to the direct branch path and the pause branch path in the inter-word pause path network, and determines the reference text containing reference pause information based on the optimal path, and performs pronunciation fluency evaluation based on the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated, wherein the probability weight is obtained by analyzing the pause-free text based on an inter-word pause prediction model, and the inter-word pause prediction model is trained based on fluent pronunciation speech samples, and the reference pause information reflects the pause habits existing in fluent speech, is more adapted to the characteristics of pronunciation pauses, and effectively improves the evaluation effect of pronunciation fluency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 is a flow chart of a pronunciation fluency evaluation method provided in an embodiment of the present application;

[0021] Figure 2 is a flow chart of another pronunciation fluency evaluation method provided in an embodiment of the present application;

[0022] Figure 3 It is a structural diagram of a weighted finite state transfer network provided in an embodiment of the present application;

[0023] Figure 4 It is a structural schematic diagram of a pronunciation fluency evaluation device provided in an embodiment of the present application;

[0024] Figure 5 It is a structural schematic diagram of a pronunciation fluency evaluation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the present application clearer, the specific embodiments of the present application are further described in detail below in conjunction with the accompanying drawings. It is understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for the convenience of description, only the part related to the present application but not all the contents are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of each operation can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to a method, a function, a procedure, a subroutine, a subprogram, etc.

[0026] Figure 1 A flowchart of a pronunciation fluency evaluation method provided in an embodiment of the present application is given. The pronunciation fluency evaluation method provided in an embodiment of the present application can be performed by a pronunciation fluency evaluation device, which can be implemented by hardware and / or software and integrated in a pronunciation fluency evaluation device.

[0027] The following description is made by taking the pronunciation fluency evaluation device executing the pronunciation fluency evaluation method as an example. Figure 1 , the pronunciation fluency evaluation method includes:

[0028] S101: Acquire a speech to be evaluated, and perform speech recognition on the speech to be evaluated to obtain a test text including test pause information.

[0029] The speech to be evaluated is the speech that needs to be evaluated for pronunciation fluency, which can be obtained by collecting the speech of the user when reading aloud through a microphone, or the speech file that needs to be evaluated for pronunciation fluency can be used as the speech to be evaluated according to the user's selection.

[0030] Optionally, the speech to be evaluated provided in this embodiment can be obtained by reading aloud based on a closed-ended question type or an open-ended question type. A closed-ended question type is a question type with a reference text, and the user needs to read aloud with reference to the reference text, and the reference text refers to a text stored or available in the pronunciation fluency evaluation device; an open-ended question type is a question type without a reference text, that is, the pronunciation fluency evaluation device does not obtain the corresponding reference text, and the user can read aloud without reference.

[0031] Exemplarily, a speech to be evaluated that needs to be evaluated for pronunciation fluency is obtained, and based on speech recognition of the speech to be evaluated, a test text containing test pause information is obtained. The test text includes text words and test pause information. The text words can be understood as each word or text corresponding to the user when reading, and the test pause information can be understood as the pauses between text words when reading. It can be understood that a pause represents a pronunciation pause process, and a pause is manifested as a period of no sound, or external noise, onomatopoeia, subconsciously uttered "oh", "um" and other modal particles.

[0032] For example, a user reads aloud based on an open-ended question to record a speech to be evaluated. The recognized text read by the user is "I like thatto tell you the truth", and there are pauses between "that" and "to", and between "you" and "truth". Then, the test text obtained after speech recognition of the speech to be evaluated is "I likethat|to tell you|the truth", where "I", "like", etc. are text words, and "|" represents a pause between text words.

[0033] In a possible embodiment, for the speech to be evaluated obtained under a closed-ended question type, since the closed-ended question type has a corresponding reference text, the speech to be evaluated can be forcibly aligned according to the reference text, that is, the speech to be evaluated recorded based on the reference text is obtained, and the speech to be evaluated is forcibly aligned to obtain a test text containing test pause information. Specifically, the speech to be evaluated recorded based on the reference text is obtained, and the speech to be evaluated is forcibly aligned with reference to the reference text to obtain a test text containing test pause information. In this embodiment, the speech to be evaluated obtained under a closed-ended question type is forcibly aligned to obtain a test text containing test pause information, and there is no need to perform speech recognition on the speech to be evaluated separately.

[0034] S102: removing the test pause information in the test text to obtain a pause-free text, and constructing an inter-word pause path network based on the pause-free text, wherein the inter-word pause path network records direct branch paths between text words and pause branch paths containing inter-word pauses, as well as probability weights corresponding to the direct branch paths and the pause branch paths, wherein the probability weights are obtained by analyzing the pause-free text using a trained inter-word pause prediction model.

[0035] Exemplarily, after obtaining the test text corresponding to the speech to be evaluated, the test pause information in the test text is removed, and the test text after removing the test pause information is used as the pause-free text. At this time, only the corresponding text words are recorded in the pause-free text.

[0036] For example, after obtaining the test text "I like that|to tell you|the truth" as described above, after removing the test pause information in the test text, the obtained pause-free text is "I like that to tell you thetruth".

[0037] In a possible embodiment, for the test text corresponding to the closed-ended question type, since the user reads aloud according to the reference text, the test text after removing the test pause information is consistent with the reference text, and the reference text can be directly used as the pause-free text.

[0038] Furthermore, after obtaining the pause-free text, an inter-word pause path network is constructed based on the pause-free text. Exemplarily, an inter-word pause is added between each text word in the pause-free text, and the adjacent text words before and after are respectively connected as direct branch paths, and the adjacent text words before and after and the inter-word pause are respectively connected as pause branch paths, and the basic network structure of the inter-word pause path network is composed of text words, inter-word pauses, direct branch paths, and pause branch paths.

[0039] For example, if a pause is added between each word of the pause-free text "I like that to tell you the truth", the longest path in the inter-word pause path network is "I|like|that|to|tell|you|the|truth". Then for the text word "like", it is connected to a direct branch path from "I" to "like", as well as pause branch paths from "I" to "|" and "|" to "like".

[0040] Among them, in the inter-word pause path network provided by this embodiment, text words corresponding to the pause-free text, the inter-word pauses set between the text words, the direct branch paths between the text words, and the pause branch paths containing the inter-word pauses are recorded. The direct branch paths can be understood as the connection between adjacent text words in the pause-free text, and the pause branch paths can be understood as the connection between adjacent text words and the inter-word pauses.

[0041] In addition, the inter-word pause path network provided in the embodiment of the present application also records the probability weights corresponding to the direct branch path and the pause branch path. The probability weight of the direct branch path can be understood as the probability of outputting the following text word under the premise of the previous text word sequence, and the probability weight of the pause branch path can be understood as the probability of the previous text word sequence or inter-word pause to the following text word.

[0042] The probability weight provided in this embodiment is obtained by analyzing the pause-free text based on the inter-word pause prediction model trained based on fluently pronounced speech samples. When training the inter-word pause prediction model, a large number of fluently pronounced speech samples are first collected. These speech samples can use features such as speech speed as preliminary measurement indicators, and the speech samples are decoded or aligned using speech recognition technology to obtain a text sequence containing sample pause information. These speech sequences can be used to train an n-gram language model (i.e., an inter-word pause prediction model) that predicts pauses. The inter-word pause prediction model can analyze the input voice information or text sequence (inter-word pause path network) and output the corresponding probability weight. Optionally, the language model trained by other methods can also be used to predict pauses, such as a Transformer model, etc., which is not limited in this application.

[0043] S103: determining an optimal path in the inter-word pause path network according to the probability weights, and determining a reference text containing reference pause information based on the optimal path, wherein the optimal path consists of a direct branch path and a pause branch path.

[0044] Exemplarily, after determining the probability weights corresponding to the direct branch path and the pause branch path in the inter-word pause network, the optimal path is determined in the inter-word pause network according to the probability weights. The optimal path is composed of the direct branch path and the pause branch path. For example, from the first text word to the last text word, the probability weights corresponding to the paths in the inter-word pause network are searched by the Viterbi algorithm, and the path with the largest statistical probability weight is taken as the optimal path, and the reference text containing the reference pause information is determined according to the optimal path.

[0045] The optimal path consists of a direct branch path and a pause branch path. For example, if the optimal path is determined to be "I like that | to tell you the truth", then the reference pause information in the optimal path indicates that there is a pause between "that" and "to", and the direct branch paths included therein include "I-like", "like-that", "to-tell", "tell-you", "you-the" and "the-truth", and the pause branch paths include "that-|-to".

[0046] In a possible embodiment, after obtaining a reference text containing reference pause information, a test text containing test pause information is further processed according to the pause / non-pause position of the reference text. At the pause position of the reference text, it is determined whether the pause duration of the test text at the same position is greater than the set first duration threshold, and if so, it is determined to be a pause; at the non-pause position of the reference text, it is determined whether the pause duration of the test text at the same position is greater than the set second duration threshold, and if so, it is determined to be a pause; thereby obtaining a test text containing test pause information. It needs to be explained that since the pause of the reference text reflects the rhythmic phrase pause, the pause duration is shorter, and the pause of the test text reflects the rhythmic phrase pause and the unfluent pause, the pause duration is longer, and the first duration threshold is generally smaller than the second duration threshold.

[0047] S104: Performing a pronunciation fluency evaluation according to the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated.

[0048] Exemplarily, after obtaining the reference text, a pronunciation fluency assessment is performed based on the test text and the reference text to obtain a pronunciation fluency assessment result.

[0049] For example, using a model evaluation strategy, the precision and / or recall rate are calculated according to the test text and the reference text, and the precision and / or recall rate are used as the pronunciation fluency evaluation result. Wherein the precision rate (Precision) indicates how many of the positions predicted to pause in the test text are really required to pause, and the recall rate (Recall) indicates how many of the positions required to pause in the reference text are predicted to pause. It can also be to calculate the pause evaluation index of the machine learning model according to the precision rate and the recall rate, and use the pause evaluation index as the pronunciation fluency evaluation result, for example, using the F score (F-Measure) as the pause evaluation index. In a possible embodiment, it can also be used as the pronunciation fluency evaluation result by the perplexity difference, that is, the language model perplexity (Perplexity) of the reference text and the language model perplexity of the test text are calculated based on the n-gram model respectively, and the difference of the perplexity of the two language models is used as the perplexity difference. In addition, the F score and the perplexity difference can also be weighted and summed as the pronunciation fluency evaluation result. It can be understood that precision, recall, F-score, and perplexity difference are used as evaluation indicators for the quality of the model, which themselves reflect the quality of the model itself. This embodiment uses precision, recall, F-score, and perplexity difference as pronunciation fluency evaluation results, and uses the evaluation indicators of the quality of the model to evaluate the user's pronunciation fluency, and uses reference text as a reference answer to effectively reflect the fluency of the user's pronunciation.

[0050] In the above, by performing speech recognition on the speech to be evaluated, a test text containing test pause information is obtained, and the test pause information in the test text is removed to obtain a pause-free text, and an inter-word pause path network is constructed according to the pause-free text, and according to the probability weights corresponding to the direct branch path and the pause branch path in the inter-word pause path network, the optimal path is determined from the inter-word pause path network, and the reference text containing reference pause information is determined according to the optimal path, and pronunciation fluency evaluation can be performed according to the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated, wherein the probability weight is obtained by analyzing the pause-free text based on the inter-word pause prediction model, and the inter-word pause prediction model is trained based on fluent pronunciation speech samples, and the reference pause information reflects the pause habits existing in fluent speech, is more suitable for the characteristics of pronunciation pauses, and effectively improves the evaluation effect of pronunciation fluency.

[0051] Based on the above embodiments, Figure 2 A flowchart of another pronunciation fluency evaluation method provided in an embodiment of the present application is given, and the pronunciation fluency evaluation method is a specific embodiment of the above-mentioned pronunciation fluency evaluation method. Figure 2 , the pronunciation fluency evaluation method includes:

[0052] S201: Acquire the speech to be evaluated, and perform speech recognition on the speech to be evaluated to obtain a test text including test pause information.

[0053] S202: removing the test pause information in the test text to obtain a pause-free text.

[0054] S203: using the pause prediction model to predict pauses between words for the pause-free text, so as to obtain the prediction probability corresponding to each text word of the pause-free text.

[0055] This embodiment builds a pause prediction model between words, collects a large number of fluently pronounced speech samples, and performs speech recognition on the speech samples to obtain a text sequence. The text sequence records sample words and sample pause information obtained by speech recognition of the speech samples, wherein the sample pause information records the pauses between the sample words. This embodiment regards the pauses between the sample words as special words and records them in the text sequence.

[0056] Further, these text sequences are used as training samples to train the pause prediction model between words, until the number of training times reaches the set number of training times or the accuracy reaches the set accuracy requirement. After the training of the pause prediction model between words is completed, a text without pauses is input to the pause prediction model between words, and the pause prediction model between words will output the corresponding prediction probability of each word in the input text (including the prediction probability that there is no pause between adjacent words and there is a pause before a word).

[0057] Specifically, after obtaining the pause-free text, the pause-free text is input into the trained inter-word pause prediction model, and the inter-word pause prediction model predicts the inter-word pauses for the pause-free text and outputs the prediction probability corresponding to each text word of the pause-free text (including the prediction probability that there is no pause between adjacent text words and the prediction probability that there is a pause before a text word).

[0058] S204: constructing a weighted finite state transition network based on the pause-free text and the predicted probability.

[0059] This embodiment uses a weighted finite state transducer network as a word pause path (Weighted Finite-State Transducers, WFST) network. After obtaining the pause-free text and the predicted probability corresponding to each text word, a weighted finite state transducer network is constructed based on the pause-free text and the predicted probability. Among them, the weighted finite state transducer network records the direct branch paths between text words and the pause branch paths containing pauses between words, as well as the probability weights corresponding to the direct branch paths and the pause branch paths. Among them, the weighted finite state transducer network can be obtained by processing the pause-free text using the OpenFST tool.

[0060] The probability weight provided in this embodiment is determined based on the predicted probability corresponding to each text word of the pause-free text, and this embodiment uses the predicted probability as the probability weight. Specifically, for a direct branch path, the corresponding probability weight is the predicted probability that there is no pause between adjacent words; for a pause branch path preceded by a pause and followed by a text word, the corresponding probability weight is the predicted probability that there is a pause before the text word.

[0061] Figure 3 A structural diagram of a weighted finite state transfer network provided in an embodiment of the present application is shown in FIG. Figure 3 As shown in FIG. 1 , assuming that when the pause-free text is determined to be “I like that to tell you the truth”, the weighted finite state transfer network constructed based on the pause-free text is as follows: Figure 3As shown. Among them, sil is the pause label (i.e., the "|" in the text), the symbol before the ":" is the input label, the symbol after the ":" is the output label, and P(W2|W1) represents the probability weight (i.e., the corresponding predicted probability) that the current label is W2 and the previous label is W1. It can be understood that the longest paths in the weighted finite state transfer network are numbered from 0 to 13, and the corresponding text is "I|like|that|to|tell|you|the|truth", while the text "I like that|to tell youthe truth" corresponds to the paths numbered 0-1-3-5-8-7-9-11-13, there is an inter-word pause between "that" and "to", 5-8 and 8-7 are pause branch paths, and the rest are direct branch paths.

[0062] S205: Determine an optimal path in the weighted finite state transition network according to the probability weight.

[0063] After constructing the weighted finite state transfer network, the optimal path is determined in the weighted finite state transfer network according to the probability weights of each direct branch path and the pause branch path. The optimal path can be determined according to the accumulation of the probability weights of each complete path (including the path of the first text word and the last text word), that is, the complete path with the highest accumulated probability weight is taken as the optimal path. The optimal path can also be searched in the inter-word pause path network based on the probability weight and the existing path search algorithm.

[0064] In a possible embodiment, the optimal path is determined according to the Viterbi algorithm, that is, according to the probability weight, the optimal path is searched in the inter-word pause path network based on the Viterbi algorithm. Searching for the optimal path by the Viterbi algorithm can effectively reduce the amount of data processing and speed up the search efficiency for the optimal path.

[0065] S206: Determine a reference text containing reference pause information based on the optimal path, wherein the optimal path consists of a direct branch path and a pause branch path.

[0066] After determining the optimal path, the positions of the pauses between words are determined according to the optimal path, and the pauses between words are added to the reference text according to the determined positions, thereby obtaining a reference text containing reference pause information. It can be understood that for a reference text with pauses between words, the corresponding optimal path consists of a direct branch path and a pause branch path, while for a reference text without pauses between words, the corresponding optimal path consists of only a direct branch path.

[0067] For example, according to Figure 3The provided weighted finite state transfer network determines the optimal path to be 0-1-3-5-8-7-9-11-1. It can be seen that 5-8 and 8-7 are pause branch paths, and the remaining branch paths are direct branch paths. Then the corresponding reference pause information indicates that there is an inter-word pause between "that" and "to", and the corresponding reference text is "I like that|totell you the truth".

[0068] S207: Based on the model evaluation index of the machine learning model, pause evaluation is performed according to the test text and the reference text to obtain a pause evaluation result.

[0069] This embodiment is based on the model evaluation index of the machine learning model, and performs pause evaluation according to the test text and the reference text to obtain the pause evaluation result. It should be explained that the model evaluation index of the machine learning model itself reflects the quality of the model itself. This embodiment uses the reference text containing reference pause information as a reference control, and uses the pause evaluation result to evaluate the fluency of the user's pronunciation, which effectively reflects the fluency of the user's pronunciation.

[0070] In a possible embodiment, using the F score as a pause evaluation indicator, that is, performing pause evaluation based on the test text and the reference text, includes:

[0071] S2071: Based on the test text and the reference text, calculate the precision and recall corresponding to the speech to be evaluated.

[0072] S2072: Calculate a pause evaluation result corresponding to the speech to be evaluated according to the precision and the recall.

[0073] In the embodiment of the present application, the reference text containing reference pause information is used as the positive class of the actual category, and the test text containing test pause information is used as the positive class of the predicted category, and the precision and recall corresponding to the speech to be evaluated are calculated.

[0074] Specifically, the calculation formula for the accuracy rate provided in this embodiment is:

[0075]

[0076] Among them, T P is the number of pause positions that are consistent between the test text and the reference text, F P The number of pause positions in the reference text where there were no pauses for the test text;

[0077] The calculation formula of the recall rate provided in this embodiment is:

[0078]

[0079] Among them, F N The number of pause positions where pauses were made in the reference text but not in the test text;

[0080] The calculation formula of the pause evaluation index (F score) provided in this embodiment is:

[0081]

[0082] Among them, β is a set balance parameter used to balance precision and recall, and its value is 0≤β≤+∞, which can be adjusted according to the actual application scenario, N* is the number of words in the reference text, and ω is a set adjustment factor. This embodiment takes into account the impact of text length on pronunciation fluency. The longer the text length, the more likely it is to cause pauses in pronunciation. This embodiment introduces a sigmoid-like function Reduce the impact of text length on pronunciation pauses.

[0083] It can be understood that the higher the pause evaluation index (F score), the better the pronunciation fluency of the speech to be evaluated. When the pause evaluation index is low, it means that the speech to be evaluated pauses at positions where it should not pause, and the situation where there is no pause at positions where it is required is more serious.

[0084] S208: Obtain a text evaluation result according to the perplexity difference between the first language model perplexity of the reference text and the second language model perplexity of the test text.

[0085] This embodiment calculates the language model perplexity (including the first language model perplexity and the second language model perplexity) corresponding to the test text and the reference text based on the n-gram model, and uses the perplexity difference between the two as the text evaluation result. In other embodiments, the conversion ratio between the perplexity difference and the text evaluation result can also be pre-set, and the text evaluation result corresponding to the perplexity difference is calculated according to the conversion ratio. It needs to be explained that the perplexity itself reflects the quality of the language model. This embodiment uses the reference text containing reference pause information as a reference control, and uses the text evaluation result to evaluate the user's pronunciation fluency, which effectively reflects the fluency of the user's pronunciation.

[0086] In a possible embodiment, calculating the language model perplexity based on the n-gram model, that is, the perplexity difference between the first language model perplexity according to the reference text and the second language model perplexity of the test text, includes:

[0087] S2081: Calculate the first language model perplexity according to the reference text based on an n-gram model.

[0088] S2082: Calculate the second language model perplexity according to the test text based on the n-gram model.

[0089] S2083: Taking the difference between the second language model perplexity and the first language model perplexity as the perplexity difference.

[0090] Specifically, the calculation formula for the first language model perplexity of the reference text provided in this embodiment is:

[0091]

[0092] Where N* is the number of words in the reference text, w* is the number of words in the reference text, Represents a text sequence of When , the n-gram model outputs probability.

[0093] The calculation formula of the second language model perplexity of the test text provided in this embodiment is:

[0094]

[0095] Where N is the number of words in the test text, w is the number of words in the test text, and w is the number of words in the test text. P(w i |w1,...,w i-1 ) indicates that the text sequence is w1,...,w i-1 When the n-gram model outputs w i probability.

[0096] The calculation formula of the perplexity difference provided in this embodiment is:

[0097] G=PP(S * )-PP(S)

[0098] This embodiment uses the language model perplexity of the text path to calculate the language model perplexity based on the test text and the reference text. When pausing at an inappropriate place, the language model perplexity will be higher, so the text evaluation index can be obtained by the relative difference in the language model perplexity of the two text paths to measure the fluency of the pronunciation.

[0099] S209: performing weighted summation on the pause evaluation result and the text evaluation result to obtain a comprehensive evaluation result, and determining a pronunciation fluency evaluation result corresponding to the speech to be evaluated according to the comprehensive evaluation result.

[0100] Specifically, after obtaining the pause evaluation result and the text evaluation result, a weighted sum is performed on the pause evaluation result and the text evaluation result to obtain a comprehensive evaluation result.

[0101] The calculation formula of the comprehensive evaluation result provided in this embodiment is:

[0102] Score flency =λF break +(1-λ)G

[0103] Among them, λ is the set weight factor, which is used to balance the two evaluation indicators of pause evaluation results (F score) and text evaluation results (perplexity difference).

[0104] Further, after the comprehensive evaluation result is determined, the pronunciation fluency evaluation result corresponding to the speech to be evaluated is determined according to the comprehensive evaluation result. In this embodiment, the comprehensive evaluation result is used as the pronunciation fluency evaluation result corresponding to the speech to be evaluated. In other embodiments, a mapping relationship between the comprehensive evaluation result and the pronunciation fluency evaluation result can also be established in advance, and after the comprehensive evaluation result is determined, the corresponding pronunciation fluency evaluation result is determined according to the mapping relationship, so that the pronunciation fluency evaluation result is more intuitive.

[0105] This embodiment takes into account the different degrees of influence of the pause evaluation results and the text evaluation results on the pronunciation fluency evaluation results, and takes the weighted sum of the pause evaluation results and the text evaluation results as the pronunciation fluency evaluation result. In a possible embodiment, the pause evaluation results or the text evaluation results can also be used separately as the pronunciation fluency evaluation results.

[0106] In the above, by performing speech recognition on the speech to be evaluated, a test text containing test pause information is obtained, and the test pause information in the test text is removed to obtain a pause-free text, and a word pause path network is constructed according to the pause-free text, and the probability weights corresponding to the direct branch path and the pause branch path in the word pause path network are determined, and the optimal path is determined from the word pause path network, and the reference text containing reference pause information is determined according to the optimal path, and the pronunciation fluency evaluation can be performed according to the test text and the reference text to obtain the pronunciation fluency evaluation result corresponding to the speech to be evaluated, wherein the probability weight is obtained by analyzing the pause-free text based on the word pause prediction model, and the word pause prediction model is trained based on the pronunciation fluent speech sample, and the reference pause information reflects the pause habit existing in the fluent speech, which is more suitable for the characteristics of pronunciation pause, and effectively improves the evaluation effect of pronunciation fluency. At the same time, the pronunciation fluency of the user is evaluated with the help of the evaluation index of the model quality and the language model perplexity, which effectively reflects the fluency of the user's pronunciation.

[0107] Figure 4 A structural diagram of a pronunciation fluency evaluation device provided in an embodiment of the present application is given. Figure 4 The pronunciation fluency evaluation device includes a text extraction module 31, a network construction module 32, a path determination module 33 and a pronunciation evaluation module 34.

[0108] Among them, the text extraction module 31 is used to obtain the speech to be evaluated, and perform speech recognition on the speech to be evaluated to obtain a test text containing test pause information; the network construction module 32 is used to remove the test pause information in the test text to obtain a pause-free text, and construct an inter-word pause path network based on the pause-free text, wherein the inter-word pause path network records the direct branch paths between text words and the pause branch paths containing inter-word pauses, as well as the probability weights corresponding to the direct branch paths and the pause branch paths, and the probability weights are obtained by analyzing the pause-free text with a trained inter-word pause prediction model; the path determination module 33 is used to determine the optimal path in the inter-word pause path network according to the probability weights, and determine a reference text containing reference pause information based on the optimal path, wherein the optimal path consists of a direct branch path and a pause branch path; the pronunciation evaluation module 34 is used to perform pronunciation fluency evaluation according to the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated.

[0109] In the above, by performing speech recognition on the speech to be evaluated, a test text containing test pause information is obtained, and the test pause information in the test text is removed to obtain a pause-free text, and an inter-word pause path network is constructed according to the pause-free text, and according to the probability weights corresponding to the direct branch path and the pause branch path in the inter-word pause path network, the optimal path is determined from the inter-word pause path network, and the reference text containing reference pause information is determined according to the optimal path, and pronunciation fluency evaluation can be performed according to the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated, wherein the probability weight is obtained by analyzing the pause-free text based on the inter-word pause prediction model, and the inter-word pause prediction model is trained based on fluent pronunciation speech samples, and the reference pause information reflects the pause habits existing in fluent speech, is more suitable for the characteristics of pronunciation pauses, and effectively improves the evaluation effect of pronunciation fluency.

[0110] In a possible embodiment, the device further includes a reference extraction module, wherein the reference extraction module is configured to:

[0111] The speech to be evaluated is obtained by recording based on the reference text, and forced alignment is performed on the speech to be evaluated to obtain a test text containing test pause information.

[0112] In a possible embodiment, the inter-word pause path network is a weighted finite state transfer network, and the network construction module 32 constructs the inter-word pause path network based on the pause-free text by:

[0113] Using the inter-word pause prediction model, predicting inter-word pauses for the pause-free text to obtain prediction probabilities corresponding to each text word of the pause-free text;

[0114] A weighted finite state transition network is constructed based on the pause-free text and the prediction probability.

[0115] In a possible embodiment, when the path determination module 33 determines the optimal path in the inter-word pause path network according to the probability weight, specifically: searching for the optimal path in the inter-word pause path network based on the Viterbi algorithm according to the probability weight.

[0116] In a possible embodiment, the pronunciation evaluation module 34 is specifically used to: based on the model evaluation index of the machine learning model, perform pause evaluation according to the test text and the reference text to obtain a pause evaluation result; obtain a text evaluation result according to the perplexity difference between the first language model perplexity of the reference text and the second language model perplexity of the test text; perform weighted summation of the pause evaluation result and the text evaluation result to obtain a comprehensive evaluation result, and determine the pronunciation fluency evaluation result corresponding to the speech to be evaluated according to the comprehensive evaluation result.

[0117] In a possible embodiment, when the pronunciation evaluation module 34 performs pause evaluation according to the test text and the reference text, specifically:

[0118] Based on the test text and the reference text, calculating the precision and recall corresponding to the speech to be evaluated;

[0119] The pause evaluation result corresponding to the speech to be evaluated is calculated according to the precision and the recall.

[0120] In a possible embodiment, the calculation formula of the accuracy rate is:

[0121]

[0122] Among them, T P is the number of pause positions that are consistent between the test text and the reference text, F P The number of pause positions in the reference text where there were no pauses for the test text;

[0123] The calculation formula of the recall rate is:

[0124]

[0125] Among them, F N The number of pause positions where pauses were made in the reference text but not in the test text;

[0126] The calculation formula of the pause evaluation index is:

[0127]

[0128] Among them, β is the set balance parameter, N* is the number of words in the reference text, and ω is the adjustment factor.

[0129] In a possible embodiment, the calculation formula of the first language model perplexity of the reference text is:

[0130]

[0131] Where N* is the number of words in the reference text, w* is the number of words in the reference text, Represents a text sequence of Output probability;

[0132] The calculation formula of the second language model perplexity of the test text is:

[0133]

[0134] Where N is the number of words in the test text, w is the number of words in the test text, and P(w i |w1,...,w i-1 ) indicates that the text sequence is w1,...,w i-1 Output w i probability;

[0135] The calculation formula of the perplexity difference is:

[0136] G=PP(S * )-PP(S)

[0137] The calculation formula of the comprehensive evaluation result is:

[0138] Score flency =λF break +(1-λ)G

[0139] Among them, λ is the set weight factor.

[0140] The embodiment of the present application also provides a pronunciation fluency evaluation device, which can integrate the pronunciation fluency evaluation apparatus provided in the embodiment of the present application. Figure 5 Schematic diagram of a pronunciation fluency evaluation device provided in an embodiment of the present application. Figure 5The pronunciation fluency evaluation device includes: an input device 43, an output device 44, a memory 42 and one or more processors 41; the memory 42 is used to store one or more programs; when the one or more programs are executed by the one or more processors 41, the one or more processors 41 implement the pronunciation fluency evaluation method provided in the above embodiment. The input device 43, the output device 44, the memory 42 and the processor 41 can be connected by a bus or other means. Figure 5 The example of connecting through bus is taken in the following.

[0141] The memory 42, as a computing device readable storage medium, can be used to store software programs, computer executable programs and modules, such as the program instructions / modules corresponding to the pronunciation fluency evaluation method described in any embodiment of the present application (for example, the text extraction module 41, the network construction module 42, the path determination module 43 and the pronunciation evaluation module 44 in the pronunciation fluency evaluation device). The memory 42 may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to the use of the device, etc. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some instances, the memory 42 may further include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0142] The input device 43 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the device. The output device 44 may include a display device such as a display screen.

[0143] The processor 41 executes the software programs, instructions and modules stored in the memory 42 to perform various functional applications and data processing of the device, that is, to implement the above-mentioned pronunciation fluency evaluation method.

[0144] The pronunciation fluency evaluation apparatus, device and computer provided above can be used to execute the pronunciation fluency evaluation method provided in any of the above embodiments, and have corresponding functions and beneficial effects.

[0145] The embodiment of the present application also provides a storage medium containing computer executable instructions, which are used to perform the pronunciation fluency evaluation method provided in the above embodiment when executed by a computer processor. The pronunciation fluency evaluation method includes: obtaining a speech to be evaluated, and performing speech recognition on the speech to be evaluated to obtain a test text containing test pause information; removing the test pause information in the test text to obtain a pause-free text, and constructing an inter-word pause path network based on the pause-free text, wherein the inter-word pause path network records direct branch paths between text words and paths containing test pauses; A pause branch path for inter-word pauses, and probability weights corresponding to the direct branch path and the pause branch path, wherein the probability weights are obtained by analyzing the pause-free text using a trained inter-word pause prediction model; an optimal path is determined in the inter-word pause path network according to the probability weights, and a reference text containing reference pause information is determined based on the optimal path, wherein the optimal path consists of a direct branch path and a pause branch path; and a pronunciation fluency evaluation is performed according to the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated.

[0146] Storage medium - any of various types of memory devices or storage devices. The term "storage medium" is intended to include: installation media, such as CD-ROM, floppy disk or tape device; computer system memory or random access memory, such as DRAM, DDR RAM, SRAM, EDO RAM, Rambus RAM, etc.; non-volatile memory, such as flash memory, magnetic media (such as hard disk or optical storage); registers or other similar types of memory elements, etc. Storage media may also include other types of memory or combinations thereof. In addition, the storage medium may be located in the first computer system in which the program is executed, or may be located in a different second computer system, which is connected to the first computer system via a network (such as the Internet). The second computer system can provide program instructions to the first computer for execution. The term "storage medium" may include two or more storage media that may reside in different locations (for example, in different computer systems connected by a network). The storage medium may store program instructions (for example, embodied as a computer program) that can be executed by one or more processors.

[0147] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application is not limited to the pronunciation fluency evaluation method as described above, and can also execute related operations in the pronunciation fluency evaluation method provided in any embodiment of the present application.

[0148] The pronunciation fluency evaluation apparatus, device and storage medium provided in the above embodiments can execute the pronunciation fluency evaluation method provided in any embodiment of the present application. For technical details not described in detail in the above embodiments, please refer to the pronunciation fluency evaluation method provided in any embodiment of the present application.

[0149] The above are only preferred embodiments of the present application and the technical principles used. The present application is not limited to the specific embodiments described herein, and various obvious changes, readjustments and substitutions that can be made by those skilled in the art will not deviate from the scope of protection of the present application. Therefore, although the present application is described in more detail through the above embodiments, the present application is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the scope of the claims.

Claims

1. A pronunciation fluency evaluation method, characterized in that: include: Acquire the speech to be evaluated, and perform speech recognition on the speech to be evaluated to obtain a test text containing test pause information; The test pause information in the test text is removed to obtain a pause-free text, and an inter-word pause path network is constructed based on the pause-free text, wherein the inter-word pause path network records direct branch paths between text words and pause branch paths containing inter-word pauses, as well as probability weights corresponding to the direct branch paths and the pause branch paths, wherein the probability weights are obtained by analyzing the pause-free text using a trained inter-word pause prediction model; Determine an optimal path in the inter-word pause path network according to the probability weights, and determine a reference text containing reference pause information based on the optimal path, wherein the optimal path consists of a direct branch path and a pause branch path; Pronunciation fluency evaluation is performed based on the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated.

2. The pronunciation fluency evaluation method according to claim 1, characterized in that: The pronunciation fluency evaluation method further comprises: The speech to be evaluated is obtained by recording based on the reference text, and forced alignment is performed on the speech to be evaluated to obtain a test text containing test pause information.

3. The pronunciation fluency evaluation method according to claim 1, characterized in that: The inter-word pause path network is a weighted finite state transfer network, and the inter-word pause path network is constructed based on the pause-free text, including: Using the inter-word pause prediction model, predicting inter-word pauses for the pause-free text to obtain prediction probabilities corresponding to each text word of the pause-free text; A weighted finite state transition network is constructed based on the pause-free text and the prediction probability.

4. The pronunciation fluency evaluation method according to claim 1, characterized in that: Determining the optimal path in the inter-word pause path network according to the probability weights includes: According to the probability weights, an optimal path is searched in the inter-word pause path network based on the Viterbi algorithm.

5. The pronunciation fluency evaluation method according to claim 1, characterized in that: The performing of pronunciation fluency evaluation according to the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated includes: Based on the model evaluation index of the machine learning model, performing pause evaluation according to the test text and the reference text to obtain a pause evaluation result; Obtaining a text evaluation result according to a perplexity difference between the first language model perplexity of the reference text and the second language model perplexity of the test text; A weighted sum is performed on the pause evaluation result and the text evaluation result to obtain a comprehensive evaluation result, and a pronunciation fluency evaluation result corresponding to the speech to be evaluated is determined according to the comprehensive evaluation result.

6. The pronunciation fluency evaluation method according to claim 5, characterized in that: The performing pause evaluation according to the test text and the reference text comprises: Based on the test text and the reference text, calculating the precision and recall corresponding to the speech to be evaluated; The pause evaluation result corresponding to the speech to be evaluated is calculated according to the precision and the recall.

7. The pronunciation fluency evaluation method according to claim 6, characterized in that: The calculation formula of the accuracy is: Among them, T P is the number of pause positions that are consistent between the test text and the reference text, F P The number of pause positions in the reference text where there were no pauses for the test text; The calculation formula of the recall rate is: Among them, F N The number of pause positions where pauses were made in the reference text but not in the test text; The calculation formula of the evaluation index is: Among them, β is the set balance parameter, N* is the number of words in the reference text, and ω is the adjustment factor.

8. The pronunciation fluency evaluation method according to claim 7, characterized in that: The calculation formula of the first language model perplexity of the reference text is: Where N* is the number of words in the reference text, w* is the number of words in the reference text, Represents a text sequence of Output The probability of The calculation formula of the second language model perplexity of the test text is: Where N is the number of words in the test text, w is the number of words in the test text, and P(w i |w1,...,w i-1 ) indicates that the text sequence is w1,...,w i-1 Output w i The probability of The calculation formula of the perplexity difference is: G=PP(S * )-PP(S) The calculation formula of the comprehensive evaluation result is: Score flency =λF break +(1-λ)G Among them, λ is the set weight factor.

9. A pronunciation fluency evaluation device, characterized in that: It includes a text extraction module, a network construction module, a path determination module and a pronunciation evaluation module, among which: The text extraction module is used to obtain the speech to be evaluated, and perform speech recognition on the speech to be evaluated to obtain a test text containing test pause information; The network construction module is used to remove the test pause information in the test text to obtain a pause-free text, and to construct an inter-word pause path network based on the pause-free text, wherein the inter-word pause path network records direct branch paths between text words and pause branch paths containing inter-word pauses, as well as probability weights corresponding to the direct branch paths and the pause branch paths, wherein the probability weights are obtained by analyzing the pause-free text using a trained inter-word pause prediction model; The path determination module is used to determine an optimal path in the inter-word pause path network according to the probability weight, and determine a reference text containing reference pause information based on the optimal path, wherein the optimal path consists of a direct branch path and a pause branch path; The pronunciation evaluation module is used to perform pronunciation fluency evaluation based on the test text and the reference text to obtain a pronunciation fluency evaluation result corresponding to the speech to be evaluated.

10. A pronunciation fluency evaluation device, characterized in that: include: memory and one or more processors; The memory is used to store one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the pronunciation fluency evaluation method as described in any one of claims 1 to 8.

11. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions are used to execute the pronunciation fluency evaluation method according to any one of claims 1 to 8 when executed by a computer processor.

Images