Pronunciation training decoded graph construction method, pronunciation evaluation method and device

Abstract

The embodiment of the invention relates to the technical field of speech analysis, in particular to a pronunciation training decoded graph construction method and a pronunciation evaluation method and device. According to the method, an element sequence appearing in a preset text is obtained as a preset text sequence, a first probability and a second probability are calculated according to maximum likelihood estimation of elements in the preset text sequence, the first probability is the conditional probability of appearing of a sequence inconsistent with the preset text in pronunciation to be evaluated, and the second probability is the conditional probability of appearing of a sequence inconsistent with the preset text in the pronunciation to be evaluated. The second probability is a conditional probability of occurrence of a preset text sequence in the to-be-evaluated pronunciation; and constructing an n-gram grammar model according to the obtained preset text sequence, the first probability and the second probability, and constructing a decoded graph corresponding to the n-gram grammar model. According to the method provided by the embodiment of the invention, the conditions of skip reading, repeated reading, disorder and the like can be processed, only the maximum likelihood estimation needs to be calculated, the implementation is simple, the consumed labor cost is low, and massive text training does not need to be used.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of speech analysis, and in particular to a pronunciation training decoding graph construction method, a pronunciation evaluation method and a device. Background Art

[0002] In the pronunciation training task, you need to first specify the preset text of the pronunciation content, and then evaluate the correctness of the learner's pronunciation based on the preset text. Among them, the commonly used pronunciation evaluation indicators include pronunciation accuracy, fluency, completeness, etc.

[0003] Generally, the pronunciation training system includes the following steps: constructing a decoding graph based on the preset text; inputting the learner's pronunciation audio into the acoustic model, which is one of the most important parts of the speech recognition system and can output the corresponding acoustic features based on the input pronunciation audio. Then, an optimal path is searched using the output of the acoustic model and the corresponding weights on the decoding graph; finally, the pronunciation result is evaluated based on the information on the optimal path.

[0004] During pronunciation training, learners often skip or repeat certain sounds or words, and the order of certain sounds or words is disordered. In order to better correct pronunciation and improve learning efficiency, the pronunciation training system needs to detect skipping, repeating, and out-of-order reading, so that the evaluation index is more accurate. In the prior art, a finite state machine is usually used to complete the detection of skipping or repeating reading. However, the decoding graph based on the finite state machine structure has limited functions and cannot support out-of-order reading. It lacks statistical theoretical support, is inconvenient to adjust parameters, and requires a large number of samples for training. Summary of the invention

[0005] The embodiments of the present application provide a pronunciation training decoding graph construction method, a pronunciation evaluation method and a device, which can solve the technical problems in the prior art that the decoding graph based on the finite state machine structure cannot support out-of-order reading and requires a large number of samples for training.

[0006] The purpose of the embodiment of the present invention is achieved through the following technical solutions:

[0007] To solve the above technical problems, in the first aspect, an embodiment of the present invention provides a method for constructing a pronunciation training decoding graph, including: obtaining a sequence of elements appearing in a preset text as a preset text sequence, and calculating a first probability and a second probability respectively according to the maximum likelihood estimate of the elements in the preset text sequence, wherein the first probability is a conditional probability of the occurrence of a sequence inconsistent with the preset text in the pronunciation to be evaluated, and the second probability is a conditional probability of the occurrence of the preset text sequence in the pronunciation to be evaluated; constructing an n-gram grammar model according to the obtained preset text sequence, the first probability and the second probability, and constructing a decoding graph corresponding to the n-gram grammar model.

[0008] In some embodiments, the obtaining of the sequence of elements appearing in the preset text as the preset text sequence specifically includes: performing word segmentation on the preset text, and / or performing phoneme analysis on the preset text; taking one or more words after word segmentation, and / or one or more phonemes after phoneme analysis, as an element in the sequence; arranging the elements in the sequence according to the order of their respective positions in the preset text, and taking the sorted sequence of elements as the preset text sequence.

[0009] In some embodiments, the first probability and the second probability are calculated respectively according to the maximum likelihood estimate of the elements in the preset text sequence, specifically including: based on the preset text, calculating the maximum likelihood estimate of each element in the preset text sequence; calculating a first weight parameter of the first probability, calculating an adjustment factor according to the first weight parameter and the maximum likelihood estimate of each element, and calculating a first probability of a sequence inconsistent with the preset text according to the adjustment factor and the first weight parameter; calculating a second weight parameter according to the first weight parameter, and calculating a second probability of the preset text sequence according to the maximum likelihood estimate of each element and the second weight parameter, wherein the sum of the first weight parameter and the second weight parameter is 1.

[0010] In some embodiments, the method of calculating the maximum likelihood estimate of each element in the sequence based on the preset text specifically includes: when the preset text has been determined, calculating the true conditional probability of occurrence of each element in the preset text as the element probability, and calculating the maximum likelihood estimate of each element when the preset text has been determined based on the element probability.

[0011] In some embodiments, the first probability of a sequence inconsistent with a preset text is calculated based on an adjustment factor and a first weight parameter, specifically comprising: when the preset text has been determined, obtaining elements inconsistent with the preset text sequence, obtaining the element probability that each inconsistent element may appear, and taking the sum of all possible element probabilities as the first weight parameter.

[0012] In some embodiments, the method of constructing an n-gram grammar model based on the acquired preset text sequence, the first probability and the second probability specifically includes: constructing all elements in the preset text sequence into an initial state transition sequence, each element corresponds to a state in the initial state transition sequence, the order of transitions between different states is consistent with the order in which the elements appear in the preset text sequence, and calculating the transition probability between corresponding states in the initial state transition sequence according to the second probability; representing states that do not appear in the preset text with preset states, adding state transitions between each state and the preset state in the initial state transition sequence, and calculating the transition probability between the corresponding state and the preset state according to the first probability.

[0013] In some embodiments, the construction of a decoding graph corresponding to the n-gram grammar model specifically includes: constructing a grammar state machine based on preset text and the n-gram grammar model; and constructing a corresponding decoding graph by combining one or more of a Markov model, a context-based phoneme model, a pronunciation dictionary, and a grammar state machine.

[0014] In a second aspect, an embodiment of the present invention provides a pronunciation evaluation method, comprising: using the pronunciation training decoding graph constructed in the first aspect, specifically: inputting the pronunciation to be evaluated into the acoustic model, searching for a path with the highest transition probability on the decoding graph based on the output of the acoustic model and the corresponding transition probability on the decoding graph, and evaluating the pronunciation based on the information on the searched path.

[0015] In some embodiments, searching for a path with the highest transition probability on the decoding graph specifically includes: if the current pronunciation is consistent with the next element in the preset text sequence, transferring to the state corresponding to the next element; if the current pronunciation is inconsistent with the next element in the preset text sequence, transferring to the preset state, and then transferring to the state corresponding to the current pronunciation in the preset text.

[0016] In some embodiments, the pronunciation is evaluated based on the information on the searched path, specifically including: when the status items included in the path, and the order of the status items, are consistent with the preset text sequence, determining that the order of pronunciation is correct; and / or, when the path includes the preset state and the elements in the preset text sequence are missing, determining that there are skipped readings in the pronunciation; and / or, when the path includes the preset state and the number of elements corresponding to one or more status items in the path is greater than the number of corresponding elements in the preset text sequence, determining that there are repeated readings in the pronunciation; and / or, when the path includes the preset state and the order of elements corresponding to the status items in the path is different from the order of elements in the preset text sequence, determining that there are out-of-order readings in the pronunciation.

[0017] In a third aspect, an embodiment of the present invention provides a pronunciation evaluation device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the first aspect above.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: Different from the prior art, the embodiments of the present invention provide a method for constructing a pronunciation training decoding graph, a pronunciation evaluation method and a device, which, when constructing a language model, not only considers the conditional probability of a sequence that is consistent with a preset text, but also considers the conditional probability of a sequence that is inconsistent with the preset text, so that the established n-gram grammar model can handle skipping, repeated reading, disorder, etc. Moreover, the method only needs to calculate the maximum likelihood estimate, is simple to implement, consumes low labor costs, and does not require the use of massive text training. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise specified, the figures in the drawings do not constitute proportional limitations.

[0020] Figure 1 is a schematic diagram of a finite state machine used in the prior art;

[0021] Figure 2 is another finite state machine schematic diagram used in the prior art;

[0022] Figure 3 It is a flow chart of a method for constructing a pronunciation training decoding graph provided in Embodiment 1 of the present invention;

[0023] Figure 4 It is a flowchart of another method for constructing a pronunciation training decoding graph provided in the first embodiment of the present invention;

[0024] Figure 5 It is a flowchart of another method for constructing a pronunciation training decoding graph provided in the first embodiment of the present invention;

[0025] Figure 6 It is a flowchart of another method for constructing a pronunciation training decoding graph provided in Embodiment 1 of the present invention;

[0026] Figure 7 It is a flowchart of another method for constructing a pronunciation training decoding graph provided in the first embodiment of the present invention;

[0027] Figure 8 is a schematic diagram of a decoding graph constructed according to the method for constructing a decoding graph for pronunciation training provided in Embodiment 1 of the present invention;

[0028] Fig. 9 It is a hardware structure diagram of a pronunciation evaluation device provided in Embodiment 2 of the present invention. DETAILED DESCRIPTION

[0029] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0030] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0031] It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, and all are within the scope of protection of this application. In addition, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be performed in a different order than the module division in the device or the order in the flow chart.

[0032] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0033] When performing pronunciation evaluation, for the preset text "singasong", the grammar finite state machine for alignment shown in 1 can be used for pronunciation evaluation. However, the state machine has only one set of state transition sequences. In the actual pronunciation process of learners, skipping, repeated reading, and out-of-order reading may occur. In these cases, the phoneme sequence in the pronunciation will be inconsistent with the phoneme sequence in the preset text. If the existing state machine is used for evaluation, starting from the first skipped, repeated, or out-of-order phoneme, the subsequent phonemes will be judged as incorrect even if they are pronounced correctly, so only the pronunciation consistent with the preset text sequence can be evaluated. For example, "singsong", even if "song" is pronounced correctly, the comparison element of the next state in the state machine is "a", so it will also be judged as a pronunciation error.

[0034] In the prior art, usually Figure 1 Some additional edges are added to the finite state machine shown to implement skip read and duplicate read detection, such as Figure 2 As shown, where ε represents a null input or a null output. Figure 2The finite state machine can establish a mapping from the acoustic model modeling unit to the word sequence, which is the decoding graph in the pronunciation training system. However, the decoding graph constructed by this method does not support out-of-order reading well during the decoding process. For example, "a sing song" cannot be detected, the function is limited, and there is a lack of statistical theory support, which makes it inconvenient to adjust parameters.

[0035] For the problem of out-of-order reading, the n-gram model method can also be used in the prior art. However, the existing n-gram model still has the following problems:

[0036] 1. If only one reference text is used to train the existing n-gram model, due to the uncertainty of the reference text, this training process needs to be implemented in each evaluation process. The training process requires the calculation of many statistics, which is complex to implement, has a long development cycle, and high labor costs;

[0037] 2. If we use massive amounts of text to train the n-gram model proposed by previous researchers, the training time will be long and the resources consumed will be high. In addition, since the trained model is large, the decoding time will also be long in actual applications.

[0038] During pronunciation training, learners often skip, repeat, or read certain sounds or words out of order. In order to better correct pronunciation and improve learning efficiency, an embodiment of the present invention provides a method for constructing a decoding graph for pronunciation training, which can construct a corresponding decoding graph based on an n-gram state machine without using a large number of samples, thereby easily detecting skip, repeat, or read out of order, thereby making the evaluation index more accurate.

[0039] Specifically, the embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0040] Embodiment 1

[0041] The embodiment of the present invention provides a method for constructing a pronunciation training decoding graph, see Figure 3 , which shows the process of the pronunciation training decoding graph construction method provided by the embodiment of the present invention, which can be applied in pronunciation evaluation, and the pronunciation training decoding graph construction method includes but is not limited to the following steps:

[0042] Step 101: Obtain a sequence of elements appearing in a preset text as a preset text sequence, and calculate a first probability and a second probability respectively according to a maximum likelihood estimate of the elements in the preset text sequence, wherein the first probability is a conditional probability of the occurrence of a sequence inconsistent with the preset text in the pronunciation to be evaluated, and the second probability is a conditional probability of the occurrence of the preset text sequence in the pronunciation to be evaluated.

[0043] The pronunciation to be evaluated contains one or more phonemes, which are the smallest speech units divided according to the natural properties of speech. They are analyzed according to the pronunciation actions in the syllables, and one action constitutes a phoneme. When performing pronunciation evaluation, it is necessary to evaluate whether each phoneme in the input pronunciation is consistent with the corresponding phoneme in the preset text based on the content of the preset text. Therefore, it is necessary to obtain the sequence of phonemes that may appear in the preset text, and construct a state machine for pronunciation evaluation based on the sequence. In this embodiment, the preset text sequence can be a text-based word sequence, or a pronunciation-based sound sequence, or other sequences that can be used as pronunciation evaluation criteria. The words or sounds in the sequence can correspond to the phonemes in the state machine and can be used as the evaluation criteria for pronunciation. In this embodiment, the words or sounds that can be used as the pronunciation evaluation criteria, as well as other independent units that can be used as evaluation criteria, are collectively referred to as elements in the sequence.

[0044] In the pronunciation actually input, in addition to the correct pronunciation consistent with the preset text, it also includes incorrect pronunciations such as skipping, repeated reading, and out-of-order reading that are inconsistent with the preset text. Therefore, in this embodiment, the probability of transition between different states in each sequence is also different. In order to support the determination of pronunciations inconsistent with the preset text, in addition to calculating the second probability of the occurrence of the preset text sequence, it is also necessary to assign the first probability that may appear to the sequence inconsistent with the preset text. Since the sequence of the preset text has been determined, the second probability of the occurrence of the preset text sequence is the largest, and the first probability of the occurrence of the sequence inconsistent with the preset text is smaller. In this embodiment, since the preset text has been determined, the first probability and the second probability are both probabilities generated under the condition that the preset text has been determined, and therefore are both conditional probabilities under this condition. Conditional probability refers to the probability of an event occurring under the condition that another event has occurred. In this embodiment, the first probability represents the conditional probability that the sequence in the pronunciation to be evaluated is inconsistent with the preset text sequence when the preset text determines that the event has occurred; the second probability represents the conditional probability that the sequence in the pronunciation to be evaluated is consistent with the preset text sequence when the preset text determines that the event has occurred. . Furthermore, in order to avoid the problem of requiring a large number of sample training in the prior art, in the method provided in this embodiment, the corresponding conditional probability is calculated based on the maximum likelihood estimate of each element in the sequence.

[0045] Step 102: construct an n-gram grammar model according to the acquired preset text sequence, the first probability and the second probability, and construct a decoding graph corresponding to the n-gram grammar model.

[0046] N-gram, also known as n-gram grammar, is a sequence consisting of n consecutive items, which can be words, phonemes, etc. After defining some probability calculation methods on n-gram, an n-gram model can be constructed, so that the probability of a given sequence occurring can be calculated. In step 101, a preset text sequence and a corresponding second probability have been obtained, and a first probability of a sequence that is inconsistent with the preset text in the pronunciation to be evaluated has also been obtained. Therefore, a corresponding n-gram model can be constructed by the obtained preset text sequence, the first probability and the second probability.

[0047] After obtaining the n-gram model, a mapping can be established from the modeling unit of the acoustic model to a preset text sequence and a sequence inconsistent with the preset text. This mapping is used as a decoding graph in the pronunciation training system for pronunciation judgment.

[0048] After step 101 and step 102 provided in this embodiment, the construction of the n-gram grammar model can be completed through simple calculations, and by including sequences inconsistent with the preset text in the n-gram grammar model, evaluation of skipping, repeated reading, and out-of-order reading can be achieved.

[0049] In order to build an n-gram model, we first need to obtain the possible sequences that may appear in the preset text. Figure 4 As shown, the following methods can be used to obtain it.

[0050] Step 201: performing word segmentation on a preset text and / or performing phoneme analysis on the preset text.

[0051] When evaluating pronunciation, word sequences or sound sequences can be used as evaluation criteria. The preset text contains word sequences and sound sequences corresponding to the text. Therefore, the word sequence in the preset text can be obtained through word segmentation, or the sound sequence can be obtained through phoneme analysis corresponding to the text.

[0052] Step 202: taking one or more words after word segmentation and / or one or more phonemes after phoneme analysis as an element in a sequence.

[0053] After word segmentation or phoneme analysis, mutually independent words or phonemes can be obtained. In specific implementations, each word or phoneme can be used as an element in a sequence as needed; or multiple words or phonemes can be combined as an element in a sequence.

[0054] In the following, for the sake of simplicity, w is used. i (i=1...n) represents the elements in the sequence. i (i=1...n) are all variables, the values ​​of the variables correspond to the elements split out from the preset text, and n is the number of elements.

[0055] Step 203: Arrange the elements in the sequence according to the order in which they appear in the preset text, and use the sorted element sequence as the preset text sequence.

[0056] After obtaining the elements in the sequence, it is also necessary to arrange the elements into a preset text sequence according to the order of the preset text to indicate the order of state transition.

[0057] In the following, for the sake of simplicity, we use Represents a series of n words w 1 w 2 …w n The sequence is composed of i (i=1...n) is the element obtained in step 202.

[0058] After steps 201 to 203 provided in this embodiment, the required preset text sequence can be obtained according to the preset text.

[0059] On the other hand, in order to build an n-gram model, it is also necessary to obtain the conditional probabilities of different state transitions. Figure 5 As shown, the following method can be used to calculate the conditional probability.

[0060] Step 301: Based on the preset text, calculate the maximum likelihood estimate of each element in the preset text sequence.

[0061] In this embodiment, the conditional probability of the possible sequence is calculated by calculating the maximum likelihood estimate of each element, including the first probability of the sequence inconsistent with the preset text, and the second probability of the preset text sequence. The maximum likelihood estimate can use known samples to find the parameters that are most likely to generate the sample, without using a large number of real samples for statistical analysis to obtain the parameters for generating the sample. Therefore, using the maximum likelihood estimate to calculate the conditional probability can avoid using a large number of samples for training, and only a simple calculation is required to obtain an available n-gram model. In this embodiment, the condition of state transition in the conditional probability is the preset text. In a specific implementation, the real conditional probability of each element appearing in the preset text can be calculated as the element probability when the preset text has been determined. In this embodiment, the element probability represents the conditional probability of the occurrence of a certain element in the pronunciation to be evaluated when the preset text determines that this event has occurred. For example: when the preset text has been determined to be "sing a song", the conditional probability of the element "sing" appearing, that is, the conditional probability of the element "sing" appearing in the pronunciation to be evaluated when the preset text is the event that the preset text is "sing a song". Since the preset text has been determined, the element probability of each element appearing in the preset text has been determined and can be obtained through simple calculation. Therefore, the maximum likelihood estimate of each element when the preset text has been determined can be calculated based on the element probability, and then the first probability and the second probability can be calculated without using a large number of samples for analysis.

[0062] Step 302: Calculate a first weight parameter of a sequence inconsistent with a preset text, calculate an adjustment factor according to the first weight parameter and a maximum likelihood estimate of each element, and calculate a first probability of the sequence inconsistent with the preset text according to the adjustment factor and the first weight parameter.

[0063] In pronunciation evaluation, in the case of skipping, repeated reading and out-of-order reading, a sequence inconsistent with the preset text will appear. In the method provided in this embodiment, it is necessary to handle the situations that are different from the preset text sequence, such as skipping, repeated reading and out-of-order reading. Therefore, in addition to calculating the second probability of transition between different states in the sequence of the preset text, it is also necessary to assign the first probability of possible occurrence to the sequence that did not appear. In actual implementation, the first weight parameter can be specified according to the order in which the sequence may appear. Specifically: when the preset text has been determined, the elements inconsistent with the preset text sequence are obtained. In actual implementation, the elements inconsistent with the preset text sequence may include elements that do not exist in the preset text sequence, elements that are inconsistent with the order of appearance of the preset text sequence, and elements that are inconsistent with the number of preset text sequences. The element probability that each inconsistent element may appear is obtained, and the sum of the probabilities of all possible elements is used as the first weight parameter. Further, in order to facilitate the adjustment of the first probability, the adjustment factor can also be used to adjust the allocated first probability. The larger the adjustment factor, the greater the probability that the sequence that did not appear may appear, and the easier it is to detect situations such as skipping, repeated reading, and out-of-order reading.

[0064] Step 303: Calculate a second weight parameter according to the first weight parameter, and calculate a second probability of a sequence consistent with the preset text according to the maximum likelihood estimate of each element and the second weight parameter.

[0065] After reserving the first probability for the sequence that did not appear, the second probability of the sequence with the preset text can be calculated based on the conditional probability of the sequence different from the preset text. For the sequence with the preset text, since the text sequence is determined, the conditional probability of the transition between different states is also determined, and the conditional probability can be calculated based on the maximum likelihood estimation. However, in the method provided in this embodiment, in order to support the evaluation of the sequence inconsistent with the preset text, it is necessary not only to calculate according to the maximum likelihood estimation, but also to calculate according to the first weight parameter. Since the first weight parameter is used to adjust the weight of the first probability of the sequence inconsistent with the preset text, and the second weight parameter is used to adjust the weight of the second probability of the sequence with the preset text, since the sum of the probabilities of a sequence that may appear and that that cannot appear is 1, the sum of the first weight parameter and the second weight parameter is 1.

[0066] After steps 301 to 303 provided in this embodiment, the conditional probability of transition between states can be obtained.

[0067] After obtaining the sequence and transition probability, the corresponding n-gram model can be constructed. The n-gram model is a probabilistic language model based on the (n-1)-order Markov chain, which infers the structure of the sentence through the probability of the occurrence of n words. Figure 6 As shown, the n-gram model can be constructed based on conditional probability in the following way.

[0068] Step 401: All elements in the preset text sequence are constructed into an initial state transition sequence, each element is used as a state in the initial state transition sequence, the order of transitions between different states is consistent with the order in which the elements appear in the preset text sequence, and the transition probability between corresponding states in the initial state transition sequence is calculated according to the second probability.

[0069] The pronunciation of the preset text can be regarded as a state transition process starting from the first element of the sequence and transferring to the last element one by one. When it is confirmed that the pronunciation of the next element is consistent with the pronunciation corresponding to the state to be transferred, it means that the state transfer is completed. Therefore, it is first necessary to construct an initial state transfer sequence consistent with the preset text according to the preset text sequence to represent the state transition corresponding to the correct pronunciation, and the state transition probability between different states is calculated according to the second probability. Furthermore, in actual implementation, in order to facilitate the identification of the beginning and end of a pronunciation, it is also necessary to add the corresponding states of the start and end. In the case of correct pronunciation, the state machine is evaluated by the "start" state and transferred to the corresponding elements in the initial state transfer sequence one by one until the "end" state completes the evaluation.

[0070] Step 402: adding a state transition between each state and a preset state in the initial state transition sequence, and calculating a transition probability between the corresponding state and the preset state according to a first probability.

[0071] In the present embodiment, in order to support a sequence that is inconsistent with the preset text, it is therefore necessary to add a state that does not appear in the preset text. In the present embodiment, a preset state is used to represent a state transition that is inconsistent with the preset text. In actual implementation, the preset element can be set as needed. For example, an empty state is used as a preset element, and an empty state is used to represent "a state corresponding to an element that is inconsistent with the next element in the preset text". When an element that does not appear in the preset text appears in the pronunciation to be evaluated, or the next state to be transferred is different from the next state in the initial state transfer sequence, the state is transferred to. The state transition probability between different states is calculated based on the first probability.

[0072] After step 401 and step 402 provided in this embodiment, an n-ary state transition model corresponding to the preset text can be constructed.

[0073] After obtaining the n-ary state transition model, such as Figure 7 As shown, a decoding graph can be constructed based on the above n-gram grammar model.

[0074] Step 501: construct a grammar state machine according to a preset text and an n-gram grammar model.

[0075] After a preset text is given, a weighted finite state machine can be constructed according to the above n-gram grammar model. The weighted finite state machine represents the language model itself. Usually, this finite state machine is called a grammar state machine.

[0076] Step 502: Construct a corresponding decoding graph by combining one or more of the Markov model, the context-based phoneme model, the pronunciation dictionary, and the n-gram state machine.

[0077] In actual implementation, the acoustic model can be constructed by combining the deep neural network-hidden Markov model (HMM-DNN) with the context-based phoneme modeling. DNN-HMM is an algorithm for speech recognition. It combines DNN (deep neural network) with HMM (hidden Markov model) and trains DNN to predict the state transition probability in HMM. After the acoustic model is built, the decoding graph can be easily constructed using open source tools by merging HMM, context-based phoneme model, pronunciation dictionary and grammar state machine.

[0078] After step 401 and step 402 provided in this embodiment, a decoding graph corresponding to the n-ary state transition model can be constructed.

[0079] Embodiment 2

[0080] After obtaining the decoding graph according to the method provided in the first embodiment, the pronunciation to be evaluated can be input into the acoustic model, and a path with the highest transition probability can be searched on the decoding graph according to the output of the n-gram model and the corresponding transition probability on the decoding graph, and the pronunciation can be evaluated according to the information on the searched path. Specifically: the learner's pronunciation audio is input into the acoustic model, and a path with the best probability is searched using the output of the acoustic model and the corresponding probability on the decoding graph; finally, the information on the best path can be used to evaluate and feedback the pronunciation results, such as calculating indicators such as fluency, accuracy, and completeness.

[0081] When performing pronunciation evaluation, it is first necessary to use an acoustic model to analyze the sound data of the pronunciation, and parse out the corresponding sound features or phonemes as output, so as to correspond the parsed phonemes to the elements in the n-gram model. In actual implementation, the state transition path that is most consistent with the output of the acoustic model can be found in the decoding diagram according to the conditional probability in the n-gram model, and the state transition sequence of the path is compared with the preset text sequence. Since the n-gram model constructed by the method provided in this embodiment already contains a sequence that is inconsistent with the preset text, it can be determined whether there are skipped readings, repeated readings, out-of-order readings, etc. that do not appear in the preset text. On the other hand, by comparing each output phoneme with the element corresponding to each state on the path, the pronunciation evaluation can be completed.

[0082] When searching for a path, if the current pronunciation is consistent with the next element in the preset text sequence, it will be transferred to the state corresponding to the next element. If the current pronunciation is inconsistent with the next element in the preset text sequence, it will be transferred to the preset state, and then transferred to the state corresponding to the current pronunciation in the preset text.

[0083] After searching for the state transition path, the pronunciation can be evaluated based on the information on the searched path.

[0084] 1. When the status items contained in the path, and the order of the status items, are consistent with the preset text sequence, it is determined that the order of pronunciation is correct.

[0085] 2. When the path contains a preset state and an element in the preset text sequence is missing, it is determined that there is a skip in the pronunciation.

[0086] 3. When the path contains a preset state, and the number of elements corresponding to one or more state items in the path is greater than the number of corresponding elements in the preset text sequence, it is determined that there is repeated reading in the pronunciation.

[0087] 4. When the path contains a preset state, and the order of elements corresponding to the state items in the path is different from the order of elements in the preset text sequence, it is determined that there is out-of-order reading in the pronunciation.

[0088] It can be seen that the pronunciation training decoding graph construction method in this embodiment proposes a simple n-gram grammar model, based on statistical theory, calculates the conditional probability through maximum likelihood estimation, and assigns a first probability to the sequence that does not appear, and adds a preset state in the n-gram grammar model to match the sequence that is inconsistent with the preset text. This method only needs to change the first weight parameter, the second weight parameter and the adjustment factor in the conditional probability to adjust the probability of detecting a sequence that is inconsistent with the preset text. The parameter adjustment is convenient, and the detection of skipped reading, repeated reading, and out-of-order reading in the pronunciation training task can be effectively realized.

[0089] Embodiment 3

[0090] Based on the pronunciation training decoding graph construction method provided in Example 1, in this embodiment, according to steps 301-303, a specific formula for calculating conditional probability in an n-gram model is provided. It is understandable that the following formula is only a principle description of the method in Example 1, and a specific implementation method that can be referenced by the method in Example 1, and is not intended to limit the scope of protection. In actual implementation, the conditional probability can be calculated with reference to the following method, or the following formula can be adjusted without creative work to meet the evaluation needs in different implementation scenarios.

[0091] In a specific implementation scenario, the preset text sequence obtained from the preset text is Hereinafter referred to as sequence. i (i=1...n) is an element in the sequence. Under the condition that the preset text has been determined, w n The recursive definition of the conditional probability of is as follows.

[0092]

[0093] in, Indicates the first probability that a sequence inconsistent with the preset text may appear, Indicates the second probability of the occurrence of the preset text sequence. Indicates that a sequence has been given Under the condition of The maximum likelihood estimate of ; Indicates that a sequence has been given Under the condition of , the probability of the element appearing in the word that is inconsistent with the preset text sequence in the pronunciation to be evaluated; the adjustment factor To ensure the defined element probability to w n The sum of all values ​​of is 1. After the first weight parameter β and the maximum likelihood estimates are obtained, the adjustment factor The value of can be derived according to the condition "the second probability of the occurrence of the preset text sequence + the first probability of the occurrence of the sequence inconsistent with the preset text = 1", and its calculation formula is shown in the following formula (5); the first weight parameter β is set to assign some appropriate probability values ​​to the sequence that has not appeared. After setting the first weight parameter, a certain first probability is assigned to skipping, repeated reading, random reading, etc., so that the n-gram model in the first embodiment can support skipping, repeated reading, and random reading. Correspondingly, the second weight parameter (1-β) can also be calculated according to the value of the first weight parameter β to obtain the second probability of the occurrence of the preset text sequence.

[0094] In the scenario where the pronunciation is completely wrong, the elements in the pronunciation are completely inconsistent with the sequence in the preset text, and the sequence Did not appear. In this scenario, does not exist Obviously, in this case, the value of the first weight parameter can only be 1. Therefore, the first weight parameter β can be defined by formula (2):

[0095]

[0096] Among them, α is a constant greater than zero and less than 1, which means that there is , assigned to all w does not appear under the condition n The probability of an element The sum of is the only hyperparameter in the method provided in Example 1. For the sake of modeling simplicity, an element w is used here n Represents all possible elements that are different from the next element in the preset text sequence under the condition that a certain element in the preset text sequence appears. For example: mispronunciation, elements that repeat the previous element in the preset text sequence, elements that are different from the next element in the preset text sequence, etc., so as to support mispronunciation, repeated reading, skipped reading, out-of-order reading, etc. In the existing n-gram grammar model, there are similar parameters, but similar parameters are estimated through some statistics and other hyperparameters, and the calculation is relatively complex and the implementation is relatively cumbersome. In the method provided in Example 1, by giving a sequence The probability that wn does not appear under the condition The sum is calculated without using a large number of samples for statistics or training.

[0097] Generally, the larger the value of parameter α is, the larger the first probability assigned to sequences that do not appear in the preset text, such as skipped reading, repeated reading, and out-of-order reading, which means that these non-appearing sequences are easier to detect. Therefore, by adjusting this parameter, the strictness of detecting skipped reading, repeated reading, and out-of-order reading during evaluation can be adjusted.

[0098] Furthermore, in formula (1), if we follow the probability definition of the unigram model and do not assign additional probability to non-appearing words, we can use the maximum likelihood estimation, that is,

[0099]

[0100] According to the probability constraints

[0101]

[0102] Easy to export

[0103]

[0104] It can be seen that the required values ​​of the first weight parameter, the second weight parameter and the adjustment factor can be calculated through the maximum likelihood probability, and the corresponding first probability and second probability can be calculated according to these parameter values.

[0105] Embodiment 4

[0106] Based on the pronunciation training decoding graph construction method provided in the first embodiment and the conditional probability calculation method provided in the second embodiment, this embodiment provides a specific process for performing pronunciation evaluation in an actual scenario.

[0107] In the scenario of this embodiment, the preset text is: "sing a song".

[0108] According to step 101, the preset text can obtain the three words "sing", "a", and "song" through word segmentation, or obtain the three phonemes corresponding to these three words. Then the three words "sing", "a", and "song" obtained by word segmentation are used as three elements in the preset text sequence, or "sing" is used as an element and "a song" is combined as an element. For the sake of simplicity, in this embodiment, the three words "sing", "a", and "song" are used as elements in the preset text sequence. That is, the preset text sequence is: Among them, w 1 = "sing", w 2 = "a", w 3 =“song”.

[0109] Add the following to the above preset text: <s> and< / s> , respectively representing the beginning and end of the sentence, and the sequence " <s>In i In 2 In 3 < / s> ", and construct a grammar state machine according to the sequence. In order to further support the judgment of skipping, repeated reading, out-of-order reading, etc., the grammar state machine also includes a preset state ∈, which corresponds to a preset text sequence The element w that does not appear under the condition n , to indicate empty input or output, that is, skipping, repeated reading, out-of-order reading, etc. and state transitions that do not exist in the preset text. <s>、< / s> Both ∈ and ∈ are abstract states and only serve to assist in constructing the grammar state machine.

[0110] The constructed n-gram state machine is as follows Figure 8 In this n-gram state machine, states 1-5 represent <s>,In 1 ,In 2 ,In 3 、< / s> ; State 0 indicates the preset state ∈. When evaluating, <s>As the starting state for state transition, the w in the preset text sequence 1 w 2 w 3 Perform conditional transfers in the order specified in the preset text. <s>,In 1 ,In 2 ,In 3 、< / s> can be transferred to the preset state ∈, indicating that each element may be followed by a state that does not appear in the sequence of the preset text; it can also be transferred from the preset state ∈ to <s>,In 1 ,In 2 ,In 3 、< / s> Any state in the sequence is selected to continue evaluating the subsequent elements, so as to judge the disorder. The state transition probability between different states is represented by the calculation parameters in the corresponding formula of the second embodiment.

[0111] In the n-gram state machine, different states in pronunciation can be transferred through different paths in the state machine.

[0112] If the current pronunciation is consistent with the next element in the preset text sequence, the state is transferred to the corresponding state of the next element. For example, if the current pronunciation is "sing", the current state is 2. The pronunciation of the next item is "a", which is consistent with the preset text, so it is transferred to the corresponding state 3.

[0113] If the current pronunciation is inconsistent with the next element in the preset text sequence, it will be transferred to the state corresponding to the preset state, and then transferred to the state corresponding to the current pronunciation in the preset text. For example: if the current pronunciation is "sing", the current state is 2. The pronunciation of the next item is "song", which is inconsistent with "a" in the preset text, it will be transferred to the preset state 0, and then transferred to the state 4 corresponding to the element "song" in the preset text sequence.

[0114] According to this method, the searched paths under different pronunciation situations are as follows, and pronunciation evaluation can be performed according to the following paths.

[0115] 1. When the status items contained in the path, and the order of the status items, are consistent with the preset text sequence, it is determined that the order of pronunciation is correct.

[0116] In the case of normal pronunciation, the status items in the path include: <s> ,"sing","a","song",< / s> , that is, the output corresponding to the state sequence 1, 2, 3, 4, 5 is w 1 w 2 w 3 ,. This path is consistent with the sequence of the preset text, and it can be determined that the order of pronunciation is correct.

[0117] 2. When the path contains a preset state and an element in the preset text sequence is missing, it is determined that there is a skip in the pronunciation.

[0118] In the case of skipping, the status items in the path include: <s> , "a", "song",< / s> , that is, the path output corresponding to the state sequence 1, 0, 3, 4, 5 is w 2 w 3 , the absence of state 2 means that the corresponding element "sing" is skipped.

[0119] 3. When the path contains a preset state, and the number of elements corresponding to one or more state items in the path is greater than the number of corresponding elements in the preset text sequence, it is determined that there is repeated reading in the pronunciation.

[0120] In the case of repeated reads, the status items in the path include: <s> ,"song","a","a","song",< / s> , that is, the path output corresponding to the state sequence 1, 2, 3, 0, 3, 4, 5 is w 1 w 2 w 2 w 3 , contains two states 3, it can be determined that the corresponding element "a" is read repeatedly.

[0121] 4. When the path contains a preset state, and the order of elements corresponding to the state items in the path is different from the order of elements in the preset text sequence, it is determined that there is disordered reading in the pronunciation.

[0122] In the case of out-of-order reading, the status items in the path include: <s> ,"song","a","sing",< / s> , for example, the output of the path corresponding to the state sequence 1, 0, 4, 0, 3, 0, 2, 0, 5 is w 3 w 2 w 1 ,The order of the elements is different from the order of the preset text, and it can be determined that there is out-of-order reading.

[0123] In actual pronunciation evaluation, the above situations 1-4 may overlap with each other, and can also be judged by corresponding state sequences.

[0124] It can be seen that the n-gram state machine constructed by the method of Example 1 can support the recognition of normal reading, skipping reading, repeated reading, out-of-order reading, etc., realize pronunciation evaluation in different situations, and avoid the influence of skipping reading, repeated reading, out-of-order reading, etc. on the accuracy of pronunciation evaluation.

[0125] Embodiment 5

[0126] The embodiment of the present invention also provides a pronunciation evaluation device based on a language model, see Fig. 9 , which shows a hardware structure, which can: based on the first and second embodiments, according to Figures 3 to 7 The method shown is constructed as Figure 8 Decoding diagram shown; Based on embodiments three and four, based on Figure 8 Pronunciation evaluation was performed based on the decoding graphs in the text box.

[0127] Said Fig. 9 The device comprises: at least one processor 11; and a memory 12 connected to the at least one processor 11 for communication. Fig. 9 The memory 12 stores instructions that can be executed by the at least one processor 11, and the instructions are executed by the at least one processor 11 so that the at least one processor 11 can perform the above Figures 3 to 7 The pronunciation training decoding graph construction method described above is Figure 8 The processor 11 and the memory 12 can be connected via a bus or other means. Fig. 9 The example of connecting through bus is taken in the following.

[0128] The memory 12 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the program instructions / modules corresponding to the pronunciation training decoding graph construction method and the pronunciation evaluation method in the embodiment of the present application. The processor 11 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions and modules stored in the memory 12, that is, the methods of the above-mentioned method embodiments 1 to 4 are implemented.

[0129] The memory 12 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the pronunciation evaluation device based on the language model, etc. In addition, the memory 12 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device or other non-volatile solid-state storage device. In some embodiments, the memory 12 may optionally include a memory remotely arranged relative to the processor 11, and these remote memories may be connected to the pronunciation evaluation device based on the language model via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0130] The one or more modules are stored in the memory 12, and when executed by the one or more processors 11, the pronunciation training decoding graph construction method and the pronunciation evaluation method in any of the above method embodiments are executed, for example, the above described Figures 3 to 7 method steps.

[0131] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0132] In an embodiment of the present invention, a method for evaluating pronunciation based on a language model is provided. When constructing a language model, the method not only considers the conditional probability of a sequence that is consistent with a preset text, but also considers the conditional probability of a sequence that is inconsistent with the preset text, so that the established n-gram model can handle skipping, repeated reading, disorder, etc. Moreover, the method only needs to calculate the maximum likelihood estimate, is simple to implement, consumes low labor costs, and does not require the use of massive text training.

[0133] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0134] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.< / s>

Claims

1. A method for constructing a decoding graph for pronunciation training, characterized in that: include: Obtaining a sequence of elements appearing in a preset text as a preset text sequence, and calculating a first probability and a second probability respectively according to a maximum likelihood estimate of the elements in the preset text sequence, wherein the first probability is a conditional probability that a sequence inconsistent with the preset text appears in the pronunciation to be evaluated, and the second probability is a conditional probability that the preset text sequence appears in the pronunciation to be evaluated; An n-gram grammar model is constructed according to the acquired preset text sequence, the first probability and the second probability, and a decoding graph corresponding to the n-gram grammar model is constructed.

2. The method for constructing a pronunciation training decoding graph according to claim 1, wherein: The obtaining of the element sequence appearing in the preset text as the preset text sequence specifically includes: Performing word segmentation on a preset text, and / or performing phoneme analysis on the preset text; Taking one or more words after word segmentation and / or one or more phonemes after phoneme analysis as an element in a sequence; The elements in the sequence are arranged in order according to their respective positions in the preset text, and the sorted element sequence is used as the preset text sequence.

3. The method for constructing a pronunciation training decoding graph according to claim 1, wherein: The calculating the first probability and the second probability respectively according to the maximum likelihood estimation of the elements in the preset text sequence specifically includes: Based on the preset text, calculate the maximum likelihood estimate of each element in the preset text sequence; Calculating a first weight parameter of a first probability, calculating an adjustment factor according to the first weight parameter and a maximum likelihood estimate of each element, and calculating a first probability of a sequence inconsistent with a preset text according to the adjustment factor and the first weight parameter; A second weight parameter is calculated according to the first weight parameter, and a second probability of a preset text sequence is calculated according to the maximum likelihood estimate of each element and the second weight parameter, wherein the sum of the first weight parameter and the second weight parameter is 1.

4. The method for constructing a pronunciation training decoding graph according to claim 3, wherein: The step of calculating the maximum likelihood estimate of each element in the sequence based on the preset text specifically includes: When the preset text is determined, the true conditional probability of each element in the preset text appearing is calculated as the element probability, and the maximum likelihood estimate of each element when the preset text is determined is calculated based on the element probability.

5. The method for constructing a pronunciation training decoding graph according to claim 3, wherein: The calculating the first probability of a sequence inconsistent with the preset text according to the adjustment factor and the first weight parameter specifically includes: When the preset text is determined, elements inconsistent with the preset text sequence are obtained, the element probability of each inconsistent element that may appear is obtained, and the sum of all possible element probabilities is used as the first weight parameter.

6. The method for constructing a pronunciation training decoding graph according to claim 1, wherein: The step of constructing an n-gram model according to the acquired preset text sequence, the first probability and the second probability specifically includes: All elements in the preset text sequence are constructed into an initial state transition sequence, each element corresponds to a state in the initial state transition sequence, the order of transitions between different states is consistent with the order in which the elements appear in the preset text sequence, and the transition probabilities between corresponding states in the initial state transition sequence are calculated according to the second probability; A preset state is used to represent a state that does not appear in the preset text, a state transition between each state and the preset state is added to the initial state transition sequence, and a transition probability between the corresponding state and the preset state is calculated according to the first probability.

7. The method for constructing a pronunciation training decoding graph according to claim 1, wherein: The constructing of a decoding graph corresponding to the n-gram model specifically includes: Construct a grammar state machine based on the preset text and n-gram model; A corresponding decoding graph is constructed by combining one or more of a Markov model, a context-based phoneme model, a pronunciation dictionary, and a grammar state machine.

8. A pronunciation evaluation method, characterized in that: A decoding graph for pronunciation training constructed using any one of claims 1 to 7, specifically: The pronunciation to be evaluated is input into the acoustic model. Based on the output of the acoustic model and the corresponding transition probability on the decoding graph, a path with the highest transition probability is searched on the decoding graph, and the pronunciation is evaluated based on the information on the searched path.

9. The pronunciation evaluation method according to claim 8, characterized in that: The step of searching for a path with the highest transition probability on the decoding graph specifically includes: If the current pronunciation is consistent with the next element in the preset text sequence, it will transfer to the state corresponding to the next element; If the current pronunciation is inconsistent with the next element in the preset text sequence, it is transferred to the preset state, and then transferred to the state corresponding to the current pronunciation in the preset text.

10. The pronunciation evaluation method according to claim 8, characterized in that: The step of evaluating the pronunciation according to the information on the searched path specifically includes: When the state items contained in the path, and the order of the state items, are consistent with the preset text sequence, it is determined that the order of pronunciation is correct; and / or, when the path includes a preset state and an element in a preset text sequence is missing, determining that there is a skip in the pronunciation; and / or, when the path includes a preset state, and the number of elements corresponding to one or more state items in the path is greater than the number of corresponding elements in the preset text sequence, it is determined that there is repeated reading in the pronunciation; And / or, when the path includes a preset state, and the order of elements corresponding to the state items in the path is different from the order of elements in the preset text sequence, it is determined that there is out-of-order reading in the pronunciation.

11. A pronunciation evaluation device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can construct a decoding graph according to the pronunciation training decoding graph construction method described in any one of claims 1-7, and evaluate pronunciation according to the pronunciation evaluation method described in any one of claims 8-10.

Images