Pronunciation evaluation method and system and electronic equipment

Abstract

The invention provides a pronunciation evaluation method and system, and electronic equipment. The method comprises the steps of obtaining a to-be-evaluated audio and a to-be-evaluated text corresponding to the to-be-evaluated audio; obtaining a word-phoneme sequence which comprises a plurality of words, a phoneme sequence of an American phoneme sequence corresponding to each word under a unified phoneme rule, and a phoneme sequence of an English phoneme sequence corresponding to each word under the unified phoneme rule; the unified phoneme rule comprises a one-to-one correspondence relationship between British phonetic symbols and phonemes and a one-to-one correspondence relationship between American phonetic symbols and phonemes, and in the unified phoneme rule, at least one phoneme corresponds to one British phonetic symbol and also corresponds to one American phonetic symbol; constructing a decoding network based on the word-phoneme sequence and the to-be-evaluated text; acquiring an acoustic model; and outputting evaluation information based on the audio to be evaluated, the acoustic model and the decoding network. The method does not need to employ a American sound evaluation model and an English sound evaluation model for respective evaluation, and reduces the complexity of the evaluation mode.

Description

Technical Field

[0001] The embodiments of the present application relate to the technical field of pronunciation evaluation, and in particular to a pronunciation evaluation method and system, and electronic equipment. Background Art

[0002] Pronunciation assessment technology is a sub-area of ​​computer-assisted language learning. It can analyze the user's pronunciation audio and output the user's scores on indicators such as pronunciation accuracy, fluency, and completeness, so that the user can correct his or her own pronunciation problems based on the scores of various indicators.

[0003] In the case of English learning, since there are two major types of pronunciation criteria for British and American English, and the user's pronunciation preference is unknown in advance, the pronunciation evaluation system needs to be compatible with both British and American pronunciation evaluation technologies. The existing technologies for evaluating English pronunciation are divided into two categories. The first category is to specify the pronunciation criteria (American English / British English) in advance, and then use the corresponding pronunciation model for evaluation. Since this type of technology evaluates American English and British English independently, it cannot solve the situation where the user's pronunciation preference is unknown in advance; the second category is to use the British pronunciation evaluation model and the American pronunciation evaluation model to calculate the pronunciation probability under the corresponding British and American English rules for the input pronunciation audio at the same time, and select the best one as the final result with a certain strategy. That is, at present, when evaluating English pronunciation, two independent models for evaluating British English and American English are required, which makes the evaluation method more complicated. Summary of the invention

[0004] The embodiments of the present application provide a pronunciation evaluation method and system, and an electronic device, which reduce the complexity of the evaluation method.

[0005] In a first aspect, an embodiment of the present application provides a pronunciation evaluation method, the method comprising: obtaining audio to be evaluated and a text to be evaluated corresponding to the audio to be evaluated; obtaining a word-phoneme sequence, wherein the word-phoneme sequence comprises a plurality of words, a phoneme sequence of an American phonetic symbol sequence corresponding to each of the words under a unified phoneme rule, and a phoneme sequence of a British phonetic symbol sequence corresponding to each of the words under the unified phoneme rule, the unified phoneme rule comprising a one-to-one correspondence between British phonetic symbols and phonemes, and a one-to-one correspondence between American phonetic symbols and phonemes, and in the unified phoneme rule, at least one phoneme corresponds to both a British phonetic symbol and an American phonetic symbol; constructing a decoding network based on the word-phoneme sequence and the text to be evaluated; obtaining an acoustic model; and outputting evaluation information based on the audio to be evaluated, the acoustic model and the decoding network.

[0006] In some embodiments, the method for constructing the word-phoneme sequence includes: obtaining multiple words, British phonetic symbol sequences corresponding to each of the words, and American phonetic symbol sequences corresponding to each of the words; obtaining the unified phoneme rule; according to the unified phoneme rule and each of the British phonetic symbol sequences, obtaining the phoneme sequence of the British phonetic symbol sequence corresponding to each of the words under the unified phoneme rule; according to the unified phoneme rule and each of the American phonetic symbol sequences, obtaining the phoneme sequence of the American phonetic symbol sequence corresponding to each of the words under the unified phoneme rule; constructing the word-phoneme sequence based on each of the words, the phoneme sequence of the American phonetic symbol sequence corresponding to each of the words under the unified phoneme rule, and the phoneme sequence of the British phonetic symbol sequence corresponding to each of the words under the unified phoneme rule.

[0007] In some embodiments, obtaining the acoustic model includes: obtaining training audio and training text corresponding to the training audio; training a preset model based on the training audio, the training text and the word-phoneme sequence to obtain the acoustic model.

[0008] In some embodiments, the acoustic model is a deep neural network-hidden Markov model.

[0009] In some embodiments, the outputting of evaluation information based on the audio to be evaluated, the acoustic model and the decoding network includes: determining the acoustic score corresponding to each audio frame in the audio to be evaluated based on the acoustic model; determining the optimal path based on each of the acoustic scores and the decoding network; and outputting the evaluation information based on the optimal path.

[0010] In some embodiments, determining the optimal path based on each of the acoustic scores and the decoding network includes: determining the optimal path using a Viterbi algorithm based on each of the acoustic scores and the decoding network.

[0011] In some embodiments, the evaluation information includes at least one of the boundary information of each phoneme, the posterior probability information corresponding to the phoneme output at each moment, and the pronunciation mode.

[0012] In a second aspect, an embodiment of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in any one of the embodiments of the first aspect above.

[0013] In a third aspect, an embodiment of the present application provides a pronunciation evaluation system, which includes a voice collection device and the electronic device as described in the second aspect, wherein the voice collection device is connected to the electronic device.

[0014] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method described in any one of the embodiments of the first aspect above.

[0015] In a fifth aspect, an embodiment of the present application further provides a computer program product, wherein the computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer executes the method described in any one of the embodiments of the first aspect above.

[0016] Compared with the prior art, the beneficial effect of the present application is: different from the prior art, the embodiment of the present application provides a pronunciation evaluation method and system, and an electronic device, the method comprising obtaining an audio to be evaluated and a text to be evaluated corresponding to the audio to be evaluated; obtaining a word-phoneme sequence; constructing a decoding network based on the word-phoneme sequence and the text to be evaluated; obtaining an acoustic model; and outputting evaluation information based on the audio to be evaluated, the acoustic model, and the decoding network. In the method, since the British phonetic symbol sequence and the American phonetic symbol sequence of each word in the word-phoneme sequence are both represented by phonemes in a unified phoneme rule, then, in the process of evaluating the audio to be evaluated using an acoustic model and a decoding network constructed based on a word-phoneme sequence under a unified phoneme rule, only one acoustic model is needed to obtain the evaluation information, and there is no need to use an American voice evaluation model and a British voice evaluation model for evaluation, thereby reducing the complexity of the evaluation method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplified descriptions do not constitute limitations on the embodiments. Elements / modules and steps with the same reference numerals in the drawings are represented as similar elements / modules and steps. Unless otherwise specified, the figures in the drawings do not constitute proportional limitations.

[0018] Figure 1 It is a structural diagram of a pronunciation evaluation system provided in an embodiment of the present application;

[0019] Figure 2 is a structural block diagram of an electronic device provided in an embodiment of the present application;

[0020] Figure 3 It is a flowchart of a pronunciation evaluation method provided in an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a decoding network provided in an embodiment of the present application;

[0022] Figure 5 It is a schematic diagram of another decoding network provided in an embodiment of the present application. DETAILED DESCRIPTION

[0023] The present application is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present application, but do not limit the present application in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can also be made without departing from the concept of the present application. These all belong to the protection scope of the present application.

[0024] For ease of understanding of the present application, the present application is described in more detail below in conjunction with the accompanying drawings and specific embodiments. Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those generally understood by those skilled in the art of the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not used to limit the present application. The term "and / or" used in this specification includes any and all combinations of one or more related listed items.

[0025] It should be noted that, if there is no conflict, the various features in the embodiments of the present application can be combined with each other, all within the scope of protection of the present application. In addition, although the functional module division is performed in the device schematic, in some cases, the module division can be different from that in the device. In addition, the words "first", "second", etc. used herein do not limit the data and execution order, but only distinguish the same items or similar items with basically the same functions and effects.

[0026] At present, in the English pronunciation evaluation algorithm, two independent American pronunciation evaluation models and British pronunciation evaluation models are usually used to evaluate the audio to obtain the probability of American pronunciation and the probability of British pronunciation. For example, a database of British and American accents can be established first, and the data can be annotated at the phoneme level; then a multi-task acoustic model is trained, generally using a neural network model, with a shared bottom layer and an output layer with two branches, one for the phoneme classification of American accent and the other for the phoneme classification of British accent; then, according to the voice information input by the user, the text input by the user is divided into word sequences, and the British and American phoneme sequences of each word are extracted; then the probability of British / American pronunciation of each word is calculated; finally, after normalization, the probability of British / American pronunciation of each word is converted, and the probability of British / American pronunciation of the entire text is obtained. It can be seen that in the above evaluation method, American pronunciation evaluation and British pronunciation evaluation need to be performed separately, and the evaluation method is complicated, resulting in low evaluation efficiency.

[0027] In order to solve the above technical problems, the embodiments of the present application provide a method, system and electronic device for pronunciation evaluation. In this method, an acoustic model and a decoding network constructed by a word-phoneme sequence under a unified phoneme rule are used to output evaluation information. There is no need to use an American English evaluation model and a British English evaluation model for evaluation respectively, which reduces the complexity of the evaluation method and improves the evaluation efficiency.

[0028] First, see Figure 1 , Figure 1 1 is a schematic diagram of a pronunciation evaluation system provided in an embodiment of the present application. The system includes an electronic device 100 and a voice collection device 200, wherein the electronic device 100 is connected to the voice collection device 200, wherein the voice collection device 200 and the electronic device 100 can be directly or indirectly connected via wired or wireless communication, such as via a network connection, and the network can be a wide area network or a local area network, or a combination of the two, and the present application does not limit this.

[0029] The voice collection device 200 can be used to obtain training audio or audio to be evaluated, and the voice collection device 200 may include a microphone and other devices. It can record the user's audio and send the audio to the electronic device 100. The electronic device 100 obtains the training audio or the audio to be evaluated, executes the pronunciation evaluation method provided in the embodiment of the present application, and obtains corresponding evaluation information.

[0030] The electronic device 100 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.

[0031] In this embodiment, the electronic device 100 can be used to execute the pronunciation evaluation method provided in this application, and has the corresponding functional modules and beneficial effects of the execution method. For technical details not described in detail in this embodiment, please refer to the method provided in the embodiment of this application.

[0032] Second, see Figure 2 , which shows the hardware structure of an electronic device 100 capable of executing the pronunciation evaluation method described in this application. The electronic device 100 may be Figure 1 An electronic device 100 is shown.

[0033] See also Figure 2 , the electronic device 100 includes a processor 201 and a memory 202 connected in communication. Here, the communication connection can be connected via a bus, Figure 2 The bus connection is used as an example. It can be understood that Figure 2 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0034] For details, see Figure 2 , which shows the hardware structure of an electronic device 100 capable of executing the pronunciation evaluation method described in this application. The electronic device 100 may be Figure 1 An electronic device 100 is shown.

[0035] See also Figure 2 , the electronic device 100 includes a processor 101 and a memory 102 connected in communication. Here, the communication connection can be connected via a bus, Figure 2 The bus connection is used as an example. It can be understood that Figure 2 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0036] Wherein, the processor 101 is configured to support the electronic device 100 to perform the corresponding functions in the pronunciation evaluation method. The processor 101 can be a central processing unit (CPU), a network processor (NP), a hardware chip or any combination thereof. The above-mentioned hardware chip can be an application specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof. The above-mentioned PLD can be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL) or any combination thereof.

[0037] The memory 102 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the pronunciation evaluation method in the embodiment of the present application. The processor 101 can implement the pronunciation evaluation method in any of the following method embodiments by running the non-transient software programs, instructions and modules stored in the memory 102.

[0038] The memory 102 may include a volatile memory (VM), such as a random access memory (RAM); the memory 102 may also include a non-volatile memory (NVM), such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD); the memory 102 may also include a combination of the above types of memory.

[0039] It is understandable that the electronic device 100 also includes other hardware and software that support operation. The hardware may include a keyboard, a mouse, a display, etc. The software may include an operating system, etc. The operating system is a program that manages and controls the hardware and software resources of the electronic device. The software may also include various application programs (applications, apps). Here, other parts of the electronic device 100 that are not related to the improvement of the embodiment of the present application will not be described.

[0040] It can be understood from the above that the pronunciation evaluation method provided in the embodiment of the present application can be implemented by various types of electronic devices with processing capabilities, such as by a processor of an electronic device or by other devices with computing and processing capabilities. Other devices with computing and processing capabilities can be intelligent terminals or servers that are connected to the electronic device for communication.

[0041] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0042] The pronunciation evaluation method provided by the embodiment of the present application is described below in conjunction with the exemplary application and implementation of the electronic device provided by the embodiment of the present application. Figure 3 , Figure 3 : is a flowchart of the pronunciation evaluation method provided in the embodiment of the present application. It is understandable that the execution subject of the training method can be one or more processors of the electronic device. The method includes:

[0043] Step S100: Acquire the audio to be evaluated and the text to be evaluated corresponding to the audio to be evaluated.

[0044] The text to be evaluated refers to a word sequence. In the present application, the text to be evaluated is an English word sequence. The text to be evaluated can be acquired or pre-stored. For example, the text to be evaluated read by a user can be acquired in real time through an interactive application such as a language learning application.

[0045] The audio to be evaluated refers to the voice data based on the text to be evaluated for pronunciation evaluation. For example, the voice data that needs to be evaluated for spoken language can be used as the audio to be evaluated. In this application, the audio to be evaluated is in English, which can be a British-pronounced voice, an American-pronounced voice, or a mixture of British and American pronunciations. The audio to be evaluated is obtained in real time. For example, the voice data of the user reading the text to be evaluated can be obtained in real time through an interactive application such as a language learning application as the audio to be evaluated.

[0046] Step S200: Obtain a word-phoneme sequence, wherein the word-phoneme sequence includes a plurality of words, a phoneme sequence of an American phonetic symbol sequence corresponding to each word under a unified phoneme rule, and a phoneme sequence of a British phonetic symbol sequence corresponding to each word under a unified phoneme rule, wherein the unified phoneme rule includes a one-to-one correspondence between British phonetic symbols and phonemes, and a one-to-one correspondence between American phonetic symbols and phonemes, and in the unified phoneme rule, at least one phoneme corresponds to both a British phonetic symbol and an American phonetic symbol.

[0047] A phoneme sequence contains multiple phonemes arranged in order. A phoneme refers to the smallest speech unit divided according to the natural properties of speech. Based on the analysis of pronunciation movements, the pronunciation of a word can be composed of one or more phonemes.

[0048] In the unified phoneme rule, one British phonetic symbol corresponds to one phoneme, one American phonetic symbol corresponds to one phoneme, and there is at least one phoneme that corresponds to both one British phonetic symbol and one American phonetic symbol. The unified phoneme rule may be a correspondence table between a phonetic symbol and a phoneme, the correspondence table comprising a plurality of phonemes, a correspondence between each phoneme and the British phonetic symbol, and a correspondence between each phoneme and the American phonetic symbol, wherein the total number of phonemes is less than the sum of the total number of British phonetic symbols and the total number of American phonetic symbols.

[0049] Specifically, for American and British phonetic symbols, the markings of consonant phonetic symbols are consistent. Therefore, in the unified phoneme rule, the phonemes corresponding to consonant phonetic symbols can be directly represented by American or British phonetic symbols, that is, there are 24 phonemes corresponding to consonant phonetic symbols. The phonemes corresponding to vowel phonetic symbols can be partially represented by unified symbols, as shown in Table 1 below.

[0050] Table 1. Correspondence between phonemes, American phonetic symbols and British phonetic symbols

[0051]

[0052]

[0053] As can be seen from Table 1, there are 23 phonemes corresponding to vowel phonetic symbols. Therefore, in the unified phoneme rule, the total number of phonemes is 47, which is less than the sum of the total number of British phonetic symbols and the total number of American phonetic symbols. After obtaining the above unified phoneme rule, the American phonetic symbol sequence and the British phonetic symbol sequence of the word can be converted to obtain the phoneme sequence under the unified phoneme rule. In practical applications, the phoneme symbols in the unified phoneme rule can be formulated according to actual needs, and there is no need to stick to the limitations in this embodiment.

[0054] In some embodiments, the method for constructing a word-phoneme sequence includes step S211: obtaining a plurality of words, a British phonetic symbol sequence corresponding to each word, and an American phonetic symbol sequence corresponding to each word. Step S212: obtaining a unified phoneme rule. Step S213: obtaining a phoneme sequence of the British phonetic symbol sequence corresponding to each word under the unified phoneme rule according to the unified phoneme rule and each British phonetic symbol sequence. Step S214: obtaining a phoneme sequence of the American phonetic symbol sequence corresponding to each word under the unified phoneme rule according to the unified phoneme rule and each American phonetic symbol sequence. Step S215: constructing a word-phoneme sequence based on each word, the phoneme sequence of the American phonetic symbol sequence corresponding to each word under the unified phoneme rule, and the phoneme sequence of the British phonetic symbol sequence corresponding to each word under the unified phoneme rule.

[0055] Among them, the electronic device can obtain multiple words and the American phonetic symbol sequence corresponding to each word by obtaining the American word-phoneme sequence, and obtain multiple words and the American phonetic symbol sequence corresponding to each word by obtaining the British word-phoneme sequence. After obtaining the American phonetic symbol sequence and the British phonetic symbol sequence, data integration can be performed to obtain a correspondence table between each word, the American phonetic symbol sequence and the British phonetic symbol sequence, as shown in Table 2 below.

[0056] Table 2 Correspondence between words, American phonetic symbol sequences and British phonetic symbol sequences

[0057]

[0058] Next, a unified phoneme rule is obtained. For example, after obtaining a correspondence table between phonetic symbols and phonemes, the American phonetic symbol sequence and the British phonetic symbol sequence of each word are represented by the phonemes in the unified phoneme rule to obtain the phoneme sequence corresponding to each word. In order to distinguish between American pronunciation and British pronunciation, a distinguishing symbol can be added before the phoneme sequence, such as using distinguishing symbols "0", "1", and "2" to distinguish between American pronunciation and British pronunciation, where "0" indicates that the British pronunciation is the same as the American pronunciation, "1" indicates that the British pronunciation is unique, and "2" indicates that the American pronunciation is unique. For example, after conversion, the above Table 2 can obtain the following Table 3, that is, the word-phoneme sequence.

[0059] Table 3 Word-phoneme sequence

[0060]

[0061] It can be seen that the word-phoneme sequence is a set of phoneme sequences including each word, the American phonetic symbol sequence corresponding to each word under the unified phoneme rule, and the British phonetic symbol sequence corresponding to each word under the unified phoneme rule, and after the above steps, the electronic device can construct the word-phoneme sequence.

[0062] Step S300: construct a decoding network based on the word-phoneme sequence and the text to be evaluated.

[0063] The decoding network can be called a search graph, representing the space of all possible languages.

[0064] Specifically, if the text to be evaluated contains "teacher", the unified phoneme identifier corresponding to the text to be evaluated "teacher" can be obtained based on Table 3, and the following is constructed: Figure 4 The decoding network shown in the figure, where, in the expression of the form "x:y", the character "x" on the left side of the character ":" represents the input phoneme of the node, the character "y" on the right side of the character ":" represents the output character of the node, sil represents a silent phoneme, SIL represents a silent character, and eps represents a meaningless character. For example, for the path from node 4 to node 6, it indicates that when the input phoneme of node 4 is "er", the output character of node 4 is "teacher_us".

[0065] In addition, in the decoding network, pronunciation indicator symbols can be added in the British pronunciation path and the American pronunciation path, such as Figure 4 The characters "_us" are used to indicate the American pronunciation path, and the characters "_uk" are used to indicate the British pronunciation path. In actual applications, the pronunciation indicator symbols can be set according to actual needs, and there is no need to stick to the limitations in this embodiment. Figure 4 As shown, it includes two British pronunciation paths and one American pronunciation path, wherein the British pronunciation paths are 0-1-2-3-4-7-8-9 and 0-1-2-3-4-5-9, and the American pronunciation path is 0-1-2-3-4-6.

[0066] It should be noted that Figure 4 The decoding network shown is a decoding diagram at the phoneme level. During actual decoding, since the acoustic model outputs the acoustic score corresponding to the phoneme state, the decoding network needs to be expanded to a decoding diagram at the state level. Usually, a phoneme can correspond to at least three acoustic states. For example, a phoneme can correspond to three acoustic states, and the three acoustic states correspond to the beginning, middle, and end of the pronunciation. For example, for the path from node 4 to node 6, the decoding diagram at the state level is as follows: Figure 5 As shown, Figure 5 Node 0 in corresponds to Figure 4 Node 4 in Figure 5 Node 3 in corresponds to Figure 5 Node 6 in Figure 5 In , node 0 has one candidate input state, one candidate directed path, and one candidate output character. Node 1 and node 2 are both set with two candidate input states, two candidate directed paths, and two candidate output characters. One candidate input state corresponds to one candidate directed path and one candidate output character. Figure 5 For node 1 in , its candidate input states are er_0 and er_1, the candidate directed path is node 1-node 1, and its corresponding candidate output character is eps. Another candidate directed path is node 1-node 2, and its corresponding candidate output character is eps.

[0067] In this embodiment, by constructing a decoding network, the target state corresponding to each audio frame of the audio data can be analyzed frame by frame, and the pronunciation information in the audio data can be determined. In addition, by adding pronunciation indicator symbols to the British and American paths when constructing the decoding network, it can be obtained after subsequent decoding whether the user's pronunciation method tends to be American pronunciation or British pronunciation.

[0068] Step S400: Acquire an acoustic model.

[0069] The acoustic model is an artificial intelligence model used for acoustic recognition, which can be obtained by machine learning training in advance. The acoustic model can output the acoustic score corresponding to the audio to be evaluated. The acoustic model can be a model based on HMM (Hidden Markov Model, Hidden Markov Model), for example, Gaussian Mixture Model-Hidden Markov Model (Gaussian Mixture Model-Hidden Markov Model, GMM-HMM) model or Deep Neural Network-Hidden Markov Model (Deep Neural Network-Hidden Markov Model, DNN-HMM) model, etc.

[0070] The acoustic model can be modeled with a single phoneme as a unit. However, the pronunciation of a phoneme usually varies depending on the phonemes before and after it, that is, it has contextual relevance, so the acoustic model can also be modeled with triphones as the modeling unit. Triphones are composed of three phonemes. For example, a-b+c represents the pronunciation of phoneme b, the preceding phoneme is a, and the following phoneme is c. In addition, a phoneme uses at least 3 acoustic states, such as the acoustic states corresponding to a phoneme at the beginning, pronunciation, and end stages. Then, the final number of acoustic state classifications is at least the number of phonemes*3. It can be understood that the larger the number of classifications, the higher the computational complexity of the model is generally.

[0071] Preferably, DNN-HMM can be used as the acoustic model, which has better phoneme state classification accuracy than GMM-HMM, improving the evaluation accuracy. The acoustic model can be pre-stored in a database, so that the electronic device can obtain the acoustic model from the database.

[0072] Step S500: outputting evaluation information based on the audio to be evaluated, the acoustic model and the decoding network.

[0073] Specifically, an acoustic model may be used to determine the acoustic score corresponding to each frame of the audio to be evaluated, and then an optimal path may be determined based on a decoding network, and finally evaluation information may be obtained based on the optimal path.

[0074] The evaluation information may include at least one of the boundary information of each phoneme, the posterior probability information corresponding to the phonemes output at each time, and the pronunciation mode. The boundary information of each phoneme is the start frame and the end frame corresponding to each phoneme, and the posterior probability information corresponding to the phonemes output at each time may be the posterior probability values ​​corresponding to each phoneme corresponding to the same time, and the maximum value of all posterior probabilities at the same time. The pronunciation mode is British pronunciation or American pronunciation. After obtaining the above evaluation information, it can be used as an input of the subsequent scoring system to obtain a score, or it can be evaluated based on a variety of preset evaluation indicators to obtain corresponding evaluation results to show to the user. The preset evaluation indicators may include at least one of the indicators such as pronunciation accuracy, sentence completeness, fluency and stress position accuracy, and the evaluation result may be at least one evaluation result of pronunciation accuracy, sentence completeness, fluency and stress position accuracy.

[0075] In this embodiment, since the British phonetic symbol sequence and the American phonetic symbol sequence of each word in the word-phoneme sequence are both represented by phonemes in the unified phoneme rule, then, in the process of evaluating the audio to be evaluated by using an acoustic model and a decoding network constructed based on the word-phoneme sequence under the unified phoneme rule, only one acoustic model is needed to obtain the evaluation information, and there is no need to use the American and British evaluation models for evaluation respectively, thereby reducing the complexity of the evaluation method.

[0076] In some embodiments, obtaining an acoustic model includes:

[0077] Step S410: Obtain training audio and training text corresponding to the training audio.

[0078] The training text refers to a word sequence. In the present application, the training text is an English word sequence. The training text may be acquired or pre-stored, for example, the training text read by the user may be acquired in real time through an interactive application such as a language learning application.

[0079] Training audio refers to speech data based on training text to be evaluated for pronunciation. For example, speech data that needs to be evaluated for oral speech can be used as training audio. In the present application, the training audio is in English, and can be speech with British pronunciation, speech with American pronunciation, or speech with a mixture of British pronunciation and American pronunciation. The training audio can be acquired in real time or stored in advance. For example, speech data of a user reading a training text can be acquired in real time through an interactive application such as a language learning application as training audio.

[0080] Step S420: Based on the training audio, training text and word-phoneme sequence, the preset model is trained to obtain an acoustic model.

[0081] Specifically, the training audio, training text and word-phoneme sequence can be used to train the preset model to obtain DNN-HMM. It should be noted that the distinguishing symbols in the word-phoneme sequence do not need to be input into the preset model, that is, they do not need to participate in the model training. The specific training process can refer to the prior art and is not limited here.

[0082] In this embodiment, when the acoustic model is not pre-stored, the acoustic model can be obtained by training the training audio, training text and word-phoneme sequence.

[0083] In some embodiments, step S500 includes:

[0084] Step S510: Determine the acoustic score corresponding to each audio frame in the audio to be evaluated based on the acoustic model.

[0085] Specifically, the audio to be tested is input into the acoustic model, and the acoustic model can obtain the corresponding acoustic score for each frame of audio based on the audio features of the audio to be tested.

[0086] Step S520: Determine the optimal path based on the acoustic scores and the decoding network.

[0087] Next, in the constructed decoding network, starting from the starting point 0, the Viterbi algorithm is used to search for the optimal path frame by frame until all audio frame information is consumed. Each forward search consumes the information of one audio frame. For example, at the 20th frame, there are 2 candidate nodes, namely Figure 5Node 1 and node 2 in the audio file, then, for candidate node 1, its forward radiating path is node 1-node 1, node 1-node 2, and for candidate node 2, its forward radiating path is node 2-node 2, node 2-node 3, that is, there are 4 candidate paths in total, and the above 4 candidate paths are traversed, and the cumulative probability or cumulative cost on each candidate path is calculated respectively. The node on the path with the maximum probability or the minimum cumulative cost is taken as the best node corresponding to the 20th frame, and a one-step search is completed, that is, the information of 1 audio frame is consumed. In this way, after all audio frames are consumed, the optimal path is obtained by connecting the best nodes corresponding to each audio frame. Usually, the sum of the transfer weight cost and the acoustic cost can be used as the cumulative cost on the candidate path.

[0088] Step S530: output evaluation information based on the optimal path.

[0089] Based on the optimal path, at least one of the boundary information of each phoneme, the posterior probability information corresponding to the phonemes output at each moment, and the pronunciation mode can be obtained.

[0090] It can be seen that based on the above method, the evaluation information can be determined. In addition, the Viterbi algorithm is used to gradually calculate the best node corresponding to each audio frame, avoiding exhaustive enumeration of all possible results. This efficient calculation method can improve the efficiency of searching for the optimal path while ensuring accuracy.

[0091] The present application embodiment also provides a non-volatile computer-readable storage medium, the computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are executed by one or more processors, for example, to execute the pronunciation evaluation method described in the above embodiment. The computer-readable storage medium can be a storage medium that can be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; it can also be various devices including one of the above memories or any combination. The executable instructions can be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted language, or declarative or procedural language), and it can be deployed in any form, including being deployed as an independent program or being deployed as a module, component, subroutine or other unit suitable for use in a computing environment.

[0092] The embodiment of the present application also provides a computer program product, including a computer program stored on a non-volatile computer-readable storage medium, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the pronunciation evaluation method described in the above embodiment.

[0093] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0094] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions to use at least one computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiment.

[0095] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Under the concept of the present application, the technical features in the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other changes in different aspects of the present application as described above, which are not provided in detail for the sake of simplicity. Although the present application has been described in detail with reference to the aforementioned embodiments, a person of ordinary skill in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A pronunciation evaluation method, characterized in that: include: Obtaining an audio to be evaluated and a text to be evaluated corresponding to the audio to be evaluated; Acquire a word-phoneme sequence, wherein the word-phoneme sequence includes a plurality of words, a phoneme sequence of an American phonetic symbol sequence corresponding to each of the words under a unified phoneme rule, and a phoneme sequence of a British phonetic symbol sequence corresponding to each of the words under the unified phoneme rule, wherein the unified phoneme rule includes a one-to-one correspondence between British phonetic symbols and phonemes, and a one-to-one correspondence between American phonetic symbols and phonemes, and in the unified phoneme rule, at least one phoneme corresponds to both a British phonetic symbol and an American phonetic symbol; Constructing a decoding network based on the word-phoneme sequence and the text to be evaluated; Obtaining an acoustic model; Output evaluation information based on the audio to be evaluated, the acoustic model and the decoding network.

2. The pronunciation evaluation method according to claim 1, characterized in that: The method for constructing the word-phoneme sequence comprises: Obtain a plurality of words, a British phonetic symbol sequence corresponding to each of the words, and an American phonetic symbol sequence corresponding to each of the words; Acquiring the unified phoneme rule; According to the unified phoneme rule and each of the British phonetic symbol sequences, obtaining the phoneme sequence of the British phonetic symbol sequence corresponding to each of the words under the unified phoneme rule; According to the unified phoneme rule and each of the American phonetic symbol sequences, obtaining the phoneme sequence of the American phonetic symbol sequence corresponding to each of the words under the unified phoneme rule; The word-phoneme sequence is constructed based on the words, the phoneme sequence of the American phonetic symbol sequence corresponding to each word under the unified phoneme rule, and the phoneme sequence of the British phonetic symbol sequence corresponding to each word under the unified phoneme rule.

3. The pronunciation evaluation method according to claim 2, characterized in that: The obtaining of the acoustic model comprises: Obtaining training audio and training text corresponding to the training audio; Based on the training audio, the training text and the word-phoneme sequence, a preset model is trained to obtain the acoustic model.

4. The pronunciation evaluation method according to claim 3, characterized in that: The acoustic model is a deep neural network-hidden Markov model.

5. The pronunciation evaluation method according to any one of claims 1 to 4, characterized in that: The outputting evaluation information based on the audio to be evaluated, the acoustic model and the decoding network includes: Determining, based on the acoustic model, an acoustic score corresponding to each audio frame in the audio to be evaluated; Determining an optimal path based on each of the acoustic scores and the decoding network; Based on the optimal path, the evaluation information is output.

6. The pronunciation evaluation method according to claim 5, characterized in that: The determining of the optimal path based on each of the acoustic scores and the decoding network comprises: Based on the acoustic scores and the decoding network, the optimal path is determined using a Viterbi algorithm.

7. The pronunciation evaluation method according to any one of claims 1 to 4, characterized in that: The evaluation information includes at least one of the boundary information of each phoneme, the posterior probability information corresponding to the phonemes output at each moment, and the pronunciation mode.

8. An electronic device, characterized in that: include: at least one processor; as well as, a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

9. A pronunciation evaluation system, characterized in that: It comprises a voice collection device and the electronic device as claimed in claim 8; The voice collection device is connected to the electronic device.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the method according to any one of claims 1 to 7.

Images