Oral evaluation method, device, equipment and storage medium

Abstract

A method and device for oral evaluation, equipment and storage medium, the method comprising: obtaining speech data and reference text to be evaluated; constructing a first decoding network based on the reference text; performing a first decoding process on the speech data based on an acoustic model and the first decoding network to obtain time boundary information corresponding to each reference phoneme in the reference text; obtaining a phoneme feature corresponding to each reference phoneme based on the time boundary information; performing a first scoring process on the speech data based on the phoneme feature and a scoring model to obtain a first scoring result; traversing each reference phoneme in the reference text to obtain a competing phoneme corresponding to the current reference phoneme; obtaining the output of the acoustic model within the corresponding time range based on the time boundary information; obtaining the user's actual pronunciation phoneme based on the competing phoneme and the output of the acoustic model. The embodiment of the present invention integrates pronunciation scoring and phoneme error detection, which is conducive to improving the effect of oral evaluation.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of speech evaluation, and in particular to a spoken language evaluation method and apparatus, device and storage medium. Background Art

[0002] With the development of society, more and more people are learning multiple languages. When learning a language, people usually use language learning software to assist learning, such as using language learning software to conduct oral assessments, score their pronunciation, and correct their pronunciation based on the scoring results.

[0003] Oral assessment refers to the use of speech recognition technology to evaluate a person's oral proficiency overall given text and audio. For example, in the field of oral assessment, when scoring pronunciation, the phoneme information after text-speech is usually used to calculate the goodness of pronunciation of each word and each phoneme in the text, and then judge the speaker's oral proficiency based on these features.

[0004] However, the effectiveness of oral assessment still needs to be improved. Summary of the invention

[0005] The problem solved by the embodiments of the present invention is to provide a spoken language evaluation method and apparatus, a device and a storage medium, which integrate pronunciation scoring and phoneme error detection, thereby improving the effect of spoken language evaluation.

[0006] To solve the above problems, an embodiment of the present invention provides a spoken language evaluation method, comprising: obtaining speech data to be evaluated and a reference text of the speech data; constructing a first decoding network based on the reference text; performing a first decoding process on the speech data based on an acoustic model and the first decoding network to obtain time boundary information corresponding to each reference phoneme in the reference text; obtaining a phoneme feature corresponding to each reference phoneme based on the time boundary information; performing a first scoring process on the speech data based on the phoneme feature and a scoring model to obtain a first scoring result; traversing each reference phoneme in the reference text to obtain a competing phoneme corresponding to the current reference phoneme; obtaining the output of the acoustic model within the corresponding time range based on the time boundary information; and obtaining the user's actual pronunciation phoneme based on the competing phonemes and the output of the acoustic model.

[0007] Optionally, the phoneme features include any one or more of duration information, a pronunciation goodness value, and a generalized posterior probability.

[0008] Optionally, the generalized posterior probability is calculated by the following formula:

[0009]

[0010] Among them, m i is the current reference phoneme, b i is the starting frame of the current reference phoneme, e i is the end frame of the current reference phoneme, It is the model probability score of the speech segment corresponding to the current reference phoneme.

[0011] Optionally, the spoken language evaluation method further includes: using the phoneme features to train the scoring model.

[0012] Optionally, based on the competitive phonemes and the output of the acoustic model, the step of obtaining the user's actual pronounced phonemes includes: traversing each reference phoneme in the reference text, and constructing a second decoding network of competitive phonemes corresponding to the current reference phoneme; based on the second decoding network and the output of the acoustic model, obtaining the user's actual pronounced phonemes.

[0013] Optionally, constructing a second decoding network for competing phonemes corresponding to the current reference phoneme includes: acquiring one or more competing phonemes corresponding to the current reference phoneme; and using a network structure constructed in parallel with the current reference phoneme and the one or more competing phonemes as the second decoding network.

[0014] Optionally, based on the second decoding network and the output of the acoustic model, the step of obtaining the user's actual pronunciation phonemes includes: based on the second decoding network, performing a second decoding process on the output of the acoustic model to obtain N decoding paths with the highest confidence, each decoding path including any one of the competing phonemes and the current reference phonemes, and taking the phonemes corresponding to the Top1 decoding path as the user's actual pronunciation phonemes.

[0015] Optionally, after performing a second decoding process on the output of the acoustic model, the oral evaluation method further includes: obtaining the phoneme features corresponding to the current reference phoneme and the phoneme features corresponding to the competing phonemes based on the time boundary information; performing a second scoring process on the output of the acoustic model based on the phoneme features corresponding to the current reference phoneme, the phoneme features corresponding to the competing phonemes, and the scoring model to obtain a second scoring result corresponding to the current reference phoneme and each competing phoneme in the second decoding network.

[0016] Optionally, based on the competing phonemes and the output of the acoustic model, the step of obtaining the user's actual pronunciation phonemes includes: based on the current reference phonemes and the corresponding competing phonemes, utilizing a deep learning network to perform phoneme classification processing on the output of the acoustic model, outputting the probability corresponding to the current reference phoneme and the probability corresponding to each competing phoneme, and taking the competing phoneme or the current reference phoneme with the highest probability as the user's actual pronunciation phoneme.

[0017] Optionally, after using a deep learning network to perform phoneme recognition and classification processing on the output of the acoustic model, the oral evaluation method also includes: obtaining a second scoring result corresponding to the current reference phoneme and each competing phoneme based on the probability corresponding to the current reference phoneme and the probability corresponding to each competing phoneme.

[0018] Correspondingly, an embodiment of the present invention also provides a spoken language evaluation device, including: an audio text acquisition module, used to acquire speech data to be evaluated and a reference text of the speech data; a first network construction module, used to construct a first decoding network based on the reference text; a decoding module, used to perform a first decoding process on the speech data based on an acoustic model and the first decoding network, and obtain time boundary information corresponding to each reference phoneme in the reference text; a feature extraction module, used to obtain a phoneme feature corresponding to each reference phoneme based on the time boundary information; a scoring module, used to perform a first scoring process on the speech data based on the phoneme feature and the scoring model, and obtain a first scoring result; a competitive phoneme acquisition module, used to traverse each reference phoneme in the reference text, and obtain a competitive phoneme corresponding to the current reference phoneme; an acoustic model output acquisition module, used to obtain the output of the acoustic model within the corresponding time range based on the time boundary information; an actual phoneme acquisition module, used to obtain the user's actual pronunciation phoneme based on the competitive phoneme and the output of the acoustic model.

[0019] Correspondingly, an embodiment of the present invention further provides a device, comprising at least one memory and at least one processor, wherein the memory stores one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the oral assessment method provided by the embodiment of the present invention.

[0020] Correspondingly, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the oral language evaluation method provided by the embodiment of the present invention.

[0021] Compared with the prior art, the technical solution of the embodiment of the present invention has the following advantages:

[0022] The oral evaluation method provided by the embodiment of the present invention performs a first scoring process on the speech data based on the phoneme feature and the scoring model to obtain a first scoring result; further, based on the time boundary information, the output of the acoustic model within the corresponding time range is obtained; based on the competitive phonemes and the output of the acoustic model, the user's actual pronunciation phonemes are obtained, so that while the pronunciation of the user's spoken language is scored, the phoneme error detection of the user's spoken pronunciation is also achieved, which correspondingly improves the effect and comprehensiveness of the oral evaluation; further, the same acoustic model is used for phoneme error detection and pronunciation scoring, which is conducive to reducing the complexity and amount of calculation of the oral evaluation, correspondingly improving the response speed and reducing the real time factor (RTF); in addition, based on the time boundary information, the output of the acoustic model within the corresponding time range is obtained, and then based on the competitive phonemes and the output of the acoustic model, the user's actual pronunciation phonemes are obtained, thereby limiting the recognition type of the phoneme and the recognition time range, which is not only conducive to improving the accuracy of phoneme error detection, but also does not require additional training data for phoneme-level recognition, which is conducive to reducing costs.

[0023] In the oral evaluation device provided by the embodiment of the present invention, the scoring module performs a first scoring process on the speech data based on the phoneme feature and the scoring model to obtain a first scoring result; furthermore, the competitive phoneme acquisition module traverses each reference phoneme in the reference text to obtain the competitive phoneme corresponding to the current reference phoneme, and the actual phoneme acquisition module is used to obtain the user's actual pronunciation phoneme based on the competitive phoneme and the output of the acoustic model, so that while the pronunciation of the user's spoken language is scored, the phoneme error detection of the user's spoken pronunciation is also realized, which correspondingly improves the effect and comprehensiveness of the oral evaluation; furthermore, the same acoustic model is used for phoneme error detection and pronunciation scoring, which is conducive to reducing the complexity and amount of calculation of the oral evaluation device, correspondingly improving the response speed and reducing the real time factor (RTF); in addition, the actual phoneme acquisition module obtains the user's actual pronunciation phoneme based on the competitive phoneme and the output of the acoustic model, thereby limiting the recognition type and recognition time range of the phoneme, which is not only conducive to improving the accuracy of phoneme error detection, but also does not need to obtain additional training data for phoneme-level recognition, which is conducive to reducing costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of an embodiment of a spoken language evaluation method of the present invention;

[0025] Figure 2 yes Figure 1 A schematic diagram of the process of an embodiment of step S8;

[0026] Figure 3 is a functional block diagram of an embodiment of a spoken language evaluation device of the present invention;

[0027] Figure 4 yes Figure 3 Functional block diagram of an embodiment of an actual phoneme acquisition module in FIG.

[0028] Figure 5 It is a hardware structure diagram of a device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] As can be seen from the background technology, the effect of oral evaluation still needs to be improved.

[0030] Specifically, currently, in the same oral evaluation device, it is not possible to simultaneously implement pronunciation scoring and phoneme error detection. That is to say, when scoring the user's pronunciation, it is not possible to identify the specific errors in the user's pronunciation, or, when error detection is performed on the user's pronunciation, it is not possible to simultaneously score the pronunciation, resulting in an incomplete oral evaluation.

[0031] However, if two acoustic models are used at the same time to perform pronunciation scoring and phoneme error detection respectively, the system will easily become more complex and difficult to meet the needs of rapid online response.

[0032] In order to solve the above technical problems, an embodiment of the present invention provides a spoken language evaluation method. Figure 1 , which shows a flow chart of an embodiment of a spoken language evaluation method of the present invention.

[0033] In this embodiment, the oral evaluation method includes the following basic steps:

[0034] Step S1: Acquire speech data to be evaluated and a reference text of the speech data;

[0035] Step S2: constructing a first decoding network based on the reference text;

[0036] Step S3: performing a first decoding process on the speech data based on the acoustic model and the first decoding network to obtain time boundary information corresponding to each reference phoneme in the reference text;

[0037] Step S4: obtaining a phoneme feature corresponding to each reference phoneme based on the time boundary information;

[0038] Step S5: Based on the phoneme features and the scoring model, perform a first scoring process on the speech data to obtain a first scoring result;

[0039] Step S6: traverse each reference phoneme in the reference text to obtain a competing phoneme corresponding to the current reference phoneme;

[0040] Step S7: based on the time boundary information, obtaining the output of the acoustic model within the corresponding time range;

[0041] Step S8: Based on the competing phonemes and the output of the acoustic model, the user's actual pronunciation phonemes are obtained.

[0042] The oral evaluation method provided by the embodiment of the present invention performs a first scoring process on the speech data based on the phoneme feature and the scoring model to obtain a first scoring result; further, based on the time boundary information, the output of the acoustic model within the corresponding time range is obtained; based on the competitive phonemes and the output of the acoustic model, the user's actual pronunciation phonemes are obtained, so that while the pronunciation of the user's spoken language is scored, the phoneme error detection of the user's spoken pronunciation is also achieved, which correspondingly improves the effect and comprehensiveness of the oral evaluation; further, the same acoustic model is used for phoneme error detection and pronunciation scoring, which is conducive to reducing the complexity and amount of calculation of the oral evaluation, correspondingly improving the response speed and reducing the real time factor (RTF); in addition, based on the time boundary information, the output of the acoustic model within the corresponding time range is obtained, and then based on the competitive phonemes and the output of the acoustic model, the user's actual pronunciation phonemes are obtained, thereby limiting the recognition type of the phoneme and the recognition time range, which is not only conducive to improving the accuracy of phoneme error detection, but also does not require additional training data for phoneme-level recognition, which is conducive to reducing costs.

[0043] In order to make the above-mentioned purposes, features and advantages of the embodiments of the present invention more obvious and understandable, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0044] refer to Figure 1 , Step S1: Acquire speech data to be evaluated and a reference text of the speech data.

[0045] The speech data is used as an object of oral evaluation.

[0046] Specifically, the voice data may be voice stream data input in real time, or file stream data in an audio file. The voice data may be data directly received and processed, or may be audio data acquired from other terminals.

[0047] As an example, voice stream data can be obtained by recording voice in real time through hardware with real-time recording function such as microphone and sound card. File stream data can be obtained by reading audio files storing recorded audio data. The format of the audio file can be arbitrary, including but not limited to .WAV, .MP3, etc.

[0048] The reference text is used as a reference benchmark for subsequent evaluation of the voice data. The reference text is also used to obtain corresponding reference phonemes.

[0049] Specifically, in the field of oral evaluation, a spoken text (or spoken voice, etc.) is usually provided to the user, who then follows the spoken text (or spoken voice, etc.) and makes corresponding sounds, and collects voice data corresponding to the sounds as the voice data to be evaluated, thereby achieving oral evaluation; wherein the spoken text, etc. provided to the user is the reference text.

[0050] As an example, the oral evaluation method is applied to oral evaluation of English, and the reference text is an English text. In other embodiments, the oral evaluation method can also be used for oral evaluation of other languages, and the reference text is a text of the corresponding language.

[0051] Continue to refer Figure 1 , Step S2: Based on the reference text, construct a first decoding network.

[0052] The first decoding network is used to provide a decoding path for subsequently performing a first decoding process on the voice data.

[0053] In this embodiment, the first decoding network is a decoding network based on WFST (weighted finite-state transducer).

[0054] Specifically, in this embodiment, since the reference text is already given, the first decoding network generally only includes one decoding path. Accordingly, the first decoding network is generally a serial network structure constructed based on the reference text.

[0055] In this embodiment, the first decoding network includes a word-level decoding network. Taking the reference text "how are you" as an example, the network structure of the corresponding first decoding network is: how->are->you.

[0056] Continue to refer Figure 1 , step S3: based on the acoustic model and the first decoding network, perform a first decoding process on the speech data to obtain time boundary information corresponding to each reference phoneme in the reference text.

[0057] The time boundary information corresponding to each reference phoneme is obtained so that the phoneme feature corresponding to each reference phoneme can be obtained according to the time boundary information. In addition, the time boundary information is also used to obtain the speech segment corresponding to the current reference phoneme in the speech data.

[0058] In the field of speech recognition, the output of the decoding process is usually the optimal path and the boundary of each node in the path. Accordingly, in this embodiment, since the path of the first decoding network has been determined by the reference text, the time boundary information corresponding to each reference phoneme in the reference text is obtained by performing the first decoding process.

[0059] As an embodiment, the time boundary information corresponding to the current reference phoneme includes: a start frame and an end frame corresponding to the current reference phoneme in the speech data.

[0060] Specifically, the first decoding process obtains the time boundary information corresponding to the reference phoneme by using the Force Alignment (FA) method. Specifically, based on the acoustic model, the corresponding phoneme sequence is searched through the pronunciation dictionary, and the input voice data is aligned with the reference text in the time dimension to obtain the time boundary information corresponding to each reference phoneme.

[0061] Taking the reference text "how are you" as an example, the sequence of reference phonemes of the reference text found through the pronunciation dictionary is: rju, perform a first decoding process on the voice data to obtain each reference phoneme (h rj u) corresponding to the time boundary information.

[0062] It should be noted that, before performing the first decoding process on the speech data, the spoken language evaluation method generally further includes: inputting the speech data into the acoustic model to obtain an output of the acoustic model.

[0063] Therefore, based on the output of the acoustic model and the first decoding network, the speech data is subjected to a first decoding process.

[0064] In a specific implementation, the output of the acoustic model includes the log likelihood of each frame on each phoneme.

[0065] Among them, the acoustic model is an acoustic model trained according to the user's standard voice, including but not limited to the traditional HMM (Hidden Markov Model) model and mainstream neural network models, such as DNN (Deep Neural Networks) model and LSTM (Long Short Term Memory) model.

[0066] In a specific implementation, in addition to the first decoding network and the acoustic model, a pronunciation dictionary is also used to perform a first decoding process on the speech data.

[0067] The acoustic model can be used to establish a mapping between speech features and phonemes in speech data; the pronunciation dictionary can be used to establish a mapping between phonemes and words, thereby obtaining reference phonemes corresponding to the reference text. According to the acoustic model, the pronunciation dictionary, and the mapping established by the first decoding network, the speech data is subjected to a first decoding process to obtain time boundary information corresponding to each reference phoneme in the reference text.

[0068] Specifically, the acoustic model is a knowledge representation of differences in acoustics, phonetics, environmental variables, speaker gender, accent, etc.; the pronunciation dictionary is a set of phoneme indexes corresponding to words.

[0069] Continue to refer Figure 1 , Step S4: Based on the time boundary information, obtain the phoneme feature corresponding to each reference phoneme.

[0070] The phoneme features are used as input for scoring speech data, so as to serve as evaluation features for oral evaluation.

[0071] In this embodiment, the phoneme features include any one or more of duration information, Goodness of Pronunciation (GOP) and Generalized Word Posterior Probability (GWPP).

[0072] As an embodiment, the GOP is calculated based on formula (I):

[0073]

[0074] Where t is time, It is the phoneme likelihood probability output by the acoustic model.

[0075] In other embodiments, other methods or algorithms may be used to calculate the GOP.

[0076] It should be noted that since the GOP algorithm performs normalization processing on the time dimension, if a single GOP dimension is used as a phoneme feature to score speech data, the impact of the time dimension on the scoring result cannot be reflected.

[0077] Therefore, in this embodiment, by making the phoneme feature include duration information, the time dimension of the phoneme is introduced as a scoring dimension for scoring the speech data. Accordingly, the pronunciation duration of the phoneme can affect the scoring result obtained by scoring the speech data, thereby improving the accuracy of the speech data scoring and further improving the accuracy and comprehensiveness of the pronunciation scoring.

[0078] For example, for long vowels and short vowels in English, the pronunciation duration of long vowels is longer than that of short vowels. By introducing the duration information of phonemes as a scoring dimension for scoring speech data, the accuracy of pronunciation scoring can be further improved accordingly.

[0079] GOP reflects the average of the difference between the probability of the reference phoneme in each frame and the maximum probability output by the acoustic model in the current frame at the frame level. Different from GOP, the GWPP characterizes the difference between the probability of the reference phoneme and other phonemes within the time boundary, and introduces a comparison with other phonemes, which can characterize whether the pronunciation is good enough from another dimension. Therefore, by introducing GWPP as a dimension of pronunciation scoring, it is helpful to improve the robustness of oral evaluation.

[0080] In this embodiment, the GWPP is calculated by formula (II):

[0081]

[0082] Among them, m i is the current reference phoneme, b i is the starting frame of the current reference phoneme, e i is the end frame of the current reference phoneme, It is the model probability score of the speech segment corresponding to the current reference phoneme.

[0083] In this embodiment, by making the phoneme feature include one or more of duration information, GOP and GWPP, the robustness of the scoring process for speech data can be improved.

[0084] Continue to refer Figure 1 , step S5: based on the phoneme features and the scoring model, perform a first scoring process on the speech data to obtain a first scoring result.

[0085] The speech data is subjected to a first scoring process, thereby performing a pronunciation score on the speech data.

[0086] It should be noted that, in this embodiment, the spoken language evaluation method further includes: using the phoneme features to train the scoring model.

[0087] In the scoring model, the GOP, GWPP and duration information each have a corresponding weight, so that the GOP, GWPP and duration information can affect the scoring result to a certain extent.

[0088] In a specific implementation, the phoneme features and manually annotated scoring values ​​are used to perform training to obtain the scoring model.

[0089] In a specific implementation, based on the phoneme features and the scoring model, a first scoring result is obtained, and the scoring result includes a phoneme score of each of the phonemes, a word score of each word, and a sentence score of each sentence.

[0090] Continue to refer Figure 1 , Step S6: traverse each reference phoneme in the reference text to obtain a competing phoneme corresponding to the current reference phoneme.

[0091] Competitive phonemes corresponding to the current reference phonemes are obtained so that the actual pronunciation phonemes used can be obtained based on the output of the competitive phonemes and the acoustic model, and then compared with the current reference phonemes to achieve phoneme error detection; and in the subsequent process of obtaining the user's actual pronunciation phonemes, the recognition type of the phonemes can be limited, which is beneficial to improving the accuracy of phoneme error detection, and there is no need to obtain additional training data for phoneme-level recognition, which is beneficial to reducing costs.

[0092] The competing phonemes are easily misinterpreted phonemes that have a high pronunciation similarity to the current reference phonemes.

[0093] As an example, if the current reference phoneme is The competing phonemes may be e,

[0094] As another example, if the current reference phoneme is The competing phonemes may be ɑ.

[0095] Continue to refer Figure 1 , step S7: based on the time boundary information, obtaining the output of the acoustic model within the corresponding time range.

[0096] The output of the acoustic model within the corresponding time range is obtained so that the user's actual pronunciation phonemes can be obtained based on the competing phonemes and the output of the acoustic model within the time range. This not only limits the type of recognized phonemes, but also limits the time range of recognition, which is correspondingly beneficial to improving the accuracy of phoneme error detection and reducing the amount of calculation, thereby reducing the real-time rate (RTF).

[0097] Specifically, the boundary information of the current reference phoneme includes the start frame and end frame of the current reference phoneme in the audio. Accordingly, the acoustic model output corresponding to the speech data between the start frame and the end frame of the current reference phoneme is the output of the acoustic model within the corresponding time range.

[0098] In this embodiment, the output of the acoustic model includes the log likelihood of each frame on each phoneme.

[0099] Continue to refer Figure 1, step S8: based on the competing phonemes and the output of the acoustic model, obtaining the user's actual pronunciation phonemes.

[0100] The user's actual pronunciation phonemes are obtained for comparison with the current reference phonemes, and then while scoring the user's spoken pronunciation, phoneme error detection of the user's spoken pronunciation is also implemented, which correspondingly improves the effectiveness and comprehensiveness of the oral assessment.

[0101] Among them, the phoneme error detection of the user's spoken pronunciation is implemented, so that the user can know the specific errors in the pronunciation, and then can carry out targeted practice on the incorrectly pronounced phonemes, which is correspondingly conducive to improving the user experience.

[0102] In this embodiment, the same acoustic model is used for phoneme error detection and pronunciation scoring, which is beneficial to reducing the complexity and amount of computation of oral evaluation, and correspondingly improves the response speed and reduces the real time factor (RTF).

[0103] In addition, based on the competing phonemes and the output of the acoustic model, the user's actual pronunciation phonemes are obtained, thereby limiting the recognition type and time range of the phonemes. This is not only beneficial to improving the accuracy of phoneme error detection, but also does not require additional training data for phoneme-level recognition, which is beneficial to reducing costs.

[0104] Combined with reference Figure 2 , indicating Figure 1 In this embodiment, based on the competitive phonemes and the output of the acoustic model, step S8 of obtaining the user's actual pronunciation phonemes includes:

[0105] Step S81: traverse each reference phoneme in the reference text, and construct a second decoding network of competing phonemes corresponding to the current reference phoneme.

[0106] The second decoding network is constructed based on the competing phonemes corresponding to the current reference phonemes, so that in the subsequent process of obtaining the user's actual pronunciation phonemes, the recognition type of the phonemes can be limited, which is beneficial to improving the accuracy of phoneme error detection. There is no need to obtain additional training data for phoneme-level recognition, which is beneficial to reducing costs.

[0107] In this embodiment, constructing a second decoding network for competing phonemes corresponding to the current reference phoneme includes: obtaining one or more competing phonemes corresponding to the current reference phoneme; and using a network structure constructed in parallel by the current reference phoneme and the one or more competing phonemes as the second decoding network.

[0108] The competing phoneme is an error-prone phoneme that has a high pronunciation similarity to the current reference phoneme, and the second decoding network corresponding to the competing phoneme at least includes the current reference phoneme and the corresponding competing phoneme.

[0109] As an example, if the current reference phoneme is The competing phonemes may be e, Accordingly, the second decoding network is “ e. "The decoding network is constructed in parallel.

[0110] As another example, if the current reference phoneme is The competing phonemes may be ɑ, accordingly, the second decoding network is " ɑ” in parallel to form a decoding network.

[0111] Step S82: Based on the output of the second decoding network and the acoustic model, obtain the user's actual pronunciation phonemes.

[0112] As one embodiment, based on the output of the second decoding network and the acoustic model, the step of obtaining the user's actual pronunciation phonemes includes: based on the second decoding network, performing a second decoding process on the output of the acoustic model to obtain N decoding paths with the highest confidence, each decoding path including any one of the competing phonemes and the current reference phonemes, and taking the phonemes corresponding to the Top1 decoding path (i.e., the first decoding path with the highest confidence) as the user's actual pronunciation phonemes.

[0113] It should be noted that, in this embodiment, after the output of the acoustic model is subjected to a second decoding process, the oral evaluation method further includes: obtaining the phoneme features corresponding to the current reference phoneme and the phoneme features corresponding to the competing phonemes based on the time boundary information; performing a second scoring process on the output of the acoustic model based on the phoneme features corresponding to the current reference phoneme, the phoneme features corresponding to the competing phonemes, and the scoring model, to obtain a second scoring result corresponding to the current reference phoneme and each competing phoneme in the second decoding network.

[0114] The second scoring results corresponding to the current reference phoneme and each competing phoneme in the second decoding network are obtained, so as to provide feedback to the user and let the user know that his actual pronunciation is closer to a certain phoneme. By giving the scores of the competing phonemes and the current reference phoneme, the degree to which the pronunciation is close to the certain phoneme can also be reflected to a certain extent, so that the spoken pronunciation can be corrected in a targeted manner, thereby improving the user's learning effect and learning experience.

[0115] In other embodiments, based on the competing phonemes and the output of the acoustic model, the step of obtaining the user's actual pronunciation phonemes includes: based on the current reference phonemes and the corresponding competing phonemes, using a deep learning network to perform phoneme classification processing on the output of the acoustic model, outputting the probability corresponding to the current reference phoneme and the probability corresponding to each competing phoneme, and taking the competing phoneme or the current reference phoneme with the highest probability as the user's actual pronunciation phoneme.

[0116] In other embodiments, after performing phoneme classification processing on the output of the acoustic model using a deep learning network, the oral evaluation method further includes: obtaining a second scoring result corresponding to the current reference phoneme and each competing phoneme based on the probability corresponding to the current reference phoneme and the probability corresponding to each competing phoneme.

[0117] The current reference phoneme and the second scoring results corresponding to each competing phoneme are obtained, so as to provide feedback to the user and let the user know that his actual pronunciation is closer to a certain phoneme. By giving the scores of the competing phonemes and the current reference phoneme, it can also reflect to a certain extent the degree to which the pronunciation is close to the certain phoneme, so that the spoken pronunciation can be corrected in a targeted manner, thereby improving the user's learning effect and learning experience.

[0118] In order to solve the technical problem, an embodiment of the present invention further provides a spoken language evaluation device. Figure 3 It is a functional block diagram of an embodiment of a spoken language evaluation device of the present invention.

[0119] refer to Figure 3 In this embodiment, the oral evaluation device includes: an audio text acquisition module 10, which is used to acquire the speech data to be evaluated and the reference text of the speech data; a first network construction module 20, which is used to construct a first decoding network based on the reference text; a decoding module 30, which is used to perform a first decoding process on the speech data based on the acoustic model and the first decoding network, and obtain the time boundary information corresponding to each reference phoneme in the reference text; a feature extraction module 40, which is used to obtain the phoneme feature corresponding to each reference phoneme based on the time boundary information; a scoring module 50, which is used to perform a first scoring process on the speech data based on the phoneme feature and the scoring model, and obtain a first scoring result; a competitive phoneme acquisition module 60, which is used to traverse each reference phoneme in the reference text and obtain the competitive phoneme corresponding to the current reference phoneme; an acoustic model output acquisition module 70, which is used to obtain the output of the acoustic model within the corresponding time range based on the time boundary information; an actual phoneme acquisition module 80, which is used to obtain the actual pronunciation phoneme of the user based on the competitive phoneme and the output of the acoustic model.

[0120] In the oral evaluation device, the scoring module 50 is used to perform a first scoring process on the speech data based on the phoneme features and the scoring model to obtain a first scoring result; moreover, the competing phoneme acquisition module 60 is used to obtain competing phonemes corresponding to the current reference phonemes, and the actual phoneme acquisition module 80 obtains the user's actual pronunciation phonemes based on the competing phonemes and the output of the acoustic model, so that while the pronunciation of the user's spoken language is scored, phoneme error detection of the user's spoken pronunciation is also achieved, which correspondingly improves the effect and comprehensiveness of the oral evaluation; moreover, the use of the same acoustic model for phoneme error detection and pronunciation scoring is conducive to reducing the complexity and amount of calculation of the oral evaluation device, correspondingly improving the response speed, and thus reducing the real-time rate.

[0121] In addition, the actual phoneme acquisition module 80 obtains the user's actual pronunciation phonemes based on the competing phonemes and the output of the acoustic model, thereby limiting the recognition type and time range of the phonemes. This is not only beneficial to improving the accuracy of phoneme error detection, but also does not require additional training data for phoneme-level recognition, which is beneficial to reducing costs.

[0122] The voice data acquired by the audio-text acquisition module 10 is used as an object of oral evaluation.

[0123] Specifically, the voice data may be voice stream data input in real time, or file stream data in an audio file. The voice data may be data directly received and processed, or may be audio obtained from other terminals.

[0124] Voice stream data can usually be obtained by real-time recording of voice through hardware with real-time recording function such as microphones and sound cards. File stream data can usually be obtained by reading audio files that store recorded audio data. The format of the audio file can be arbitrary, including but not limited to .WAV, .MP3, etc.

[0125] The audio text acquisition module 10 also acquires reference text corresponding to the voice data, and the reference text is used as a reference benchmark for subsequent evaluation of the voice data. The reference text is also used to acquire corresponding reference phonemes.

[0126] Specifically, in the field of oral evaluation, a spoken text (or spoken voice, etc.) is usually provided to the user, who then follows the spoken text (or spoken voice, etc.) and makes corresponding sounds, and collects voice data corresponding to the sounds as the voice data to be evaluated, thereby achieving oral evaluation; wherein the spoken text, etc. provided to the user is the reference text.

[0127] As an example, the oral evaluation device is applied to oral evaluation of English, and the reference text is an English text. In other embodiments, the oral evaluation device can also be used for oral evaluation of other languages, and the reference text is a text of the corresponding language.

[0128] The first network construction module 20 constructs the first decoding network, thereby providing a decoding path for the decoding module 30 to perform a first decoding process on the speech data.

[0129] In this embodiment, the first decoding network is a decoding network based on WFST (weighted finite-state transducer).

[0130] Specifically, in this embodiment, since the reference text is already given, the first decoding network generally only includes one decoding path. Accordingly, the first decoding network is generally a serial network structure constructed based on the reference text.

[0131] In this embodiment, the first decoding network includes a word-level decoding network. Taking the reference text "how are you" as an example, the network structure of the corresponding first decoding network is: how->are->you.

[0132] The decoding module 30 is used to perform a first decoding process on the speech data to obtain the time boundary information corresponding to each reference phoneme, so that the feature extraction module 40 can obtain the phoneme feature corresponding to each reference phoneme according to the time boundary information. In addition, the time boundary information is also used to obtain the speech segment corresponding to the current reference phoneme in the speech data.

[0133] In the field of speech recognition, the output of the decoding process is usually the optimal path and the boundary of each node in the path. Accordingly, in this embodiment, since the path of the first decoding network has been determined by the reference text, the time boundary information corresponding to each reference phoneme in the reference text is obtained by performing the first decoding process.

[0134] As an embodiment, the time boundary information corresponding to the current reference phoneme includes: a start frame and an end frame corresponding to the current reference phoneme in the speech data.

[0135] Specifically, the first decoding process obtains the time boundary information corresponding to the phoneme by using the Force Alignment (FA) method. Specifically, based on the acoustic model, the corresponding phoneme sequence is searched through the pronunciation dictionary, and the input voice data is aligned with the reference text in the time dimension to obtain the time boundary information corresponding to each reference phoneme.

[0136] Taking the reference text "how are you" as an example, the sequence of reference phonemes of the reference text found through the pronunciation dictionary is: rju, perform a first decoding process on the voice data to obtain each reference phoneme (h rj u) corresponding to the time boundary information.

[0137] It should be noted that, in a specific implementation, the speech data is input into the acoustic model to obtain the output of the acoustic model.

[0138] In a specific implementation, the decoding module 30 performs a first decoding process on the speech data based on the output of the acoustic model and the first decoding network.

[0139] In a specific implementation, the output of the acoustic model includes the log likelihood of each frame on each phoneme.

[0140] Among them, the acoustic model is an acoustic model trained according to the user's standard voice, including but not limited to the traditional HMM (Hidden Markov Model) model and mainstream neural network models, such as DNN (Deep Neural Networks) model and LSTM (Long Short Term Memory) model.

[0141] In a specific implementation, in addition to the first decoding network and the acoustic model, a pronunciation dictionary and a language model are also used to perform a first decoding process on the speech data.

[0142] The acoustic model can be used to establish a mapping between speech features and phonemes in the speech data; the pronunciation dictionary can be used to establish a mapping between phonemes and words, thereby obtaining reference phonemes corresponding to the reference text. According to the mapping established by the acoustic model, the pronunciation dictionary and the first decoding network, the speech data is subjected to a first decoding process to obtain time boundary information corresponding to each reference phoneme in the reference text.

[0143] Specifically, the acoustic model is a knowledge representation of differences in acoustics, phonetics, environmental variables, speaker gender, accent, etc.; the pronunciation dictionary is a set of phoneme indexes corresponding to words.

[0144] The feature extraction module 40 obtains the phoneme features, which are used as input for scoring the speech data, so as to serve as evaluation features for oral evaluation.

[0145] In this embodiment, the phoneme features include any one or more of duration information, Goodness of Pronunciation (GOP) and Generalized Word Posterior Probability (GWPP).

[0146] As an embodiment, the GOP is calculated based on formula (I):

[0147]

[0148] Where t is time, It is the phoneme likelihood probability output by the acoustic model.

[0149] In other embodiments, other methods or algorithms may be used to calculate the GOP.

[0150] It should be noted that since the GOP algorithm performs normalization processing on the time dimension, if a single GOP dimension is used as a phoneme feature to score speech data, the impact of the time dimension on the scoring result cannot be reflected.

[0151] Therefore, in this embodiment, by making the phoneme feature include duration information, the time dimension of the phoneme is introduced as a scoring dimension for scoring the speech data. Accordingly, the pronunciation duration of the phoneme can affect the scoring result obtained by scoring the speech data, thereby improving the accuracy of the speech data scoring and further improving the accuracy and comprehensiveness of the pronunciation scoring.

[0152] For example, for long vowels and short vowels in English, the pronunciation duration of long vowels is longer than that of short vowels. By introducing the duration information of phonemes as a scoring dimension for scoring speech data, the accuracy of pronunciation scoring can be further improved accordingly.

[0153] GOP reflects the average of the difference between the probability of the reference phoneme in each frame and the maximum probability output by the acoustic model in the current frame at the frame level. Different from GOP, the GWPP characterizes the difference between the probability of the reference phoneme and other phonemes within the time boundary, and introduces a comparison with other phonemes, which can characterize whether the pronunciation is good enough from another dimension. Therefore, by introducing GWPP as a dimension of pronunciation scoring, it is helpful to improve the robustness of oral evaluation.

[0154] In this embodiment, the GWPP is calculated by formula (II):

[0155]

[0156] Among them, mi is the current reference phoneme, b i is the starting frame of the current reference phoneme, e i is the end frame of the current reference phoneme, It is the model probability score of the speech segment corresponding to the current reference phoneme.

[0157] By making the phoneme feature include one or more of duration information, GOP and GWPP, the robustness of the scoring process for speech data can be improved.

[0158] The scoring module 50 performs a first scoring process on the speech data, thereby performing a pronunciation score on the speech data.

[0159] In this embodiment, the scoring model is trained using the phoneme features.

[0160] In the scoring model, the GOP, GWPP and duration information each have a corresponding weight, so that the GOP, GWPP and duration information can affect the scoring result to a certain extent.

[0161] In a specific implementation, the phoneme features and manually annotated scoring values ​​are used to perform training to obtain the scoring model.

[0162] In a specific implementation, based on the phoneme features and the scoring model, a first scoring result is obtained, and the scoring result includes a phoneme score of each phoneme, a word score of each word, and a sentence score of each sentence.

[0163] The competing phoneme acquisition module 60 obtains the competing phonemes corresponding to the current reference phonemes so that the actual phoneme acquisition module 80 obtains the user's actual pronunciation phonemes based on the competing phonemes and the output of the acoustic model, and then compares them with the current reference phonemes to achieve phoneme error detection.

[0164] The competing phonemes are easily misinterpreted phonemes that have a high pronunciation similarity to the current reference phonemes.

[0165] As an example, if the current reference phoneme is The competing phonemes may be e,

[0166] As another example, if the current reference phoneme is The competing phonemes may be ɑ.

[0167] The acoustic model output acquisition module 70 is used to obtain the output of the acoustic model within the corresponding time range based on the time boundary information, so that the actual phoneme acquisition module 80 can obtain the user's actual pronunciation phonemes within the time boundary, thereby not only limiting the recognition type of the phonemes, but also limiting the time range of recognition, which is correspondingly beneficial to improving the accuracy of phoneme error detection, and is also beneficial to reducing the amount of calculation, thereby reducing the real-time rate (RTF).

[0168] Specifically, the boundary information of the current reference phoneme includes the start frame and end frame of the current reference phoneme in the audio. Accordingly, the acoustic model output corresponding to the speech data between the start frame and the end frame of the current reference phoneme is the output of the acoustic model within the corresponding time range.

[0169] In this embodiment, the output of the acoustic model refers to the likelihood probability (log likelihood) of each frame on each phoneme.

[0170] The actual phoneme acquisition module 80 is used to obtain the user's actual pronunciation phonemes based on the competing phonemes and the output of the acoustic model, so as to compare them with the current reference phonemes, thereby not only scoring the user's spoken pronunciation, but also detecting phoneme errors in the user's spoken pronunciation, thereby correspondingly improving the effectiveness and comprehensiveness of the spoken language assessment.

[0171] Among them, by implementing phoneme error detection for the user's spoken pronunciation, the user can know the specific errors in the pronunciation and then can conduct targeted practice on the incorrectly pronounced phonemes, which is correspondingly beneficial to improving the user experience.

[0172] In this embodiment, the same acoustic model is used for phoneme error detection and pronunciation scoring, which is beneficial to reducing the complexity and computational complexity of the spoken language evaluation device, correspondingly improving the response speed and reducing the real time factor (RTF).

[0173] In addition, based on the competing phonemes and the output of the acoustic model, the user's actual pronunciation phonemes are obtained, thereby limiting the recognition type and time range of the phonemes. This is not only beneficial to improving the accuracy of phoneme error detection, but also does not require additional training data for phoneme-level recognition, which is beneficial to reducing costs.

[0174] Combined with reference Figure 4 , showing Figure 3 A functional block diagram of an embodiment of an actual phoneme acquisition module in FIG.

[0175] As an embodiment, the actual phoneme acquisition module 80 includes: a second network construction module 81, which is used to traverse each reference phoneme in the reference text and construct a second decoding network of competing phonemes corresponding to the current reference phoneme; an acquisition module 82, which obtains the user's actual pronunciation phonemes based on the output of the second decoding network and the acoustic model.

[0176] The second decoding network is constructed based on the competing phonemes corresponding to the current reference phonemes, so that in the process of obtaining the user's actual pronunciation phonemes in the acquisition module 82, the recognition type of the phonemes can be limited, which is beneficial to improving the accuracy of phoneme error detection, and there is no need to obtain additional training data for phoneme-level recognition, which is beneficial to reducing costs.

[0177] In this embodiment, the second network construction module 81 is used to obtain one or more competing phonemes corresponding to the current reference phoneme, and is also used to use the network structure constructed in parallel by the current reference phoneme and the one or more competing phonemes as the second decoding network.

[0178] The competing phoneme is an error-prone phoneme that has a high pronunciation similarity to the current reference phoneme, and the second decoding network corresponding to the competing phoneme at least includes the current reference phoneme and the corresponding competing phoneme.

[0179] As an example, if the current reference phoneme is The competing phonemes may be e, Accordingly, the second decoding network is “ e. "The decoding network is constructed in parallel.

[0180] As another example, if the current reference phoneme is The competing phonemes may be ɑ, accordingly, the second decoding network is " ɑ” in parallel to form a decoding network.

[0181] In this embodiment, the acquisition module 82 is the decoding module. By making the acquisition module 82 the decoding module, the same decoding module is used for both phoneme error detection and pronunciation scoring, without introducing additional functional modules, which is conducive to simplifying complexity.

[0182] Correspondingly, the decoding module is also used to perform a second decoding process on the output of the acoustic model based on the second decoding network to obtain N decoding paths with the highest confidence, each decoding path including any one of the competing phonemes and the current reference phonemes, and taking the phoneme corresponding to the Top1 decoding path as the user's actual pronunciation phoneme.

[0183] In this embodiment, the feature extraction module 40 is also used to obtain the phoneme features corresponding to the current reference phoneme and the phoneme features corresponding to the competing phonemes based on the time boundary information; the scoring module 50 performs a second scoring process on the output of the acoustic model based on the phoneme features corresponding to the current reference phoneme, the phoneme features corresponding to the competing phonemes, and the scoring model to obtain the second scoring results corresponding to the current reference phoneme and each competing phoneme in the second decoding network.

[0184] The second scoring results corresponding to the current reference phoneme and each competing phoneme in the second decoding network are obtained, so as to provide feedback to the user and let the user know that his actual pronunciation is closer to a certain phoneme. By giving the scores of the competing phonemes and the current reference phoneme, the degree to which the pronunciation is close to the certain phoneme can also be reflected to a certain extent, so that the spoken pronunciation can be corrected in a targeted manner, thereby improving the user's learning effect and learning experience.

[0185] In other embodiments, the actual phoneme acquisition module can also be a phoneme classifier, which is used to perform phoneme classification processing on the output of the acoustic model based on the current reference phoneme and the corresponding competitive phoneme using a deep learning network, output the probability corresponding to the current reference phoneme and the probability corresponding to each competitive phoneme, and use the competitive phoneme or the current reference phoneme with the highest probability as the actual pronunciation phoneme of the user.

[0186] In other embodiments, after using a deep learning network to perform phoneme recognition and classification processing on the output of the acoustic model, the scoring module can also be used to obtain a second scoring result corresponding to the current reference phoneme and each competing phoneme based on the probability corresponding to the current reference phoneme and the probability corresponding to each competing phoneme.

[0187] In order to solve the above problem, an embodiment of the present invention further provides a device, which can implement the oral evaluation method provided by the embodiment of the present invention by loading the above oral evaluation method in the form of a program.

[0188] An optional hardware structure of the device provided by the embodiment of the present invention can be as follows Figure 5 As shown, it includes: at least one processor 01, at least one communication interface 02, at least one memory 03 and at least one communication bus 04.

[0189] In the embodiment of the present invention, the number of the processor 01, the communication interface 02, the memory 03, and the communication bus 04 is at least one, and the processor 01, the communication interface 02, and the memory 03 communicate with each other through the communication bus 04;

[0190] Optionally, the communication interface 02 may be an interface of a communication module for network communication, such as an interface of a GSM module.

[0191] Optionally, the processor 01 may be a central processing unit CPU, or an application specific integrated circuit ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present invention.

[0192] Optionally, the memory 03 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0193] The memory 03 stores one or more computer instructions, and the one or more computer instructions are executed by the processor 01 to implement the spoken language assessment method provided in the embodiment of the present invention.

[0194] It should be noted that the above-mentioned terminal device may also include other devices (not shown) that may not be necessary for understanding the contents disclosed in the embodiments of the present invention; given that these other devices may not be necessary for understanding the contents disclosed in the embodiments of the present invention, the embodiments of the present invention will not introduce them one by one.

[0195] Correspondingly, an embodiment of the present invention further provides a storage medium, wherein the storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the spoken language evaluation method described in the embodiment of the present invention.

[0196] The storage medium is a computer-readable storage medium, and the storage medium can be a read-only memory (ROM), a random access memory (RAM), a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk, or other medium that can store program codes.

[0197] The above-mentioned embodiments of the present invention are combinations of elements and features of the present invention. Unless otherwise mentioned, the elements or features may be considered as optional. Each element or feature may be put into practice without being combined with other elements or features. In addition, the embodiments of the present invention may be constructed by combining some elements and / or features. The order of operations described in the embodiments of the present invention may be rearranged. Some constructions of any one embodiment may be included in another embodiment, and may be replaced by the corresponding construction of another embodiment. It is obvious to those skilled in the art that claims that do not have a clear reference relationship to each other in the attached claims may be combined into embodiments of the present invention, or may be included as new claims in the amendment after submitting this application.

[0198] The embodiments of the present invention can be implemented by various means such as hardware, firmware, software or a combination thereof. In a hardware configuration, the method according to the exemplary embodiment of the present invention can be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, etc.

[0199] In a firmware or software configuration, the embodiments of the present invention may be implemented in the form of modules, procedures, functions, etc. The software code may be stored in a memory unit and executed by a processor. The memory unit is located inside or outside the processor and may send data to and receive data from the processor via various known means.

[0200] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A spoken language assessment method, characterized in that: include: Acquire speech data to be evaluated and a reference text of the speech data; Based on the reference text, construct a first decoding network; Based on the acoustic model and the first decoding network, performing a first decoding process on the speech data to obtain time boundary information corresponding to each reference phoneme in the reference text; Based on the time boundary information, obtaining a phoneme feature corresponding to each reference phoneme; Based on the phoneme features and the scoring model, performing a first scoring process on the speech data to obtain a first scoring result; Traversing each reference phoneme in the reference text to obtain a competing phoneme corresponding to the current reference phoneme; Based on the time boundary information, obtaining an output of the acoustic model within a corresponding time range; Based on the competing phonemes and the output of the acoustic model, the user's actual pronunciation phonemes are obtained.

2. The oral language evaluation method according to claim 1, characterized in that: The phoneme features include any one or more of duration information, good pronunciation value and generalized posterior probability.

3. The oral language evaluation method according to claim 2, characterized in that: The generalized posterior probability is calculated by the following formula: Among them, m i is the current reference phoneme, b i is the starting frame of the current reference phoneme, e i is the end frame of the current reference phoneme, It is the model probability score of the speech segment corresponding to the current reference phoneme.

4. The oral language evaluation method according to claim 2, wherein: The spoken language evaluation method further includes: using the phoneme features to train the scoring model.

5. The oral language evaluation method according to claim 1, wherein: Based on the competitive phonemes and the output of the acoustic model, the step of obtaining the user's actual pronunciation phonemes includes: traversing each reference phoneme in the reference text, and constructing a second decoding network of competitive phonemes corresponding to the current reference phoneme; based on the second decoding network and the output of the acoustic model, obtaining the user's actual pronunciation phonemes.

6. The oral language evaluation method according to claim 5, characterized in that: Constructing a second decoding network of competing phonemes corresponding to the current reference phoneme includes: obtaining one or more competing phonemes corresponding to the current reference phoneme; and using a network structure constructed in parallel by the current reference phoneme and the one or more competing phonemes as the second decoding network.

7. The oral language evaluation method according to claim 5, characterized in that: Based on the output of the second decoding network and the acoustic model, the step of obtaining the user's actual pronunciation phonemes comprises: Based on the second decoding network, a second decoding process is performed on the output of the acoustic model to obtain N decoding paths with the highest confidence, each decoding path includes any one of the competing phonemes and the current reference phonemes, and the phoneme corresponding to the Top1 decoding path is used as the user's actual pronunciation phoneme.

8. The oral language evaluation method according to claim 7, characterized in that: After performing a second decoding process on the output of the acoustic model, the spoken language evaluation method further includes: Based on the time boundary information, obtaining a phoneme feature corresponding to the current reference phoneme and a phoneme feature corresponding to the competing phoneme; Based on the phoneme features corresponding to the current reference phoneme, the phoneme features corresponding to the competing phonemes, and the scoring model, a second scoring process is performed on the output of the acoustic model to obtain second scoring results corresponding to the current reference phoneme and each competing phoneme in the second decoding network.

9. The oral language evaluation method according to claim 1, wherein: Based on the competitive phonemes and the output of the acoustic model, the step of obtaining the user's actual pronunciation phonemes comprises: Based on the current reference phoneme and the corresponding competing phonemes, a deep learning network is used to perform phoneme classification processing on the output of the acoustic model, and the probability corresponding to the current reference phoneme and the probability corresponding to each competing phoneme are output, and the competing phoneme or the current reference phoneme with the highest probability is used as the actual pronunciation phoneme of the user.

10. The oral language evaluation method according to claim 9, characterized in that: After performing phoneme recognition and classification processing on the output of the acoustic model using a deep learning network, the spoken language evaluation method further includes: Based on the probability corresponding to the current reference phoneme and the probability corresponding to each competing phoneme, a second scoring result corresponding to the current reference phoneme and each competing phoneme is obtained.

11. A spoken language evaluation device, characterized in that: include: An audio text acquisition module, used to acquire the voice data to be evaluated and a reference text of the voice data; A first network construction module, used to construct a first decoding network based on the reference text; A decoding module, configured to perform a first decoding process on the speech data based on the acoustic model and the first decoding network, and obtain time boundary information corresponding to each reference phoneme in the reference text; A feature extraction module, used for obtaining a phoneme feature corresponding to each reference phoneme based on the time boundary information; A scoring module, configured to perform a first scoring process on the speech data based on the phoneme features and the scoring model to obtain a first scoring result; A competing phoneme acquisition module, used for traversing each reference phoneme in the reference text to obtain a competing phoneme corresponding to the current reference phoneme; An acoustic model output acquisition module, used to acquire the output of the acoustic model within a corresponding time range based on the time boundary information; The actual phoneme acquisition module is used to obtain the user's actual pronunciation phonemes based on the competitive phonemes and the output of the acoustic model.

12. A device, characterized in that: The method comprises at least one memory and at least one processor, wherein the memory stores one or more computer instructions, wherein the one or more computer instructions are executed by the processor to implement the oral language evaluation method according to any one of claims 1 to 10.

13. A storage medium, characterized in that: The storage medium stores one or more computer instructions, and the one or more computer instructions are used to implement the spoken language evaluation method according to any one of claims 1 to 10.

Images