Spoken pronunciation quality evaluation method, device, equipment and storage medium
By acquiring multiple sentence-level speech signals and using a pre-defined quality evaluation neural network to calculate sentence-level posterior probability distribution scores, the problem of unstable evaluation in existing technologies is solved, achieving high-precision and stable evaluation of English accent quality and improving the accuracy and consistency of the evaluation.
Patent Information
- Application Number
- CN202011480460.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-12-15
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2040-12-15
AI Technical Summary
Existing technologies, when assessing a speaker's spoken English proficiency, suffer from data sparsity due to the limited number of spoken sentences, leading to unstable and biased assessment results. This makes it difficult to accurately classify and evaluate the English accent quality of non-native English speakers.
By acquiring multiple sentence-level speech signals from test subjects, a pre-set quality assessment neural network is used for quality assessment, including the calculation and training of sentence-level posterior probability distribution scores. This is combined with a shared hidden layer, an average layer, a spoken pronunciation level hidden layer, and a classification prediction layer to improve the accuracy and stability of the assessment.
It achieves high-precision and stable assessment of the quality of spoken pronunciation, accurately distinguishes individual and systemic pronunciation errors, provides detailed pronunciation information, and improves the accuracy and consistency of English accent classification.
Smart Images

Figure CN112700795B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of speech recognition technology, and in particular to a method, device, equipment and storage medium for evaluating the quality of spoken pronunciation. Background Art
[0002] Speaker-level assessment of spoken language proficiency is very important for second language education, whether in certifying language teachers or providing advice and feedback to learners. Previous methods have assessed a speaker's spoken pronunciation level based on a limited number of their spoken sentences, which leads to biased or unstable results due to the sparse data that cannot fully cover a speaker's specific pronunciation errors.
[0003] English is a global lingua franca and the cultivation of English communication skills is crucial. It is estimated that there are more than 200 million students learning English in China and 1.5 billion students learning English in the world. Due to the lack of a sufficient number of native English teachers, more and more non-native English speakers are being recruited as online English teachers in China. It is urgent to propose a high-precision second language speaker level pronunciation assessment system to provide learners with training feedback and evaluate whether candidates are qualified to become English teachers.
[0004] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0005] The main purpose of the present invention is to provide a method, device, equipment and storage medium for evaluating the quality of spoken pronunciation, aiming to solve the technical problem of how to improve the accuracy and stability of accent classification.
[0006] To achieve the above object, the present invention provides a method for evaluating the quality of spoken pronunciation, the method comprising the following steps:
[0007] Acquire multiple sentence-level speech signals of the person to be tested;
[0008] Determining a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signal;
[0009] The sentence-level posterior probability distribution score is subjected to quality evaluation through a trained preset quality evaluation neural network to obtain a spoken pronunciation quality evaluation result of the person to be tested.
[0010] Optionally, the step of inputting the sentence-level posterior probability distribution score into a trained preset quality evaluation neural network to obtain an oral pronunciation quality evaluation result of the person to be tested includes:
[0011] Inputting the sentence-level posterior probability distribution score into a preset shared hidden layer to obtain hidden output feature data, wherein the trained preset quality evaluation neural network includes a preset shared hidden layer, an average layer, a spoken pronunciation level hidden layer, and a preset classification prediction layer;
[0012] Inputting the hidden output feature data into an average layer to obtain average feature data;
[0013] Inputting the average feature data into a spoken pronunciation level hidden layer to obtain target feature data;
[0014] Inputting the target feature data into a preset classification prediction layer to obtain a target label;
[0015] The oral pronunciation quality evaluation result of the person to be tested is obtained according to the target label.
[0016] Optionally, before performing quality evaluation on the sentence-level posterior probability distribution score through a trained preset quality evaluation neural network to obtain the oral pronunciation quality evaluation result of the person to be tested, the method further includes:
[0017] Get the preset sample voice set;
[0018] Determining a corresponding sentence-level posterior probability distribution sample score according to the preset sample speech set;
[0019] Create a preset quality assessment neural network;
[0020] Inputting the sentence-level posterior probability distribution sample score into the preset quality evaluation neural network to obtain the loss value of the preset quality evaluation neural network;
[0021] When the loss value is less than or equal to a preset loss threshold, the trained preset quality evaluation neural network is obtained.
[0022] Optionally, inputting the sentence-level posterior probability distribution sample score into the preset quality assessment neural network to obtain the loss value of the preset quality assessment neural network includes:
[0023] Inputting the sentence-level posterior probability distribution sample scores into a preset shared hidden layer to obtain hidden output sample data;
[0024] Inputting the hidden output sample data into the average layer to obtain average sample data;
[0025] Inputting the average sample data into a spoken pronunciation level hidden layer to obtain target sample data;
[0026] Input the target sample data into a preset classification prediction layer to obtain a prediction label;
[0027] Determine a cross entropy loss value according to the predicted label and the preset label;
[0028] The loss value of the preset quality evaluation neural network is determined according to the cross entropy loss value.
[0029] Optionally, determining the loss value of the preset quality evaluation neural network according to the cross entropy loss value includes:
[0030] The loss value of the preset quality evaluation neural network is determined according to the cross entropy loss value and the root mean square error loss value.
[0031] Optionally, before determining the loss value of the preset quality assessment neural network according to the cross entropy loss value and the root mean square error loss value, the method further includes:
[0032] Calculate the pth root mean square error based on the pth hidden output sample data and the p+1th hidden output sample data, where 1≤p≤K-1, and K is the total number of hidden output sample data;
[0033] According to the pth root mean square error, K-1 root mean square errors are obtained;
[0034] The average value of K-1 root mean square errors is obtained according to the K-1 root mean square errors;
[0035] The average value of the K-1 root mean square errors is taken as the root mean square error loss value.
[0036] Optionally, determining a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signal comprises:
[0037] Obtaining a preset standard phoneme sequence, wherein the preset standard phoneme sequence includes M preset standard phonemes;
[0038] Aligning the sentence-level speech signal with the preset standard phoneme sequence to obtain alignment segments corresponding to each preset standard phoneme, wherein the total number of alignment segments corresponding to the i-th preset standard phoneme is N, 1≤i≤M, and the t-th alignment segment contains R feature frames, 1≤t≤N;
[0039] Calculate the posterior probability distribution vector corresponding to the j-th feature frame according to the j-th feature frame and the preset standard factor sequence, wherein 1≤j≤R;
[0040] Obtain R posterior probability distribution vectors according to the posterior probability distribution vector corresponding to the j-th feature frame;
[0041] Calculate the posterior probability distribution score corresponding to the t-th alignment segment according to the R posterior probability distribution vectors;
[0042] Obtaining posterior probability distribution scores corresponding to N alignment segments according to the posterior probability distribution score corresponding to the t-th alignment segment;
[0043] The sentence-level posterior probability distribution scores of the sentence-level speech signals are calculated one-to-one according to the posterior probability distribution scores corresponding to the N aligned segments.
[0044] In addition, to achieve the above object, the present invention also provides a spoken pronunciation quality evaluation device, the spoken pronunciation quality evaluation device comprising:
[0045] An acquisition module, used to acquire multiple sentence-level speech signals of the person to be tested;
[0046] A score calculation module, used to determine a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signal;
[0047] The evaluation module is used to perform quality evaluation on the sentence-level posterior probability distribution score through a trained preset quality evaluation neural network to obtain an oral pronunciation quality evaluation result of the person to be tested.
[0048] In addition, to achieve the above-mentioned purpose, the present invention also proposes a spoken pronunciation quality evaluation device, which includes: a memory, a processor, and a spoken pronunciation quality evaluation program stored in the memory and executable on the processor, and the spoken pronunciation quality evaluation program is configured to implement the steps of the spoken pronunciation quality evaluation method as described above.
[0049] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a spoken pronunciation quality evaluation program is stored, and when the spoken pronunciation quality evaluation program is executed by a processor, the steps of the spoken pronunciation quality evaluation method as described above are implemented.
[0050] The present invention obtains multiple sentence-level speech signals of the person to be tested; determines one-to-one corresponding sentence-level posterior probability distribution scores according to the sentence-level speech signals; and performs quality evaluation on the sentence-level posterior probability distribution scores through a trained preset quality evaluation neural network to obtain the oral pronunciation quality evaluation result of the person to be tested. In the above manner, the sentence-level posterior probability distribution scores are determined according to the multiple sentence-level speech signals of the speaker. The multiple sentence-level posterior probability distribution scores are speaker-level features, which not only include detailed information and pronunciation excellence of the speaker's pronunciation, but also include the probability of misreading standard factors as other factors, and are more explanatory: they explain individual pronunciation errors and system pronunciation errors, thereby providing a data basis for the preset quality evaluation neural network, and referencing the trained preset quality evaluation neural network to perform quality evaluation on multiple sentence-level posterior probability distribution scores, thereby improving the accuracy and stability of accent classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 It is a structural diagram of a spoken pronunciation quality evaluation device in a hardware operating environment involved in an embodiment of the present invention;
[0052] Figure 2 It is a flow chart of a first embodiment of a method for evaluating spoken pronunciation quality of the present invention;
[0053] Figure 3 A schematic diagram of a preset quality evaluation neural network structure of an embodiment of a method for evaluating the quality of spoken pronunciation of the present invention;
[0054] Figure 4 It is a flow chart of a second embodiment of the method for evaluating the quality of spoken pronunciation of the present invention;
[0055] Figure 5 A performance comparison diagram of different features of an embodiment of a method for evaluating spoken pronunciation quality of the present invention;
[0056] Figure 6 It is a structural block diagram of the first embodiment of the oral pronunciation quality evaluation device of the present invention.
[0057] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0058] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0059] Reference Figure 1 , Figure 1 The present invention is a schematic diagram of the structure of a spoken pronunciation quality evaluation device in a hardware operating environment according to an embodiment of the present invention.
[0060] like Figure 1As shown, the spoken pronunciation quality evaluation device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (WIreless-FIdelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM) memory, or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk memory. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0061] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the spoken pronunciation quality evaluation device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.
[0062] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a spoken pronunciation quality evaluation program.
[0063] exist Figure 1 In the spoken pronunciation quality evaluation device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the spoken pronunciation quality evaluation device of the present invention can be arranged in the spoken pronunciation quality evaluation device, and the spoken pronunciation quality evaluation device calls the spoken pronunciation quality evaluation program stored in the memory 1005 through the processor 1001, and executes the spoken pronunciation quality evaluation method provided by the embodiment of the present invention.
[0064] The embodiment of the present invention provides a method for evaluating the quality of spoken pronunciation, referring to Figure 2 , Figure 2 Schematic diagram of the flow chart of the first embodiment of the method for evaluating the quality of spoken pronunciation of the present invention.
[0065] In this embodiment, the method for evaluating the quality of spoken pronunciation comprises the following steps:
[0066] Step S10: Acquire multiple sentence-level speech signals of the person to be tested.
[0067] It can be understood that the execution subject of this embodiment is a spoken pronunciation quality evaluation device, which can be a computer, a server, a cloud server or other devices, or a chip, and this embodiment does not limit this.
[0068] It should be noted that sentence-level speech signals refer to speech signals collected from the person to be tested reading multiple sentences. In a specific implementation, in order to make the oral pronunciation quality evaluation smoother and more efficient, reading samples can be prepared in advance, and the person to be tested can be asked to read multiple sentences based on the reading samples. The speech acquisition device collects sentence-level speech signals and sends them to the oral pronunciation quality evaluation device, thereby acquiring multiple sentence-level speech signals of the person to be tested. If the oral pronunciation quality evaluation device is equipped with a microphone or a sound card, the oral pronunciation quality evaluation device can directly collect the speech signal of the person to be tested.
[0069] Step S20: determining a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signal.
[0070] It should be noted that each sentence-level speech signal has a corresponding sentence-level posterior probability distribution score. Each sentence-level speech signal is input into a preset acoustic model containing a posterior probability distribution (PPD) algorithm to obtain a sentence-level posterior probability distribution score corresponding to the sentence-level speech signal.
[0071] Specifically, in order to obtain a sentence-level posterior probability distribution score, step S20 includes: obtaining a preset standard phoneme sequence, wherein the preset standard phoneme sequence includes M preset standard phonemes; aligning the sentence-level speech signal with the preset standard phoneme sequence to obtain alignment segments corresponding to each preset standard phoneme, wherein the total number of alignment segments corresponding to the i-th preset standard phoneme is N, 1≤i≤M, and the t-th alignment segment includes R feature frames, 1≤t≤N; calculating the posterior probability distribution vector corresponding to the j-th feature frame according to the j-th feature frame and the preset standard factor sequence, wherein 1≤j≤R; obtaining R posterior probability distribution vectors according to the posterior probability distribution vector corresponding to the j-th feature frame; calculating the posterior probability distribution score corresponding to the t-th alignment segment according to the R posterior probability distribution vectors; obtaining the posterior probability distribution scores corresponding to the N alignment segments according to the posterior probability distribution scores corresponding to the t-th alignment segment; and calculating the sentence-level posterior probability distribution scores of the sentence-level speech signals one-to-one according to the posterior probability distribution scores corresponding to the N alignment segments.
[0072] It should be noted that there are two ways to calculate the sentence-level posterior probability distribution score. The first way is: calculate the sentence-level posterior probability distribution score corresponding to the sentence-level speech signal one by one according to the posterior probability distribution scores corresponding to the N aligned segments, including: calculating the average value according to the posterior probability distribution scores corresponding to the N aligned segments to obtain the phoneme-level posterior probability distribution score corresponding to the i-th preset standard phoneme; and using the phoneme-level posterior probability distribution score as the sentence-level posterior probability distribution score.
[0073] It can be understood that for each sentence-level speech signal, the sentence-level speech signal is aligned with a preset standard phoneme sequence, which includes M preset standard phonemes. In a specific implementation, the most representative preset standard factor is selected. For example, some people tend to misread the phoneme / f / as / p / and the phoneme / p / as / b / . Therefore, the i-th preset standard phoneme can be selected as / p / or / f / , thereby determining the phoneme-level posterior probability distribution score when the speaker misreads / f / as / p / or / p / as / b / , and the phoneme-level posterior probability distribution score is used as the sentence-level posterior probability distribution score.
[0074] Specifically, the second method is: calculating the sentence-level posterior probability distribution score corresponding to the sentence-level speech signal one-to-one according to the posterior probability distribution scores corresponding to the N aligned segments, including: calculating the average value according to the posterior probability distribution scores corresponding to the N aligned segments, and obtaining the posterior probability distribution score corresponding to the i-th preset standard phoneme; calculating the average value according to the M posterior probability distribution scores, and obtaining the sentence-level posterior probability distribution score corresponding to the sentence-level speech signal one-to-one.
[0075] It should be noted that the sentence-level posterior probability distribution score derived from the preset acoustic model contains detailed information about the speaker's pronunciation. In a specific implementation, if the performance of the preset acoustic model is good enough, the sentence-level posterior probability distribution score can accurately reflect the degree of closeness between the pronunciation and each standard phoneme, including not only the goodness of pronunciation (GOP) of the standard phoneme, but also the probability of misreading the standard phoneme as other phonemes. Take the case where the speaker misreads the standard phoneme / p / as / b / as an example:
[0076] According to formula (1), the standard phoneme q is calculated i The corresponding posterior probability distribution score:
[0077]
[0078] Where q is a preset standard phoneme sequence, q=[q 1 ,q 2 ,…,q M ]; Alignment segment The j-th feature frame; is the total number of frames in this segment; N is the number of preset standard phonemes q i The total number of segments.
[0079] It should be noted that when the N segments come from only one speech sentence, the concatenated posterior probability distribution score vector of all preset standard phonemes is defined as a sentence-level feature. The speaker-level feature refers to the average of the posterior probability distribution scores of all speech sentence segments belonging to a speaker. In other words, the posterior probability distribution scores corresponding to the M preset standard phonemes are averaged to obtain the sentence-level posterior probability distribution score corresponding to the sentence-level speech signal.
[0080] Let's take the calculation of the pronunciation score (PS) of each phoneme as an example for comparison:
[0081] PS can measure the goodness of pronunciation (GOP) of each phoneme in the accented pronunciation. In the specific implementation, the pronunciation score is used as a phoneme-level feature and then converted to a sentence-level feature through several schemes, such as calculating the minimum, mean, and standard deviation of the phoneme-level pronunciation scores of vowels, consonants, and syllables in each speech sentence. However, these schemes may ignore some detailed information of each phoneme.
[0082] Since the acoustic model outputs frame-level probability, but in order to compare with the PPD proposed in this embodiment, phoneme-level features are required, so the pronunciation score of each phoneme is calculated by first calculating the forced alignment between the accented speech and the standard phoneme, and then calculating the pronunciation score of each phoneme. The pronunciation score is calculated by formula (2):
[0083]
[0084] in, Alignment segment The j-th feature frame; is the total number of frames in this segment; N is the number of preset standard phonemes q i The total number of segments.
[0085] It is understandable that although PS can describe the correctness of each standard phoneme, it cannot distinguish between individual pronunciation errors and systematic pronunciation errors, for example, a phoneme in English is pronounced as another phoneme that does not exist in English. Pronunciation errors can be divided into two major categories: (1) The first category includes individual pronunciation errors, such as unintentional mispronunciation of phonemes or words; (2) The second category of errors includes the substitution of specific phonemes, such as phonemes that exist in the native language but not in the target language. The second category of pronunciation errors is called systematic pronunciation errors.
[0086] It should be noted that, compared with the PS method, the PPD method adopted in this embodiment takes into account all preset standard phonemes when calculating the posterior probability distribution vector, while the PS method only considers the i-th preset standard phoneme. The PPD method adopted in this embodiment better describes individual pronunciation errors and system pronunciation errors.
[0087] Take the Phonological Rule (PR) as an example for comparison:
[0088] PR is a way of expressing the phonological or morphological processes of a system or the diachronic changes in a language. It can be defined as a mapping between two different levels of sound representation (substratum and surface), or it can be described as a "generalization" about how sounds are pronounced differently in different contexts. Taking some English accents as an example, PR specifically refers to common pronunciation errors among some people. For example, the / p / in "poor" is mispronounced as / b / and thus sounds like "bore".
[0089] Table 1 is an example table of common pronunciation errors of some people. Table 1 lists some common pronunciation errors that some people make when pronouncing British English. In the specific implementation, 36 1-to-1 speech rules are used, and the triggering times of each speech rule in all speech sentences are counted as speaker-level features. PR only contains speaker-level features and no sentence-level signs. Therefore, there is a difference between using PR and the PPD proposed in this embodiment. PPD contains more detailed system pronunciation error information than PR.
[0090] British Phonemes Mispronounced phonemes f p p b,f ae aa,ah ch,zh sh
[0091] Table 1
[0092] Step S30: The sentence-level posterior probability distribution score is subjected to quality evaluation through a trained preset quality evaluation neural network to obtain a spoken pronunciation quality evaluation result of the person to be tested.
[0093] It should be noted that the preset quality evaluation neural network can be a recurrent neural network (RNN). This embodiment adopts a separately designed speaker-level neural network (SL-NN) based on multiple speech sentences of the speaker, which includes a shared hidden layer, an average layer, a hidden layer and a preset classification prediction layer.
[0094] Specifically, step S30 includes: inputting the sentence-level posterior probability distribution score into a preset shared hidden layer to obtain hidden output feature data, wherein the trained preset quality evaluation neural network includes a preset shared hidden layer, an average layer, a spoken pronunciation level hidden layer and a preset classification prediction layer; inputting the hidden output feature data into the average layer to obtain average feature data; inputting the average feature data into the spoken pronunciation level hidden layer to obtain target feature data; inputting the target feature data into the preset classification prediction layer to obtain a target label; and obtaining the oral pronunciation quality evaluation result of the person to be tested according to the target label.
[0095] Reference Figure 3 , Figure 3 The schematic diagram of the preset quality evaluation neural network structure of an embodiment of the oral pronunciation quality evaluation method of the present invention is shown in FIG. The preset quality evaluation neural network includes a preset shared hidden layer, an average layer, an oral pronunciation level hidden layer and a preset classification prediction layer.
[0096] It can be understood that the shared hidden layer is a sentence-level hidden layer, which is shared between sentences, the spoken pronunciation level hidden layer is a speaker-level hidden layer, and the preset classification prediction layer is a softmax layer. Where f represents the sentence-level PPD, and h represents the hidden output of the shared hidden layer. The hidden output h is averaged before being input to the speaker-level hidden layer.
[0097] This embodiment obtains multiple sentence-level speech signals of the person to be tested; determines a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signals; and performs quality evaluation on the sentence-level posterior probability distribution score through a trained preset quality evaluation neural network to obtain the oral pronunciation quality evaluation result of the person to be tested. In the above manner, the sentence-level posterior probability distribution score is determined according to the multiple sentence-level speech signals of the speaker. The multiple sentence-level posterior probability distribution scores are speaker-level features, which not only include detailed information and pronunciation excellence of the speaker's pronunciation, but also include the probability of misreading standard factors as other factors, and are more explanatory: they explain individual pronunciation errors and system pronunciation errors, thereby providing a data basis for the preset quality evaluation neural network, and referencing the trained preset quality evaluation neural network to perform quality evaluation on multiple sentence-level posterior probability distribution scores, thereby improving the accuracy and stability of accent classification.
[0098] refer to Figure 4 , Figure 4 Schematic diagram of the flow chart of the second embodiment of the method for evaluating the quality of spoken pronunciation of the present invention.
[0099] Based on the first embodiment, the method for evaluating the quality of spoken language pronunciation in this embodiment further includes, before step S30:
[0100] Step S301: Obtain a preset sample voice set.
[0101] It is understandable that this embodiment is illustrated by taking the pronunciation level evaluation of the English accent of country A as an example, and adopts the pronunciation level evaluation data set (PEAED) of the English accent of country A. The PEAED data set collects 174 teachers of country A who teach Chinese students online and 74 speakers of country A accent from non-teaching professions of different occupations and backgrounds, such as maids, housewives, customer service personnel and non-professionals, etc. They are asked to read 3 to 10 sentences, such as "The North Wind and the Sun" in Aesop's fables. The sampling frequency of the recorded speech is 16kHz. The PEAED data set is manually annotated at both the phoneme and speaker levels. For phoneme-level annotation, the audio file is first automatically transcribed using the Mispronunciation Detection and Diagnosis (MDD) acoustic model, and then three trained annotators manually correct the errors in the automatic recording (using the Praat tool). The annotators are proficient people who use English as a second language, and they are asked to annotate the perceived phonemes based on looking at the spectrogram and listening. For speaker-level labeling, three annotators labeled each speaker as having a light accent (Light), a medium accent (Medium), or a heavy accent (Heavy). The final label was determined by a majority voting strategy from the results of the three annotators. The number of speakers labeled as having a light accent, a medium accent, and a heavy accent was 89, 84, and 75, respectively. However, the inter-annotator Kappa value (a measure of the consistency between annotators) was still very low, as shown in Table 2, which shows the inter-annotator kappa values for the PEAED corpus. This means that it is difficult for humans to easily and stably distinguish the level of a speaker's spoken pronunciation.
[0102] Annotator 1 Annotator 2 Annotator 3 Annotator 1 — 0.4190 0.3069 Annotator 2 0.4190 — 0.2900 Annotator 3 0.3069 0.2900 —
[0103] Table 2
[0104] Understandably, kappa values between 0.41-0.5 are considered moderate agreement, and between 0.21-0.4 are considered fair agreement.
[0105] Step S302: determining a corresponding sentence-level posterior probability distribution sample score according to the preset sample speech set.
[0106] It should be noted that the preset sample speech set includes multiple sentence-level speech signals from multiple speakers. The multiple sentence-level speech signals corresponding to each speaker are input into a preset acoustic model including the PDD algorithm to obtain the sentence-level posterior probability distribution scores corresponding to the sentence-level speech signals.
[0107] For example, in the preset sample speech set, housewife A is asked to read 5 sentences. Signal a in the 5 sentence-level speech signals is input into the preset acoustic model containing the PDD algorithm, and the standard phoneme q is calculated according to formula (1): i The phoneme-level posterior probability distribution score corresponding to the a signal:
[0108]
[0109] Where q is a preset standard phoneme sequence, q=[q 1 ,q 2 ,…,q M ]; Alignment segment The j-th feature frame; is the total number of frames in this segment; N is the number of preset standard phonemes q i The total number of segments.
[0110] Calculate the phoneme-level posterior probability distribution scores corresponding to all preset standard phonemes and signal a, take the average of all phoneme-level posterior probability distribution scores, obtain the sentence-level posterior probability distribution score corresponding to signal a, and obtain the sentence-level posterior probability distribution scores corresponding to the 5 sentences of housewife A in turn.
[0111] Step S303: Create a preset quality evaluation neural network.
[0112] It should be noted that the reference Figure 3 ,The preset quality evaluation neural network includes a preset shared hidden layer, an average layer, a spoken pronunciation level hidden layer and a preset classification prediction layer.
[0113] Step S304: inputting the sentence-level posterior probability distribution sample score into the preset quality assessment neural network to obtain the loss value of the preset quality assessment neural network.
[0114] It is understandable that the preset sample speech set contains artificial labels. After the sentence-level posterior probability distribution sample scores are input into the preset quality evaluation neural network, the labels predicted by the preset quality evaluation neural network can be obtained, and the loss value is calculated based on the artificial labels and the preset labels, so as to stabilize the preset quality evaluation neural network. The loss value of the preset quality evaluation neural network can be determined by the absolute value loss function, which calculates the absolute value of the difference between the predicted value and the target value; the loss value of the preset quality evaluation neural network can also be determined by the log loss function; the loss value of the preset quality evaluation neural network can also be determined by the cross entropy loss function.
[0115] Specifically, in order to make the weight update faster when the error is large, so as to quickly train the preset quality evaluation neural network, step S304 includes: inputting the sentence-level posterior probability distribution sample score into the preset shared hidden layer to obtain hidden output sample data; inputting the hidden output sample data into the average layer to obtain average sample data; inputting the average sample data into the spoken pronunciation level hidden layer to obtain target sample data; inputting the target sample data into the preset classification prediction layer to obtain a predicted label; determining a cross-entropy loss value based on the predicted label and the preset label; and determining a loss value of the preset quality evaluation neural network based on the cross-entropy loss value.
[0116] It should be noted that the preset labels are manually annotated labels in the preset sample speech set, and the cross entropy loss value is calculated according to formula (3):
[0117]
[0118] Where y is the preset label, is the predicted label returned by the softmax layer, n represents the total number of samples, and x represents the sample.
[0119] Further, determining the loss value of the preset quality assessment neural network according to the cross entropy loss value includes: determining the loss value of the preset quality assessment neural network according to the cross entropy loss value and a root mean square error loss value.
[0120] It should be understood that the root mean square error is the square root of the ratio of the square of the deviation between the predicted value and the true value to the number of observations. In the specific implementation, it can be calculated by the sentence-level posterior probability distribution score f i Its adjacent f i+1 Calculate the root mean square error to calculate the root mean square error loss value, or you can calculate the hidden output h of the shared hidden layer i With h i+1 The root mean square error between determines the root mean square error loss value.
[0121] Furthermore, in order to ensure that the input learning samples of the hidden layer remain unchanged, thereby making the classification results more accurate, before determining the loss value of the preset quality evaluation neural network according to the cross entropy loss value and the root mean square error loss value, the method also includes: calculating the pth root mean square error according to the pth hidden output sample data and the p+1th hidden output sample data, where 1≤p≤K-1, K is the total number of hidden output sample data; obtaining K-1 root mean square errors according to the pth root mean square error; obtaining the average value of K-1 root mean square errors according to the K-1 root mean square errors; and taking the average value of the K-1 root mean square errors as the root mean square error loss value.
[0122] It should be noted that after sharing the hidden layer output h, an auxiliary root mean square error loss term (RMSE) is used to calculate the root mean square error loss value. The total loss function is defined as formula (4):
[0123]
[0124] Where y is the preset label, is the predicted label returned by the softmax layer, h i is the shared hidden layer from the i-th sentence-level posterior probability distribution score f i The i-th hidden state returned, h i+1 is the shared hidden layer from the i+1th sentence-level posterior probability distribution score f i+1 The i+1th hidden state is returned. K represents the total number of hidden output sample data, that is, the number of input speech sentences.
[0125] Step S305: When the loss value is less than or equal to a preset loss threshold, the trained preset quality evaluation neural network is obtained.
[0126] It should be noted that when the loss value is greater than the preset loss threshold, the weights corresponding to each network layer are adjusted, and after multiple trainings, until the loss value is less than or equal to the preset loss threshold, the trained preset quality evaluation neural network is obtained.
[0127] The accuracy of the preset quality evaluation neural network proposed in this embodiment is evaluated, and the specific experimental process is as follows:
[0128] Multilayer Perception (MLP) was used as the basic classification model and compared with the SL-NN model proposed in this embodiment. The specific parameter settings were as follows: Adam optimizer was applied to MLP and SL-NN, and the corresponding batch sizes were 8 and 1, respectively. For MLP, the learning rate was set to 0.001, and the number of units in the first hidden layer was set to 50. For SL-NN, we used a learning rate of 0.0001 and the number of units in the shared hidden layer was 500. In both models, the first hidden layer used the ReLU activation function and the second hidden layer used the linear activation function. Finally, the output was activated by the softmax layer. Ten-fold cross validation (CV) was used to evaluate the performance of accent classification. In each ten-fold CV experiment, we randomly selected an additional validation set from the training set and stopped the training process early based on the validation loss. The final ratio of training, validation, and test sets was 8:1:1. The stability of the model was measured by the standard deviation calculated from multiple CV experiments of randomly sampling different sample sets from the dataset.
[0129] The relationship between the number of speech sentences used in classification and the corresponding model stability was verified experimentally. K = 1, 2, and 3 sentences of each speaker were randomly selected as training sets to obtain sentence-level features. In order to make the final classification results comparable between different K, speaker-level features extracted from all speech sentences of the speaker were always used during verification and testing. 50 ten-fold CV experiments were performed and the average accuracy and standard deviation were calculated for performance comparison.
[0130] Reference Figure 5 , Figure 5 This is a performance comparison diagram of different features of an embodiment of a method for evaluating spoken pronunciation quality of the present invention, from Figure 5 It can be clearly seen that using more sentences to train the MLP model leads to higher classification accuracy, and speaker-level features always have the best performance. In addition, PPD outperforms PS in all different settings. The PPD-based model can provide reasonable results when using only two or three spoken sentences, while the PS-based system requires more spoken sentences to achieve similar performance. In addition, the overall downward trend of the standard deviation shows that the classification results will be more stable as the number of spoken sentences increases.
[0131] Speaker Level Features Accuracy (%) Standard deviation (%) PR 50.77 1.52 PS 50.27 2.58 PPD 54.40 1.73 PR+PS 53.30 2.00 PR+PPD 55.58 1.83 PS+PPD 54.02 1.48 PR+PS+PPD 55.21 1.73
[0132] Table 3
[0133] Table 3 shows the performance of different speaker-level features and their combinations based on the MLP classifier. According to Table 3, PPD is better than both PS and PR because PPD can obtain more information from the acoustic model. Since PR explicitly provides the classification model with pronunciation errors related to certain English, its accuracy is slightly higher than PS. After combining with PR, the performance of both PS and PPD is improved, which means that the pronunciation error information from standard phonemes to other phonemes is very important for accent classification. It can also be observed that combining PS with PPD will reduce the accuracy (i.e., PS+PPD vs. PPD, PR+PS+PPD vs.PR+PPD), which may be because PPD and PR have a high degree of overlap in measuring pronunciation errors in standard phonemes. The combination of PPD and PR increases the redundancy of standard phoneme errors but ignores system pronunciation errors.
[0134] Furthermore, the performance of the MLP model (using PR, PS, and PPD, respectively) was compared based on the kappa values of humans and machines and the confusion matrix. Table 4 shows the kappa values of humans and machines for the PEAED speech library, and Table 5 shows the confusion matrix of the models based on PR, PS, and PPD features. According to Tables 4 and 5, compared with all annotators, the PPD-based model has a higher kappa value than PR and PS. The PR-based MLP system performs better on light and medium accents, while the PS-based MLP system performs better on heavy accents. Relatively speaking, the PPD-based system has the most balanced performance, mainly because PPD contains more detailed information about system pronunciation errors than PS and PR.
[0135]
[0136] Table 4
[0137]
[0138] Table 5
[0139] Refer to Table 6, which shows the performance comparison of the MLP and SL-NN models based on PPD. Experiments were conducted in three different environments: randomly selecting K=2, K=3 speech sentences for each speaker, and using all speech sentences as input to the model.
[0140]
[0141]
[0142] Table 6
[0143] According to Table 6, both MLP and SL-NN have improved performance when more speech sentences are used. SL-NN outperforms MLP in all three cases, especially when all speech sentences are used at the same time. This shows that the SL-NN structure proposed in this embodiment can better capture speaker-level information in multiple speech sentences. According to the comparison between SL-NN and SL-NN without RMSE, setting RMSE can improve the accuracy and stability of SL-NN.
[0144] This embodiment trains a preset quality evaluation neural network through a preset sample speech set to obtain a trained preset quality evaluation neural network, which not only considers the cross entropy loss value, but also considers the root mean square error loss value between the hidden output sample data obtained after multiple sentence-level posterior probability distribution sample scores of the same speaker pass through the preset shared hidden layer, thereby improving the accuracy and stability of the preset quality evaluation neural network. The quality of multiple sentence-level posterior probability distribution scores of the test person is evaluated through the trained preset quality evaluation neural network to obtain a more accurate evaluation result.
[0145] In addition, an embodiment of the present invention further provides a storage medium, on which a spoken pronunciation quality evaluation program is stored, and when the spoken pronunciation quality evaluation program is executed by a processor, the steps of the spoken pronunciation quality evaluation method described above are implemented.
[0146] Reference Figure 6 , Figure 6 It is a structural block diagram of the first embodiment of the oral pronunciation quality evaluation device of the present invention.
[0147] like Figure 6 As shown, the spoken pronunciation quality evaluation device proposed in the embodiment of the present invention includes:
[0148] The acquisition module 10 is used to acquire multiple sentence-level speech signals of the person to be tested.
[0149] It should be noted that sentence-level speech signals refer to speech signals collected from the person to be tested reading multiple sentences. In a specific implementation, in order to make the oral pronunciation quality evaluation smoother and more efficient, reading samples can be prepared in advance, and the person to be tested can be asked to read multiple sentences based on the reading samples. The speech collection device collects sentence-level speech signals and sends them to the acquisition module 10, thereby acquiring multiple sentence-level speech signals of the person to be tested. If the acquisition module 10 is provided with a microphone or a sound card, the acquisition module 10 can directly collect the speech signal of the person to be tested.
[0150] The score calculation module 20 is used to determine a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signal.
[0151] It should be noted that each sentence-level speech signal has a corresponding sentence-level posterior probability distribution score. Each sentence-level speech signal is input into a preset acoustic model containing a posterior probability distribution (PPD) algorithm to obtain a sentence-level posterior probability distribution score corresponding to the sentence-level speech signal.
[0152] Specifically, in order to obtain a sentence-level posterior probability distribution score, the score calculation module 20 is also used to obtain a preset standard phoneme sequence, which includes M preset standard phonemes; align the sentence-level speech signal with the preset standard phoneme sequence to obtain alignment segments corresponding to each preset standard phoneme, wherein the total number of alignment segments corresponding to the i-th preset standard phoneme is N, 1≤i≤M, and the t-th alignment segment includes R feature frames, 1≤t≤N; calculate the posterior probability distribution vector corresponding to the j-th feature frame according to the j-th feature frame and the preset standard factor sequence, wherein 1≤j ≤R; obtain R posterior probability distribution vectors according to the posterior probability distribution vector corresponding to the j-th feature frame; calculate the posterior probability distribution score corresponding to the t-th alignment segment according to the R posterior probability distribution vectors; obtain the posterior probability distribution scores corresponding to the N alignment segments according to the posterior probability distribution scores corresponding to the t-th alignment segment; calculate the sentence-level posterior probability distribution scores of the sentence-level speech signals one by one according to the posterior probability distribution scores corresponding to the N alignment segments.
[0153] It should be noted that there are two ways to calculate the sentence-level posterior probability distribution score. The first way is: calculate the sentence-level posterior probability distribution score corresponding to the sentence-level speech signal one by one according to the posterior probability distribution scores corresponding to the N aligned segments, including: calculating the average value according to the posterior probability distribution scores corresponding to the N aligned segments to obtain the phoneme-level posterior probability distribution score corresponding to the i-th preset standard phoneme; and using the phoneme-level posterior probability distribution score as the sentence-level posterior probability distribution score.
[0154] It can be understood that for each sentence-level speech signal, the sentence-level speech signal is aligned with a preset standard phoneme sequence, which includes M preset standard phonemes. In a specific implementation, the most representative preset standard factor is selected. For example, some people tend to misread the phoneme / f / as / p / and the phoneme / p / as / b / . Therefore, the i-th preset standard phoneme can be selected as / p / or / f / , thereby determining the phoneme-level posterior probability distribution score when the speaker misreads / f / as / p / or / p / as / b / , and the phoneme-level posterior probability distribution score is used as the sentence-level posterior probability distribution score.
[0155] Specifically, the second method is: calculating the sentence-level posterior probability distribution score corresponding to the sentence-level speech signal one-to-one according to the posterior probability distribution scores corresponding to the N aligned segments, including: calculating the average value according to the posterior probability distribution scores corresponding to the N aligned segments, and obtaining the posterior probability distribution score corresponding to the i-th preset standard phoneme; calculating the average value according to the M posterior probability distribution scores, and obtaining the sentence-level posterior probability distribution score corresponding to the sentence-level speech signal one-to-one.
[0156] It should be noted that the sentence-level posterior probability distribution score derived from the preset acoustic model contains detailed information about the speaker's pronunciation. In a specific implementation, if the performance of the preset acoustic model is good enough, the sentence-level posterior probability distribution score can accurately reflect the degree of closeness between the pronunciation and each standard phoneme, including not only the goodness of pronunciation (GOP) of the standard phoneme, but also the probability of misreading the standard phoneme as other phonemes. Take the case where the speaker misreads the standard phoneme / p / as / b / as an example:
[0157] According to formula (1), the standard phoneme q is calculated i The corresponding posterior probability distribution score:
[0158]
[0159] Where q is a preset standard phoneme sequence, q=[q 1 ,q 2 ,…,q M ]; Alignment segment The j-th feature frame; is the total number of frames in this segment; N is the number of preset standard phonemes q i The total number of segments.
[0160] It should be noted that when the N segments come from only one speech sentence, the concatenated posterior probability distribution score vector of all preset standard phonemes is defined as a sentence-level feature. The speaker-level feature refers to the average of the posterior probability distribution scores of all speech sentence segments belonging to a speaker. In other words, the posterior probability distribution scores corresponding to the M preset standard phonemes are averaged to obtain the sentence-level posterior probability distribution score corresponding to the sentence-level speech signal.
[0161] Let's take the calculation of the pronunciation score (PS) of each phoneme as an example for comparison:
[0162] PS can measure the goodness of pronunciation (GOP) of each phoneme in the accented pronunciation. In the specific implementation, the pronunciation score is used as a phoneme-level feature and then converted to a sentence-level feature through several schemes, such as calculating the minimum, mean, and standard deviation of the phoneme-level pronunciation scores of vowels, consonants, and syllables in each speech sentence. However, these schemes may ignore some detailed information of each phoneme.
[0163] Since the acoustic model outputs frame-level probability, but in order to compare with the PPD proposed in this embodiment, phoneme-level features are required, so the pronunciation score of each phoneme is calculated by first calculating the forced alignment between the accented speech and the standard phoneme, and then calculating the pronunciation score of each phoneme. The pronunciation score is calculated by formula (2):
[0164]
[0165] in, Alignment segment The j-th feature frame; is the total number of frames in this segment; N is the number of preset standard phonemes q i The total number of segments.
[0166] It is understandable that although PS can describe the correctness of each standard phoneme, it cannot distinguish between individual pronunciation errors and systematic pronunciation errors, for example, a phoneme in English is pronounced as another phoneme that does not exist in English. Pronunciation errors can be divided into two major categories: (1) The first category includes individual pronunciation errors, such as unintentional mispronunciation of phonemes or words; (2) The second category of errors includes the substitution of specific phonemes, such as phonemes that exist in the native language but not in the target language. The second category of pronunciation errors is called systematic pronunciation errors.
[0167] It should be noted that, compared with the PS method, the PPD method adopted in this embodiment takes into account all preset standard phonemes when calculating the posterior probability distribution vector, while the PS method only considers the i-th preset standard phoneme. The PPD method adopted in this embodiment better describes individual pronunciation errors and system pronunciation errors.
[0168] Take the Phonological Rule (PR) as an example for comparison:
[0169] PR is a way of expressing the phonological or morphological processes of a system or the diachronic changes in a language. It can be defined as a mapping between two different levels of sound representation (substratum and surface), or it can be described as a "generalization" about how sounds are pronounced differently in different contexts. Taking some English accents as an example, PR specifically refers to common pronunciation errors among some people. For example, the / p / in "poor" is mispronounced as / b / and thus sounds like "bore".
[0170] British Phonemes Mispronounced phonemes f p p b,f ae aa,ah ch,zh sh
[0171] Table 1
[0172] Table 1 is an example table of common pronunciation errors of some people. Table 1 lists some common pronunciation errors that some people make when pronouncing British English. In the specific implementation, 36 1-to-1 speech rules are used, and the triggering times of each speech rule in all speech sentences are counted as speaker-level features. PR only contains speaker-level features and no sentence-level signs. Therefore, there is a difference between using PR and the PPD proposed in this embodiment. PPD contains more detailed system pronunciation error information than PR.
[0173] The evaluation module 30 is used to perform quality evaluation on the sentence-level posterior probability distribution score through a trained preset quality evaluation neural network to obtain an evaluation result of the oral pronunciation quality of the person to be tested.
[0174] It should be noted that the preset quality evaluation neural network can be a recurrent neural network (RNN). This embodiment adopts a separately designed speaker-level neural network (SL-NN) based on multiple speech sentences of the speaker, which includes a shared hidden layer, an average layer, a hidden layer and a preset classification prediction layer.
[0175] Specifically, the evaluation module 30 is also used to input the sentence-level posterior probability distribution score into a preset shared hidden layer to obtain hidden output feature data, wherein the trained preset quality evaluation neural network includes a preset shared hidden layer, an average layer, a spoken pronunciation level hidden layer and a preset classification prediction layer; the hidden output feature data is input into the average layer to obtain average feature data; the average feature data is input into the spoken pronunciation level hidden layer to obtain target feature data; the target feature data is input into the preset classification prediction layer to obtain a target label; and the oral pronunciation quality evaluation result of the person to be tested is obtained according to the target label.
[0176] Reference Figure 3 , Figure 3 The schematic diagram of the preset quality evaluation neural network structure of an embodiment of the oral pronunciation quality evaluation method of the present invention is shown in FIG. The preset quality evaluation neural network includes a preset shared hidden layer, an average layer, an oral pronunciation level hidden layer and a preset classification prediction layer.
[0177] It can be understood that the shared hidden layer is a sentence-level hidden layer, which is shared between sentences, the spoken pronunciation level hidden layer is a speaker-level hidden layer, and the preset classification prediction layer is a softmax layer. Where f represents the sentence-level PPD, and h represents the hidden output of the shared hidden layer. The hidden output h is averaged before being input to the speaker-level hidden layer.
[0178] It should be understood that the above is only an example and does not constitute any limitation on the technical solution of the present invention. In specific applications, technicians in this field can make settings as needed, and the present invention does not limit this.
[0179] This embodiment obtains multiple sentence-level speech signals of the person to be tested; determines a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signals; and performs quality evaluation on the sentence-level posterior probability distribution score through a trained preset quality evaluation neural network to obtain the oral pronunciation quality evaluation result of the person to be tested. In the above manner, the sentence-level posterior probability distribution score is determined according to the multiple sentence-level speech signals of the speaker. The multiple sentence-level posterior probability distribution scores are speaker-level features, which not only include detailed information and pronunciation excellence of the speaker's pronunciation, but also include the probability of misreading standard factors as other factors, and are more explanatory: they explain individual pronunciation errors and system pronunciation errors, thereby providing a data basis for the preset quality evaluation neural network, and referencing the trained preset quality evaluation neural network to perform quality evaluation on multiple sentence-level posterior probability distribution scores, thereby improving the accuracy and stability of accent classification.
[0180] In one embodiment, the acquisition module 10 is further used to acquire a preset sample speech set;
[0181] Determining a corresponding sentence-level posterior probability distribution sample score according to the preset sample speech set;
[0182] Create a preset quality assessment neural network;
[0183] Inputting the sentence-level posterior probability distribution sample score into the preset quality evaluation neural network to obtain the loss value of the preset quality evaluation neural network;
[0184] When the loss value is less than or equal to a preset loss threshold, the trained preset quality evaluation neural network is obtained.
[0185] In one embodiment, the acquisition module 10 is further used to input the sentence-level posterior probability distribution sample scores into a preset shared hidden layer to obtain hidden output sample data;
[0186] Inputting the hidden output sample data into the average layer to obtain average sample data;
[0187] Inputting the average sample data into a spoken pronunciation level hidden layer to obtain target sample data;
[0188] Input the target sample data into a preset classification prediction layer to obtain a prediction label;
[0189] Determine a cross entropy loss value according to the predicted label and the preset label;
[0190] The loss value of the preset quality evaluation neural network is determined according to the cross entropy loss value.
[0191] In one embodiment, the acquisition module 10 is further used to determine the loss value of the preset quality assessment neural network according to the cross entropy loss value and the root mean square error loss value.
[0192] In one embodiment, the acquisition module 10 is further used to calculate the pth root mean square error according to the pth hidden output sample data and the p+1th hidden output sample data, wherein 1≤p≤K-1, K is the total number of hidden output sample data;
[0193] According to the pth root mean square error, K-1 root mean square errors are obtained;
[0194] The average value of K-1 root mean square errors is obtained according to the K-1 root mean square errors;
[0195] The average value of the K-1 root mean square errors is taken as the root mean square error loss value.
[0196] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0197] In addition, for technical details not described in detail in this embodiment, reference can be made to the spoken pronunciation quality evaluation method provided in any embodiment of the present invention, and will not be repeated here.
[0198] In addition, it should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0199] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0200] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory (ROM) / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0201] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
Claims
1. A method for evaluating the quality of spoken pronunciation, characterized in that: The spoken pronunciation quality evaluation method comprises: Acquire multiple sentence-level speech signals of the person to be tested; Determining a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signal; The sentence-level posterior probability distribution score is subjected to quality evaluation by a trained preset quality evaluation neural network to obtain a spoken pronunciation quality evaluation result of the person to be tested; Before the sentence-level posterior probability distribution score is subjected to quality evaluation through a trained preset quality evaluation neural network to obtain the oral pronunciation quality evaluation result of the person to be tested, the method further comprises: Get the preset sample voice set; Determining a corresponding sentence-level posterior probability distribution sample score according to the preset sample speech set; Creating a preset quality assessment neural network, wherein the preset quality assessment neural network is a speaker-level neural network model based on multiple speech sentences of a speaker, and the input of the preset quality assessment neural network is a sentence-level posterior probability distribution score; Inputting the sentence-level posterior probability distribution sample score into the preset quality evaluation neural network to obtain the loss value of the preset quality evaluation neural network; When the loss value is less than or equal to a preset loss threshold, the trained preset quality evaluation neural network is obtained; The step of inputting the sentence-level posterior probability distribution score into a trained preset quality evaluation neural network to obtain the oral pronunciation quality evaluation result of the person to be tested includes: Inputting the sentence-level posterior probability distribution score into a preset shared hidden layer to obtain hidden output feature data, the trained preset quality evaluation neural network includes a preset shared hidden layer, an average layer, a spoken pronunciation level hidden layer and a preset classification prediction layer, wherein the preset shared hidden layer is a sentence-level hidden layer, and the spoken pronunciation level hidden layer is a speaker-level hidden layer; Inputting the hidden output feature data into an average layer to obtain average feature data; Inputting the average feature data into a spoken pronunciation level hidden layer to obtain target feature data; Inputting the target feature data into a preset classification prediction layer to obtain a target label; The oral pronunciation quality evaluation result of the person to be tested is obtained according to the target label.
2. The method for evaluating the quality of spoken pronunciation as claimed in claim 1, wherein: The step of inputting the sentence-level posterior probability distribution sample score into the preset quality evaluation neural network to obtain the loss value of the preset quality evaluation neural network includes: Inputting the sentence-level posterior probability distribution sample scores into a preset shared hidden layer to obtain hidden output sample data; Inputting the hidden output sample data into the average layer to obtain average sample data; Inputting the average sample data into a spoken pronunciation level hidden layer to obtain target sample data; Input the target sample data into a preset classification prediction layer to obtain a prediction label; Determine a cross entropy loss value according to the predicted label and the preset label; The loss value of the preset quality evaluation neural network is determined according to the cross entropy loss value.
3. The method for evaluating the quality of spoken pronunciation as claimed in claim 2, wherein: The determining the loss value of the preset quality evaluation neural network according to the cross entropy loss value includes: The loss value of the preset quality evaluation neural network is determined according to the cross entropy loss value and the root mean square error loss value.
4. The method for evaluating the quality of spoken pronunciation as claimed in claim 3, wherein: Before determining the loss value of the preset quality evaluation neural network according to the cross entropy loss value and the root mean square error loss value, the method further includes: Calculate the pth root mean square error according to the pth hidden output sample data and the p+1th hidden output sample data, where 1≤p≤K-1, and K is the total number of hidden output sample data; According to the pth root mean square error, K-1 root mean square errors are obtained; The average value of K-1 root mean square errors is obtained according to the K-1 root mean square errors; The average value of the K-1 root mean square errors is taken as the root mean square error loss value.
5. The method for evaluating the quality of spoken pronunciation according to any one of claims 1 to 4, characterized in that: The step of determining a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signal comprises: Obtaining a preset standard phoneme sequence, wherein the preset standard phoneme sequence includes M preset standard phonemes; Aligning the sentence-level speech signal with the preset standard phoneme sequence to obtain alignment segments corresponding to each preset standard phoneme, wherein the total number of alignment segments corresponding to the i-th preset standard phoneme is N, 1≤i≤M, and the t-th alignment segment contains R feature frames, 1≤t≤N; Calculate the posterior probability distribution vector corresponding to the j-th feature frame according to the j-th feature frame and the preset standard factor sequence, wherein 1≤j≤R; Obtain R posterior probability distribution vectors according to the posterior probability distribution vector corresponding to the j-th feature frame; Calculate the posterior probability distribution score corresponding to the t-th alignment segment according to the R posterior probability distribution vectors; Obtaining posterior probability distribution scores corresponding to N alignment segments according to the posterior probability distribution score corresponding to the t-th alignment segment; The sentence-level posterior probability distribution scores of the sentence-level speech signals are calculated one-to-one according to the posterior probability distribution scores corresponding to the N aligned segments.
6. A device for evaluating the quality of spoken pronunciation, characterized in that: The spoken pronunciation quality evaluation device comprises: An acquisition module, used to acquire multiple sentence-level speech signals of the person to be tested; A score calculation module, used to determine a one-to-one corresponding sentence-level posterior probability distribution score according to the sentence-level speech signal; An evaluation module, used for performing quality evaluation on the sentence-level posterior probability distribution score through a trained preset quality evaluation neural network to obtain an evaluation result of the oral pronunciation quality of the person to be tested; The score calculation module is also used to obtain a preset sample speech set; determine the corresponding sentence-level posterior probability distribution sample score according to the preset sample speech set; create a preset quality evaluation neural network, wherein the preset quality evaluation neural network is a speaker-level neural network model based on multiple speech sentences of the speaker, and the input of the preset quality evaluation neural network is the sentence-level posterior probability distribution score; input the sentence-level posterior probability distribution sample score into the preset quality evaluation neural network to obtain the loss value of the preset quality evaluation neural network; when the loss value is less than or equal to the preset loss threshold, obtain the trained preset quality evaluation neural network; The evaluation module is also used to input the sentence-level posterior probability distribution score into a preset shared hidden layer to obtain hidden output feature data, wherein the trained preset quality evaluation neural network includes a preset shared hidden layer, an average layer, a spoken pronunciation level hidden layer and a preset classification prediction layer, wherein the preset shared hidden layer is a sentence-level hidden layer, and the spoken pronunciation level hidden layer is a speaker-level hidden layer; input the hidden output feature data into the average layer to obtain average feature data; input the average feature data into the spoken pronunciation level hidden layer to obtain target feature data; input the target feature data into the preset classification prediction layer to obtain a target label; and obtain the oral pronunciation quality evaluation result of the person to be tested according to the target label.
7. A device for evaluating the quality of spoken pronunciation, characterized in that: The device comprises: a memory, a processor, and a spoken pronunciation quality evaluation program stored in the memory and executable on the processor, wherein the spoken pronunciation quality evaluation program is configured to implement the steps of the spoken pronunciation quality evaluation method according to any one of claims 1 to 5.
8. A storage medium, characterized in that: The storage medium stores a spoken language pronunciation quality evaluation program, and when the spoken language pronunciation quality evaluation program is executed by the processor, the steps of the spoken language pronunciation quality evaluation method according to any one of claims 1 to 5 are implemented.
Citation Information
Patent Citations
Spoken language pronunciation detecting and evaluating method based on deep neural network posterior probability algorithm
CN104575490A
Spoken language pronunciation evaluation method based on deep neural network posterior probability algorithm
CN108364634A
Data filling method and device based on landmarks
CN111177135A
