A method, device, medium and equipment for speech evaluation

Abstract

The present application discloses a speech evaluation method, which includes: obtaining a speech to be evaluated and a reference text, extracting acoustic features from the speech, inputting the acoustic features into K sub-acoustic models of an acoustic model, and obtaining a comprehensive state posterior probability; aligning the phonemes of the speech with the reference text according to the comprehensive state posterior probability and a segmentation network constructed by the reference text, and obtaining a scoring feature based on the alignment result; and inputting the scoring feature into a scoring model to regress the score of the speech. Since the K sub-acoustic models are trained by their own in-set data and out-of-set data, and the out-of-set data of each sub-acoustic model is different, the acoustic model has a distinguishing feature for the out-of-set data, thereby improving the scoring accuracy of abnormal speech, solving the problem of unstable scoring of abnormal speech, and having high reliability and availability.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech evaluation method, apparatus, storage medium and device. Background Art

[0002] With the deepening of education reform and the increasing requirements of enterprises and other institutions for language proficiency, more and more people are committed to improving their oral expression skills and trying to obtain corresponding certificates by participating in oral examinations. Among them, reading aloud questions are the most common type of questions in oral examinations, so the accuracy of scoring reading aloud questions is crucial.

[0003] In recent years, speech recognition technology has achieved remarkable results in many fields. Therefore, many researchers have tried to apply speech recognition to oral examinations to automatically evaluate the speech of test takers, thereby reducing the workload of teachers.

[0004] At present, the scoring process of reading aloud questions in oral exams is mainly divided into the following stages: the first stage is to perform complex network forced segmentation and alignment of the reading text and the candidate's voice based on the pre-trained acoustic model to obtain information such as added and missed readings, word and phoneme-level time boundaries, and the state posterior probability of each frame; the second stage is to extract scoring features related to completeness, fluency, and pronunciation accuracy based on the forced segmentation results; the third stage is to detect abnormal speech based on normal speech (such as speech read normally according to the text) and abnormal speech (such as speech not read according to the text) using a binary classification model trained with scoring features; the fourth stage is to train a scoring feature regression model based on manual scores, and regress the machine scores in the testing phase.

[0005] However, the training data of the acoustic model is usually obtained by transcribing normal speech. The acoustic model has never seen abnormal speech during the training process, resulting in unknown classification effect of the acoustic model for abnormal speech. Therefore, the scoring features extracted based on the acoustic model will be unstable in terms of discrimination, which in turn leads to uncertainty in the scoring results of reading aloud questions. How to provide a speech evaluation method to obtain accurate scoring has become a key issue in the industry. Summary of the invention

[0006] The main purpose of the embodiments of the present application is to provide a speech evaluation method, system, computing device cluster, computer-readable storage medium and computer program product, which can improve the accuracy of abnormal speech scoring, solve the problem of unstable abnormal speech scoring, and have high reliability and availability.

[0007] The present application provides a method for speech evaluation, including:

[0008] Obtain the speech to be evaluated and the reference text;

[0009] Extracting acoustic features from the speech, inputting the acoustic features into K acoustic sub-models of the acoustic model, and obtaining a comprehensive state posterior probability, wherein the K acoustic sub-models are trained by respective in-set data and out-of-set data, and each acoustic sub-model has different out-of-set data;

[0010] According to the comprehensive state posterior probability and the segmentation network constructed by the reference text, aligning the phonemes of the speech with the reference text, and obtaining a scoring feature based on the alignment result;

[0011] The scoring features are input into a scoring model to regress the score of the speech.

[0012] The present application also provides a speech evaluation system, including:

[0013] An acquisition module, used to acquire the speech to be evaluated and the reference text;

[0014] An acoustic processing module, used for extracting acoustic features from the speech, inputting the acoustic features into K acoustic sub-models of the acoustic model, and obtaining a posterior probability of the comprehensive state, wherein the K acoustic sub-models are trained by respective in-set data and out-of-set data, and the out-of-set data of each acoustic sub-model is different;

[0015] A post-processing module, configured to align the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, and obtain a scoring feature based on the alignment result;

[0016] The evaluation module is used to input the scoring feature into the scoring model to regress the score of the speech.

[0017] An embodiment of the present application further provides a computing device cluster, the computing device cluster includes at least one computing device, and the at least one computing device includes: a processor, a memory, and a system bus.

[0018] Wherein, the processor and the memory are connected via the system bus;

[0019] The memory is used to store one or more programs, and the one or more programs include instructions, which, when executed by the processor, enable the computing device cluster to perform any one of the implementations of the above-mentioned speech evaluation method.

[0020] The embodiment of the present application further provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a computing device cluster, the computing device cluster executes any one of the implementations of the above method.

[0021] The embodiment of the present application further provides a computer program product. When the computer program product is run on a computing device cluster, the computing device cluster executes any one of the implementations of the above-mentioned speech evaluation method.

[0022] A speech evaluation method is provided in an embodiment of the present application. Specifically, the speech evaluation system obtains the speech to be evaluated and a reference text (for example, a read text), extracts acoustic features from the speech, inputs the acoustic features into K sub-acoustic models of the acoustic model, obtains the comprehensive state posterior probability, aligns the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, obtains the scoring features based on the alignment result, and inputs the scoring features into the scoring model to regress the score of the speech.

[0023] This method draws on the idea of ​​K-fold cross-validation to obtain the in-set data and out-of-set data of K sub-acoustic models, and the out-of-set data of each sub-acoustic model is different. By training the sub-acoustic models with their own in-set data and out-of-set data, the acoustic models can be distinguished from the out-of-set data, thereby improving the accuracy of abnormal speech scoring, solving the problem of unstable abnormal speech scoring, and having high reliability and availability. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0025] Figure 1 A system architecture diagram of a speech evaluation method provided in an embodiment of the present application;

[0026] Figure 2 A flowchart of a speech evaluation method provided in an embodiment of the present application;

[0027] Figure 3 A schematic diagram of a process for training an acoustic model provided in an embodiment of the present application;

[0028] Figure 4 A schematic diagram of the structure of a speech evaluation system provided in an embodiment of the present application;

[0029] Figure 5 A schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The terms "first" and "second" in the embodiments of the present application are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features.

[0031] First, some technical terms involved in the embodiments of the present application are introduced.

[0032] Speech recognition is also known as Automatic Speech Recognition (ASR), computer speech recognition, or speech-to-text. Speech recognition is used to process speech (such as human speech) into text in a written format.

[0033] Speech recognition has achieved remarkable results in many fields. Therefore, many researchers have tried to apply speech recognition to oral exams to automatically evaluate the speech of test takers, thereby reducing the workload of teachers. Similarly, speech recognition can also be applied to oral training to evaluate the speech during training, thereby assisting oral learning.

[0034] Oral exams or oral training include reading aloud questions. Reading aloud questions are questions that instruct students to read aloud according to the text. At present, the scoring process of reading aloud questions is mainly divided into the following stages: the first stage is to perform complex network forced segmentation and alignment of the reading text and the candidate's voice based on the pre-trained acoustic model to obtain information such as added and missed readings, word and phoneme-level time boundaries, and the state posterior probability of each frame; the second stage is to extract scoring features related to completeness, fluency, and pronunciation accuracy based on the forced segmentation results; the third stage is to detect abnormal speech based on normal speech (such as speech that is read normally according to the text) and abnormal speech (such as speech that is not read according to the text) using a binary classification model trained with scoring features; the fourth stage is to train a scoring feature regression model based on manual scores, and regress the machine scores in the test stage.

[0035] However, the training data of the acoustic model is usually transcribed from normal speech, and the acoustic model has not seen abnormal speech during the training process, resulting in unknown classification effect of the acoustic model for abnormal speech. Therefore, the scoring features extracted based on the acoustic model will be unstable in terms of discrimination, which in turn leads to uncertainty in the scoring results of reading aloud questions.

[0036] In order to solve the above problems, an embodiment of the present application provides a method for speech evaluation. The method can be performed by a speech evaluation system. In some embodiments, the speech evaluation system can be a software system, and a computing device or a computing device cluster executes the speech evaluation method by running the program code of the software system. In other embodiments, the speech evaluation system can also be a hardware system for speech evaluation. The embodiment of the present application is illustrated by taking the speech evaluation system as a software system.

[0037] Specifically, the speech evaluation system obtains the speech to be evaluated and a reference text (for example, a read text), extracts acoustic features from the speech, inputs the acoustic features into K sub-acoustic models of the acoustic model, obtains the comprehensive state posterior probability, aligns the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, obtains scoring features based on the alignment result, and inputs the scoring features into the scoring model to regress the score of the speech.

[0038] This method draws on the idea of ​​K-fold cross-validation, divides the state categories of the output layer of the acoustic model into K groups, and obtains the in-set data and out-of-set data of K sub-acoustic models based on the K groups of state categories, and the out-of-set data of each sub-acoustic model is different. Using their own in-set and out-of-set data to train the sub-acoustic models can make the acoustic models distinguishable from the out-of-set data, thereby improving the accuracy of abnormal speech scoring, solving the problem of unstable abnormal speech scoring, and having high reliability and availability.

[0039] In order to make the technical solution of the present application clearer and easier to understand, the application scenarios of the present application are introduced below with reference to the accompanying drawings.

[0040] See also Figure 1 The application scenario diagram of the speech evaluation method shown in FIG. Figure 1 As shown, a communication connection is established between the server 10 and the terminal 20. The communication connection may be realized, for example, by a wired network connection or a wireless network connection.

[0041] The server 10 is equipped with a speech evaluation system ( Figure 1(not shown) to provide voice evaluation services. Terminal 20 has a recording function, which is used to present the test paper of the oral test to the user and collect the user's answer voice for the test paper. Among them, the oral test can be an English oral test or a Mandarin oral test. This embodiment is illustrated by an English oral test. Specifically, the test paper includes reading questions. Terminal 20 can present the reading text corresponding to the reading question to the user, and then collect the user's voice when reading according to the reading text. Terminal 20 can use the collected voice as the voice to be evaluated and the reading text as the reference text to send to server 10, and the server 10 will evaluate it through the voice evaluation system.

[0042] Specifically, after obtaining the speech to be evaluated and the reference text, the speech evaluation system in the server 10 extracts acoustic features from the speech, inputs the acoustic features into K sub-acoustic models of the acoustic model, obtains the comprehensive state posterior probability, and then aligns the phonemes of the speech with the reference text based on the comprehensive state posterior probability and the segmentation network constructed by the reference text, obtains the scoring features based on the alignment results, and then inputs the scoring features into the scoring model to regress the score of the speech.

[0043] Furthermore, the server 10 may also return the score of the speech to the terminal 20, so that the terminal 20 may present the score of the speech to the user.

[0044] It should be noted that Figure 1 The voice evaluation system is deployed on a server for illustration only. In other possible implementations of the embodiment of the present application, the voice evaluation system may also be deployed on a terminal, or on a computing device cluster formed by multiple computing devices, and the present embodiment does not impose any restrictions on this.

[0045] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] See also Figure 2 The flowchart of the speech evaluation method shown in FIG. 1 comprises the following steps:

[0047] S202: The speech evaluation system obtains the speech to be evaluated and the reference text.

[0048] The speech to be evaluated refers to the speech when the user (such as a speaking test participant or a speaking student) reads aloud according to the reference text. The read text is a preset text, such as the text in the speaking test paper. The reference text can be an English text or a Chinese text, and accordingly, the speech to be evaluated can be an English voice or a Mandarin voice, which is not limited in this embodiment.

[0049] S204: The speech evaluation system extracts acoustic features from the speech, inputs the acoustic features into K sub-acoustic models of the acoustic model, and obtains a posterior probability of the comprehensive state.

[0050] The acoustic features may be any one or more of Linear Predictive Cepstral Coding (LPCC) features, FilterBank (fBank) features, Mel-frequency CeptraICoefficients (MFCC) features, or Perceptual Linear Predictive (PLP) features.

[0051] The speech evaluation system can extract corresponding features from the speech through an acoustic feature extraction algorithm. The following is an example of extracting MFCC features. Specifically, the speech evaluation system can first pre-process the speech, such as framing, pre-emphasis and windowing, so that multiple speech frames can be obtained, and then the speech evaluation system can convert the multiple speech frames from the time domain to the frequency domain. For example, the speech evaluation system can perform a fast Fourier transform (FFT) on multiple speech frames. The feature obtained by FFT is a complex matrix. The complex matrix represents an energy spectrum. Since the phase spectrum in the energy spectrum contains very little information, the speech evaluation system can discard the phase spectrum and retain the amplitude spectrum by squaring or finding the absolute value of the complex numbers in the complex matrix. The speech evaluation system can perform Mel filtering on the amplitude spectrum, thereby converting the linear frequency into a nonlinearly distributed Mel frequency. The feature after Mel filtering is also called fBank feature.

[0052] Since the human ear's perception of sound grows in pairs, the speech evaluation system can perform a logarithmic operation on the fBank feature to simulate the human ear's perception. Furthermore, the speech evaluation system can perform a discrete cosine transform (DCT) after the logarithmic operation to eliminate the correlation between different orders, map the signal to a low-dimensional space, and obtain the MFCC feature.

[0053] The above MFCC features are static, and the speech evaluation system can also perform differential operations on the static MFCC features to obtain dynamic MFCC features. The speech evaluation system can combine the static MFCC features with the dynamic MFCC features to improve recognition performance.

[0054] The state category of the output layer of the acoustic model refers to the hidden state of tri-phone modeling based on the Hidden Markov Model (HMM) in speech recognition. For example, the state category may include the hidden state "cup" modeled by the phonemes / k / , / uh / , and / p / . When the user utters the speech "would u like a cup of tea", the state category of a speech frame of the speech may be the above "cup".

[0055] The state categories of the output layer of the acoustic model can be divided into K groups. Among them, the state categories can be evenly divided into K groups. When grouping, the grouping can be performed randomly or according to the phoneme type. For example, English can be grouped according to vowels and consonants, and Mandarin can be grouped according to initials and finals.

[0056] The acoustic model has K sub-acoustic models. The K acoustic sub-models are trained by their own in-set data and out-of-set data. The out-of-set data of each sub-acoustic model is data in the data set whose state category belongs to one of the K groups. The out-of-set data of each sub-acoustic model is different. The in-set data of each sub-acoustic model is data in the data set whose state category belongs to the remaining K-1 groups in the K groups. The label of the in-set data is a vector of state categories, such as a one-hot vector. The out-of-set data has no corresponding state category, and its label is set to a uniform distribution of the state categories of the corresponding group.

[0057] Based on this, after the speech evaluation system inputs the acoustic features into the K acoustic sub-models of the acoustic model, for each state type, K-1 acoustic sub-models among the K acoustic sub-models can output the posterior probability of the state type. The speech evaluation system can obtain the comprehensive state posterior probability based on the K-1 posterior probabilities. For example, the speech evaluation system can determine the average value of the K-1 posterior probabilities as the comprehensive state posterior probability.

[0058] S206: The speech evaluation system aligns the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, and obtains scoring features based on the alignment result.

[0059] The segmentation network is used for forced segmentation and alignment of speech and text. The segmentation network can segment the reference text to obtain multiple possible reading paths of the reference text, such as the normal reading path, word or sentence addition, omission, rereading or sentence rereading path. Generally, the segmentation network can be constructed based on the set rules to obtain the reference text.

[0060] Forced segmentation alignment refers to using the Viterbi algorithm to find the optimal reading path based on the state posterior probability output by the segmentation network and the acoustic model, thereby achieving alignment between speech and reference text according to the optimal reading path.

[0061] Among them, the Viterbi algorithm is a dynamic programming algorithm. The algorithm is used to find the Viterbi path that is most likely to produce an observed event sequence (or called an observed event sequence), and the Viterbi path is represented by a hidden state sequence (or called a hidden state sequence). Specifically in the speech evaluation scenario, speech (for example, a phoneme sequence in speech) can be regarded as an observed event sequence, and a character sequence in a reference text can be regarded as a hidden state sequence. Therefore, the speech evaluation system can apply the Viterbi algorithm to the speech to find the most likely hidden state sequence, thereby achieving alignment between the speech and the reference text.

[0062] Specifically, the speech evaluation system can determine the target reading path through the Viterbi algorithm based on the comprehensive state posterior probability and the segmentation network constructed by the reference text. The target reading path can be, for example, the above-mentioned optimal reading path. Then, the speech evaluation system can align the phonemes of the speech with the reference text according to the target reading path to obtain the time boundary information of the phonemes.

[0063] When evaluating speech, it can be evaluated from different dimensions. Accordingly, the scoring features may include one or more of the following: goodness of pronunciation (GOP), word addition and omission ratio, and average phoneme pronunciation duration.

[0064] In this embodiment, the speech evaluation system can further calculate scoring features such as pronunciation goodness GOP, word addition and omission ratio, average phoneme pronunciation duration, etc. based on the time boundary information of phonemes to measure the user's pronunciation accuracy, completeness and fluency.

[0065] GOP is calculated based on the time boundary information of the phoneme and the comprehensive state posterior probability, and is used to evaluate the correctness of the user's pronunciation. The GOP calculation formula is as follows:

[0066]

[0067]

[0068] Where p is a phoneme, o is an observation vector (also called an observation vector), and each observation vector is generated by a state sequence with a corresponding probability density distribution. Q is the set of all phonemes, p(o|p) is the likelihood probability of phoneme p, S i is the i-th possible state sequence, the state sequence is found by the Viterbi algorithm, s i,t is the tri-phone state corresponding to time t.

[0069] The speech evaluation system can also determine at least one of the words that are read in addition and the words that are omitted based on the time boundary information of the phonemes. For example, the speech evaluation system can determine the word reading ratio based on the number of words read in addition and the total number of words, and determine the word omission ratio based on the number of words omitted and the total number of words. In addition, the speech evaluation system can also determine the pronunciation duration of each phoneme by the user based on the time boundary information of the above factors, and based on the pronunciation duration, the average phoneme pronunciation duration can be determined.

[0070] In this embodiment, the comprehensive state posterior probability is discriminative, and the scoring features such as GOP determined based on the comprehensive state posterior probability are also discriminative. The scoring features are used for speech evaluation with high accuracy.

[0071] S208: The speech evaluation system inputs the scoring features into a scoring model to regress the score of the speech.

[0072] The scoring model is a model for evaluating speech. The scoring model takes the scoring features of speech as input, such as the GOP of speech, the ratio of word addition and omission, and the average phoneme pronunciation duration as input, and takes the score of speech as output. Among them, the scoring model is usually a regression model, such as any one of a linear regression model, a decision tree regression model, a Gaussian regression model, or a neural network regression model. Accordingly, the scoring model can output the score of speech by regression, which is also called a machine score.

[0073] The scoring model can be obtained by training the initialized regression model based on the scoring features extracted from the data set and the corresponding manual scores (specifically the manually annotated scores). The speech evaluation system can use the data set to pre-train the scoring model, and then use the scoring model to perform reasoning in the inference stage, so as to achieve speech evaluation.

[0074] Based on the above description, an embodiment of the present application provides a method for speech evaluation. The method draws on the idea of ​​K-fold cross-validation, divides the state categories of the output layer of the acoustic model into K groups, and K sub-acoustic models are trained by their own in-set data and out-of-set data. The out-of-set data of each sub-acoustic model is the data in the data set whose state category belongs to one of the K groups, and the out-of-set data of each sub-acoustic model is different. The in-set data of each sub-acoustic model is the data in the data set whose state category belongs to the remaining K-1 groups in the K group. In this way, the acoustic model is discriminative for out-of-set data, thereby improving the accuracy of abnormal speech scoring, solving the problem of unstable abnormal speech scoring, and having high reliability and availability.

[0075] The speech evaluation method provided in the embodiment of the present application relies on an acoustic model, which can be pre-trained. The training process of the acoustic model is introduced below.

[0076] See also Figure 3 The flowchart of training an acoustic model is shown in FIG. 1 , wherein the method comprises:

[0077] S302: The speech evaluation system obtains a data set.

[0078] Each sample data in the data set has K labels, of which K-1 labels are vectors of the state categories corresponding to the sample data, such as one-hot vectors, and the remaining 1 label is a uniform distribution of the state categories.

[0079] Specifically, the user can divide the state categories of the output layer of the acoustic model into K groups. Assuming that the number of state categories of the output layer is N, the user can divide the state categories into K groups evenly, each group including N / K state categories. When grouping, the user can use a random method or group by phoneme. Then, drawing on the idea of ​​K-fold cross-validation, one group of state categories is eliminated in turn, and the state categories included in the remaining K-1 groups form a state category set. Finally, K state category sets can be constructed, and each state category is included in the K-1 state category set.

[0080] Next, the user can trigger the operation of generating labels for the sample data in the dataset based on each state category set. Taking one of the state category sets as an example, the sample data in the dataset can be divided into in-set data and out-of-set data. In-set data refers to sample data (such as speech frames) whose state categories are in the state category set, and out-of-set data refers to sample data whose state categories are not in the state category set. For the in-set data, determine the state category of the in-set data, and use the one-hot vector of the state category as the label. For the out-of-set data, since its state category cannot be determined, the uniform distribution of the state categories in the current group can be used as the label.

[0081] It should be noted that the data set in this embodiment may include not only sample data generated by normal speech, but also sample data generated by abnormal speech. Abnormal speech includes speech that is not read according to the reference text and / or speech with abnormal sound quality. Taking the English oral test scenario as an example, the speech that is not read according to the reference text may include Mandarin, dialect, Chinese English, typing on the keyboard, blowing into the microphone, singing or coughing, etc.

[0082] S304: The speech evaluation system trains K acoustic sub-models in the acoustic model according to the data set.

[0083] Each sample data in the data set has K labels, that is, the data set includes K groups of labels. Based on this, the speech evaluation system can train an acoustic sub-model according to a group of labels in the data set, and each group of labels is used to train a different acoustic sub-model.

[0084] In order to ensure that the final acoustic model can distinguish between the in-set data and the out-set data, two training objectives (such as two loss functions) are used to update the parameters of the acoustic sub-model during the training process. The following is an example of the training of one of the acoustic sub-models.

[0085] For the data in the set, its label is a one-hot vector, so the standard cross entropy function can be used to calculate the loss, where this loss is also called the standard cross entropy loss, denoted as In some embodiments, the Focal Loss function can also be used to calculate the loss, which is recorded as In this way, the imbalance in the number of samples of various types during model training can be alleviated, allowing the model to pay more attention to difficult-to-classify samples. and The calculation formula is as follows:

[0086]

[0087]

[0088] Among them, X is the data in the set used to train the acoustic sub-model, Y is the label of the data in the set, and f(x i ) refers to x i The model prediction results of the frame data, means x i The frame data is the model prediction result of the category label c, α c is the weight of category c, and γ is the attenuation coefficient.

[0089] For out-of-set data, whose labels are uniformly distributed across state categories, the loss can be calculated using the Kullback-Leibler Divergence. You can also use JS divergence (Jensen-Shannon Divergence) to calculate the loss The difference between the two is that JS divergence solves the asymmetric problem of KL divergence. and The calculation formula is as follows:

[0090]

[0091]

[0092] in, is the out-of-set data used to train the acoustic sub-model, μ is the label of the out-of-set data, and f(x i ) refers to x i The model prediction results of the frame data.

[0093] Based on the above loss, the loss of the acoustic sub-model can be determined as follows:

[0094]

[0095] Among them, λ is a hyperparameter with a value range of [0,1] and can usually be determined by fine-tuning the validation set in the dataset.

[0096] This method solves the problem of unstable scoring of abnormal answering speech through the K-fold acoustic model training framework, without the need to introduce a separate anomaly detection module. Specifically, firstly, the number of state categories of the output layer of the acoustic model is divided into K state category sets based on the idea of ​​K-fold cross-validation; then, according to each state category set, the sample data in the data set is divided into in-set data and out-of-set data, and the labels are represented by one-hot vectors and uniform distribution of categories respectively; then, K independent sub-acoustic models are trained separately, and when training each sub-acoustic model, the in-set data and out-of-set data are trained using two different objective functions, CE Loss and KL Loss; finally, when calculating the frame-level state posterior probability, the posterior probabilities of K-1 models are averaged, and further forced segmentation and scoring features are performed. This case introduces the K-fold acoustic model training framework when training the acoustic model, so that the model can distinguish out-of-set data, thereby improving the scoring accuracy of out-of-set abnormal speech data.

[0097] Based on the speech evaluation method provided in the embodiment of the present application, the embodiment of the present application also provides a speech evaluation system. The speech evaluation system is used to execute the above-mentioned speech evaluation method. Next, the speech evaluation system of the embodiment of the present application will be introduced in conjunction with the accompanying drawings.

[0098] See also Figure 4 The structural diagram of the speech evaluation system shown in FIG. Figure 4 As shown, the speech evaluation system 400 includes:

[0099] An acquisition module 402 is used to acquire the speech to be evaluated and the reference text;

[0100] The acoustic processing module 404 is used to extract acoustic features from the speech, input the acoustic features into K acoustic sub-models of the acoustic model, and obtain a posterior probability of the comprehensive state, wherein the K acoustic sub-models are trained by respective in-set data and out-of-set data, and each acoustic sub-model has different out-of-set data;

[0101] A post-processing module 406, configured to align the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, and obtain a scoring feature based on the alignment result;

[0102] The evaluation module 408 is used to input the scoring features into a scoring model to regress the score of the speech.

[0103] In some possible implementations, the system 400 further includes:

[0104] A training module is used to divide the state categories of the output layer of the acoustic model into K groups; the data whose state categories belong to the first group to the data whose state categories belong to the Kth group are taken as out-of-set data, and the data whose state categories belong to the remaining K-1 groups are taken as in-set data, the labels of the out-of-set data are the uniform distribution of the state categories of the corresponding groups, and the labels of the in-set data are the vectors of the corresponding state categories; the acoustic sub-model is trained by the in-set data and the out-of-set data of each acoustic sub-model in the K acoustic sub-models.

[0105] In some possible implementations, the objective function used to train the acoustic sub-model is a multi-objective function, and the multi-objective function is obtained based on a first objective function and a second objective function. The first objective function is the objective function when training the acoustic sub-model using the data within the set, and the second objective function is the objective function when training the acoustic sub-model using the data outside the set.

[0106] In some possible implementations, the acoustic processing module 404 is specifically used to:

[0107] Inputting the acoustic features into the K sub-acoustic models of the acoustic model to obtain K-1 posterior probabilities of each state type;

[0108] The comprehensive state posterior probability is determined according to the K-1 posterior probabilities of each state type.

[0109] In some possible implementations, the post-processing module 406 is specifically configured to:

[0110] Determining a target reading path by using a Viterbi algorithm according to the comprehensive state posterior probability and a segmentation network constructed by the reference text;

[0111] The phonemes of the speech are aligned with the reference text according to the target reading path to obtain time boundary information of the phonemes.

[0112] In some possible implementations, the scoring feature includes at least one of good pronunciation, word addition and omission ratio, and average phoneme pronunciation duration;

[0113] The post-processing module 406 is specifically used for:

[0114] The pronunciation goodness, the word addition / omission ratio and / or the average phoneme pronunciation duration are determined according to the time boundary information of the phoneme.

[0115] In some possible implementations, the data set also includes sample data generated by abnormal speech.

[0116] The speech evaluation system 400 according to the embodiment of the present application may correspond to the method described in the embodiment of the present application, and the above and other operations and / or functions of each module / unit of the speech evaluation system 400 are respectively to implement Figure 2 , Figure 3 For the sake of brevity, the corresponding processes of each method in the illustrated embodiment are not described in detail here.

[0117] Furthermore, the embodiment of the present application also provides a computing device cluster, which includes at least one computing device, any of which may be a server from a cloud environment or an edge environment, or a terminal device. The computing device cluster or computing device is specifically used to implement the following Figure 4 Functionality of the speech assessment system 400 in the illustrated embodiment.

[0118] Figure 5 A schematic diagram of the structure of a computing device cluster is provided, such as Figure 5 As shown, the computing device cluster 50 includes multiple computing devices 500, and the computing device 500 includes a bus 501, a processor 502, a communication interface 503, and a memory 504. The processor 502, the memory 504, and the communication interface 503 communicate with each other through the bus 501.

[0119] The bus 501 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0120] The processor 502 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0121] The communication interface 503 is used for communicating with the outside. For example, the communication interface 503 is used to obtain the speech to be evaluated and the reference file, return the speech score, and so on.

[0122] The memory 504 may include a volatile memory, such as a random access memory (RAM). The memory 504 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0123] The memory 504 stores computer-readable instructions, and the processor 502 executes the computer-readable instructions to enable the computing device 500 or the computing device cluster 50 to perform the aforementioned speech evaluation method (or implement the functions of the aforementioned speech evaluation system 400 ).

[0124] Specifically, in implementing Figure 4 In the case of the embodiment of the system shown, and Figure 4 In the case where the functions of the modules of the speech evaluation system 400 described in the above, such as the acquisition module 402, the acoustic processing module 404, the post-processing module 406, and the evaluation module 408, are implemented by software, the execution Figure 4The software or program code required for the functions of each module in the computing device cluster 50 may be stored in at least one memory 504 in the computing device cluster 50. At least one processor 502 executes the program code stored in the memory 504, so that the computing device 500 or the computing device cluster 50 performs the aforementioned speech evaluation method.

[0125] It should be noted that when the computing device cluster includes multiple computing devices, each computing device can independently execute the voice evaluation method, that is, each computing device can provide a complete voice evaluation service. In other embodiments, multiple computing devices can collaboratively execute the voice evaluation method, for example, some computing devices can execute several steps of the voice evaluation method, and other computing devices execute other steps of the voice evaluation method, and multiple computing devices provide a complete voice evaluation service through collaboration.

[0126] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on the computing device cluster 50, the computing device cluster 50 executes any implementation method of the above-mentioned speech evaluation method.

[0127] Furthermore, the embodiment of the present application also provides a computer program product. When the computer program product is run on the computing device cluster 50, the computing device cluster 50 executes any one of the implementation methods of the above-mentioned speech evaluation method.

[0128] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0129] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0130] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0131] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech evaluation method, characterized in that: include: Obtain the speech to be evaluated and the reference text; Extracting acoustic features from the speech, inputting the acoustic features into K acoustic sub-models of the acoustic model, and obtaining a comprehensive state posterior probability, wherein the K acoustic sub-models are trained by respective in-set data and out-of-set data, and each acoustic sub-model has different out-of-set data; According to the comprehensive state posterior probability and the segmentation network constructed by the reference text, aligning the phonemes of the speech with the reference text, and obtaining a scoring feature based on the alignment result; The scoring features are input into a scoring model to regress the score of the speech.

2. The method according to claim 1, characterized in that The K sub-acoustic models are trained in the following way: Dividing the state categories of the output layer of the acoustic model into K groups; The data whose status categories belong to the first group to the data whose status categories belong to the Kth group are respectively regarded as out-of-set data, and the data whose status categories belong to the remaining K-1 groups are regarded as in-set data, the labels of the out-of-set data are the uniform distribution of the status categories of the corresponding groups, and the labels of the in-set data are the vectors of the corresponding status categories; The sub-acoustic model is trained by using the in-set data and out-of-set data of each sub-acoustic model in the K sub-acoustic models.

3. The method according to claim 2, characterized in that The objective function used to train the sub-acoustic model is a multi-objective function, and the multi-objective function is obtained based on a first objective function and a second objective function. The first objective function is the objective function when training the sub-acoustic model using the data within the set, and the second objective function is the objective function when training the sub-acoustic model using the data outside the set.

4. The method according to any one of claims 1 to 3, characterized in that: The step of inputting the acoustic features into K sub-acoustic models of the acoustic model to obtain a posterior probability of a comprehensive state includes: Inputting the acoustic features into the K sub-acoustic models of the acoustic model to obtain K-1 posterior probabilities of each state type; The comprehensive state posterior probability is determined according to the K-1 posterior probabilities of each state type.

5. The method according to any one of claims 1 to 3, characterized in that: The step of aligning the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text comprises: Determining a target reading path by using a Viterbi algorithm according to the comprehensive state posterior probability and a segmentation network constructed by the reference text; The phonemes of the speech are aligned with the reference text according to the target reading path to obtain time boundary information of the phonemes.

6. The method according to claim 5, characterized in that The scoring features include at least one of good pronunciation, word addition and omission ratio, and average phoneme pronunciation duration; The step of obtaining a scoring feature based on the alignment result includes: The pronunciation goodness, the word addition / omission ratio and / or the average phoneme pronunciation duration are determined according to the time boundary information of the phoneme.

7. The method according to any one of claims 1 to 3, characterized in that: The in-set data and the out-set data are determined based on sample data in a data set, and the data set also includes sample data generated by abnormal speech.

8. A speech evaluation system, characterized in that: The system comprises: An acquisition module, used to acquire the speech to be evaluated and the reference text; An acoustic processing module, used for extracting acoustic features from the speech, inputting the acoustic features into K acoustic sub-models of the acoustic model, and obtaining a posterior probability of the comprehensive state, wherein the K acoustic sub-models are trained by respective in-set data and out-of-set data, and the out-of-set data of each acoustic sub-model is different; A post-processing module, configured to align the phonemes of the speech with the reference text according to the comprehensive state posterior probability and the segmentation network constructed by the reference text, and obtain a scoring feature based on the alignment result; The evaluation module is used to input the scoring feature into the scoring model to regress the score of the speech.

9. A computing device cluster, characterized in that: The computing device cluster includes at least one computing device, and the at least one computing device includes: a processor, a memory, and a system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the computing device cluster executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a computing device cluster, the computing device cluster executes the method according to any one of claims 1 to 7.

Images