A method, device, storage medium and equipment for pronounciation evaluation

Abstract

The present application discloses a pronunciation evaluation method, device, storage medium and equipment, the method comprising: first obtaining the target speech to be evaluated and extracting its acoustic features; then performing segmentation and alignment processing on the acoustic features and the standard text corresponding to the target speech to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information, and then performing feature fusion and encoding processing on the effective audio features of adjacent phonemes to obtain a coding information matrix; and then performing fusion decoding on the coding information matrix, as well as the phoneme sequence and / or phoneme boundary information according to the characteristics of the preset N evaluation dimensions, and performing pronunciation evaluation on the target speech according to the decoding results to obtain an evaluation result. It can be seen that the present application first encodes the global features of the target speech, and then uses the obtained global coding features to simultaneously perform pronunciation evaluation of N different dimensions, thereby being able to improve the accuracy of the evaluation results and the evaluation effect.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a pronunciation evaluation method, apparatus, storage medium and equipment. Background Art

[0002] With the continuous breakthroughs in artificial intelligence (AI) and the growing development of speech recognition technology, speech recognition is increasingly used in our daily work and lives. For example, in educational scenarios such as online language learning, the ability to effectively evaluate users' pronunciation is crucial.

[0003] At present, traditional pronunciation evaluation methods usually adopt regression evaluation schemes based on feature engineering, and have made certain progress in multi-layer model fusion. Specifically, different feature combinations are first taken, and the evaluation scores are regressed to obtain as the score features of the specific model, and then the fusion evaluation is carried out. However, this evaluation method is not only difficult to fully explore the global characteristics of the audio in the single-model regression stage, but also difficult to achieve a balance between the weights and features of each model through supervised training in the multi-model fusion stage, resulting in inaccurate evaluation results. In addition, when switching between evaluation tasks of different dimensions such as accuracy and fluency, the reusability of existing evaluation models is also poor, which leads to poor results in the final pronunciation evaluation. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to provide a pronunciation evaluation method, apparatus, storage medium and device, which can effectively improve the accuracy of the evaluation results when performing pronunciation evaluation, thereby improving the evaluation effect.

[0005] The present invention provides a method for evaluating pronunciation, including:

[0006] Acquiring a target speech to be evaluated and extracting acoustic features of the target speech;

[0007] Performing segmentation and alignment processing on the acoustic features of the target speech and the standard text corresponding to the target speech to obtain valid audio features, phoneme sequences, and / or phoneme boundary information corresponding to the target speech;

[0008] Performing feature fusion and encoding processing of adjacent phonemes on the effective audio features to obtain a coding information matrix corresponding to the target speech;

[0009] According to the characteristics of the preset N evaluation dimensions, the coding information matrix, and the phoneme sequence and / or phoneme boundary information are fused and decoded, and the pronunciation evaluation of the target speech is performed based on the decoding results to obtain the evaluation result of the target speech; N is a positive integer greater than 0.

[0010] In one possible implementation, segmenting and aligning the acoustic features of the target speech and the standard text corresponding to the target speech to obtain valid audio features, phoneme sequences, and / or phoneme boundary information corresponding to the target speech includes:

[0011] Determining a standard phoneme sequence corresponding to the standard text, and determining a motivational word based on the standard phoneme sequence;

[0012] The standard phoneme sequence and the acoustic features of the target speech are input into a preset acoustic model, the audio frame sequence and phoneme sequence of the target speech are segmented and aligned, and the processing results are optimized and adjusted using the excitation words to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information.

[0013] In one possible implementation, the preset acoustic model is a connection-based temporal classification (CTC) acoustic model.

[0014] In one possible implementation, performing feature fusion and encoding processing on adjacent phonemes of the effective audio features to obtain a coding information matrix corresponding to the target speech includes:

[0015] Inputting the effective audio features into a pre-trained acoustic model to map them to a preset fixed-dimensional vector space to obtain a vector of the preset fixed dimension as an initial acoustic representation;

[0016] Using a pre-built encoder, the initial acoustic representation is subjected to feature fusion and encoding processing of adjacent phonemes to obtain a coding information matrix corresponding to the target speech; the encoder is trained using supervised data; the encoder includes a multi-head attention mechanism layer, a residual connection layer, a regularization layer, and a forward forwarding layer.

[0017] In one possible implementation, the encoding information matrix and the phoneme sequence and / or phoneme boundary information are fused and decoded according to the characteristics of the preset N evaluation dimensions, and the target speech is evaluated based on the decoding results to obtain the evaluation result of the target speech, including:

[0018] According to the characteristics of the preset N evaluation dimensions, the encoding information matrix and the phoneme sequence and / or phoneme boundary information are input into a pre-built decoder corresponding to the evaluation dimension for fusion decoding to obtain complete acoustic features; the decoder is trained using supervised data; the encoder includes a multi-head attention mechanism layer, a residual connection layer, a regularization layer, and a forward forwarding layer;

[0019] Inputting the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic features into the scoring space of the corresponding evaluation dimension, thereby obtaining a score of the target speech corresponding to the evaluation dimension;

[0020] The scores of the target speech in the N evaluation dimensions are fused to obtain a processed score as the evaluation result of the target speech.

[0021] In one possible implementation, the N evaluation dimensions include accuracy and fluency; and according to the characteristics of the preset N evaluation dimensions, the encoding information matrix and the phoneme sequence and / or phoneme boundary information are input into a pre-built decoder corresponding to the evaluation dimension for fusion decoding to obtain a complete acoustic feature, including:

[0022] Inputting the encoding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and inputting the encoding information matrix and phoneme boundary information into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature;

[0023] Inputting the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic features into the scoring space corresponding to the evaluation dimension to obtain the score of the target speech corresponding to the evaluation dimension includes:

[0024] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; and inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space, thereby obtaining a score for the fluency of the target speech.

[0025] In one possible implementation, the N evaluation dimensions include fluency and completeness; and according to the characteristics of the preset N evaluation dimensions, the encoding information matrix and the phoneme sequence and / or phoneme boundary information are input into a pre-built decoder corresponding to the evaluation dimension for fusion decoding to obtain complete acoustic features, including:

[0026] Inputting the encoding information matrix and the phoneme boundary information into a pre-built fluency decoder for decoding to obtain a second complete acoustic feature, and inputting the encoding information matrix and the phoneme sequence into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature;

[0027] Inputting the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic features into the scoring space corresponding to the evaluation dimension to obtain the score of the target speech corresponding to the evaluation dimension includes:

[0028] Inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into a fluency scoring space to obtain a fluency score of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor for completeness to map the third complete acoustic feature into a completeness scoring space to obtain a completeness score of the target speech.

[0029] In one possible implementation, the N evaluation dimensions include accuracy and completeness; and according to the characteristics of the preset N evaluation dimensions, the encoding information matrix and the phoneme sequence and / or phoneme boundary information are input into a pre-built decoder corresponding to the evaluation dimension for fusion decoding to obtain complete acoustic features, including:

[0030] Inputting the coding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and inputting the coding information matrix and the phoneme sequence into a pre-built integrity decoder for fusion decoding to obtain a third complete acoustic feature;

[0031] Inputting the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic features into the scoring space corresponding to the evaluation dimension to obtain the score of the target speech corresponding to the evaluation dimension includes:

[0032] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor of accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor of completeness to map the third complete acoustic feature into the completeness scoring space, thereby obtaining a score for the completeness of the target speech.

[0033] In one possible implementation, the N evaluation dimensions include accuracy, fluency, and completeness; and according to the characteristics of the preset N evaluation dimensions, the encoding information matrix and the phoneme sequence and / or phoneme boundary information are input into a pre-built decoder corresponding to the evaluation dimension for fusion decoding to obtain complete acoustic features, including:

[0034] Inputting the coding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature; inputting the coding information matrix and the phoneme boundary information into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature; and inputting the coding information matrix and the phoneme sequence into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature;

[0035] Inputting the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic features into the scoring space corresponding to the evaluation dimension to obtain the score of the target speech corresponding to the evaluation dimension includes:

[0036] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space, thereby obtaining a score for the fluency of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor for completeness to map the third complete acoustic feature into the completeness scoring space, thereby obtaining a score for the completeness of the target speech.

[0037] The present application also provides a pronunciation evaluation device, comprising:

[0038] an acquisition unit, configured to acquire a target speech to be evaluated and extract acoustic features of the target speech;

[0039] an alignment unit, configured to segment and align the acoustic features of the target speech and the standard text corresponding to the target speech, to obtain valid audio features, phoneme sequences, and / or phoneme boundary information corresponding to the target speech;

[0040] An encoding unit, configured to perform feature fusion and encoding processing of adjacent phonemes on the effective audio features to obtain an encoding information matrix corresponding to the target speech;

[0041] A decoding unit is used to fuse and decode the coding information matrix, the phoneme sequence and / or the phoneme boundary information according to the characteristics of the preset N evaluation dimensions, and to perform pronunciation evaluation on the target speech according to the decoding results to obtain the evaluation result of the target speech; N is a positive integer greater than 0.

[0042] In a possible implementation, the alignment unit includes:

[0043] a determination subunit, configured to determine a standard phoneme sequence corresponding to the standard text, and determine an incentive word based on the standard phoneme sequence;

[0044] The alignment subunit is used to input the standard phoneme sequence and the acoustic features of the target speech into a preset acoustic model, perform segmentation and alignment processing on the audio frame sequence and phoneme sequence of the target speech, and optimize and adjust the processing results using the excitation words to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information.

[0045] In one possible implementation, the preset acoustic model is a connection-based temporal classification (CTC) acoustic model.

[0046] In a possible implementation, the encoding unit includes:

[0047] A first input subunit is configured to input the valid audio features into a pre-trained acoustic model to map the audio features into a preset fixed-dimensional vector space, thereby obtaining a vector of the preset fixed dimension as an initial acoustic representation;

[0048] The encoding subunit is used to use a pre-built encoder to perform feature fusion and encoding processing on the initial acoustic representation of adjacent phonemes to obtain a coding information matrix corresponding to the target speech; the encoder is trained using supervised data; the encoder includes a multi-head attention mechanism layer, a residual connection layer, a regularization layer, and a forward forwarding layer.

[0049] In one possible implementation, the decoding unit includes:

[0050] The decoding subunit is used to input the encoding information matrix and the phoneme sequence and / or phoneme boundary information into a pre-built decoder corresponding to the evaluation dimension for fusion decoding according to the characteristics of the preset N evaluation dimensions to obtain complete acoustic features; the decoder is trained using supervised data; the encoder includes a multi-head attention mechanism layer, a residual connection layer, a regularization layer, and a forward forwarding layer;

[0051] A second input subunit is configured to input the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension, so as to map the complete acoustic features into a scoring space corresponding to the evaluation dimension, and obtain a score of the target speech corresponding to the evaluation dimension;

[0052] The fusion subunit is used to perform fusion processing on the scores of the target speech in the N evaluation dimensions to obtain a processed score as the evaluation result of the target speech.

[0053] In one possible implementation, the N evaluation dimensions include accuracy and fluency; and the decoding subunit is specifically configured to:

[0054] Inputting the encoding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and inputting the encoding information matrix and phoneme boundary information into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature;

[0055] The second input subunit is specifically configured to:

[0056] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; and inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space, thereby obtaining a score for the fluency of the target speech.

[0057] In one possible implementation, the N evaluation dimensions include fluency and completeness; and the decoding subunit is specifically configured to:

[0058] Inputting the encoding information matrix and the phoneme boundary information into a pre-built fluency decoder for decoding to obtain a second complete acoustic feature, and inputting the encoding information matrix and the phoneme sequence into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature;

[0059] The second input subunit is specifically configured to:

[0060] Inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into a fluency scoring space to obtain a fluency score of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor for completeness to map the third complete acoustic feature into a completeness scoring space to obtain a completeness score of the target speech.

[0061] In one possible implementation, the N evaluation dimensions include accuracy and completeness; and the decoding subunit is specifically configured to:

[0062] Inputting the coding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and inputting the coding information matrix and the phoneme sequence into a pre-built integrity decoder for fusion decoding to obtain a third complete acoustic feature;

[0063] The second input subunit is specifically configured to:

[0064] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor of accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor of completeness to map the third complete acoustic feature into the completeness scoring space, thereby obtaining a score for the completeness of the target speech.

[0065] In one possible implementation, the N evaluation dimensions include accuracy, fluency, and completeness; and the decoding subunit is specifically configured to:

[0066] Inputting the coding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature; inputting the coding information matrix and the phoneme boundary information into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature; and inputting the coding information matrix and the phoneme sequence into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature;

[0067] The second input subunit is specifically configured to:

[0068] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space, thereby obtaining a score for the fluency of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor for completeness to map the third complete acoustic feature into the completeness scoring space, thereby obtaining a score for the completeness of the target speech.

[0069] The embodiment of the present application also provides a pronunciation evaluation device, comprising: a processor, a memory, and a system bus;

[0070] The processor and the memory are connected via the system bus;

[0071] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned pronunciation evaluation method.

[0072] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores instructions. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned pronunciation evaluation method.

[0073] The embodiment of the present application further provides a computer program product. When the computer program product is run on a terminal device, the terminal device is enabled to execute any one of the implementations of the above-mentioned pronunciation evaluation method.

[0074] The embodiments of the present application provide a pronunciation evaluation method, apparatus, storage medium, and device, which first obtain a target speech to be evaluated and extract the acoustic features of the target speech; then segment and align the acoustic features of the target speech and the standard text corresponding to the target speech to obtain valid audio features, as well as phoneme sequences and / or phoneme boundary information corresponding to the target speech; then, perform feature fusion and encoding of adjacent phonemes on the valid audio features to obtain a coding information matrix corresponding to the target speech; and then, based on the characteristics of N preset evaluation dimensions, perform fusion decoding on the coding information matrix, as well as the phoneme sequence and / or phoneme boundary information, and perform pronunciation evaluation on the target speech based on the decoding results to obtain an evaluation result of the target speech.

[0075] It can be seen that the embodiment of the present application first encodes the global features of the target speech, and then uses the obtained global encoding features to simultaneously perform pronunciation evaluation in N different dimensions such as accuracy and fluency. This not only improves the accuracy of the evaluation results, but also enables the simultaneous performance of evaluation tasks in different dimensions, thereby solving the problem of poor reusability of the evaluation model in the prior art for different dimensions, and thereby improving the pronunciation evaluation effect of the target speech in different dimensions. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0077] Figure 1 A flowchart of a pronunciation evaluation method provided in an embodiment of the present application;

[0078] Figure 2 An example diagram of the overall process of pronunciation evaluation provided in an embodiment of the present application;

[0079] Figure 3 A schematic diagram of the structure of the encoder provided in an embodiment of the present application;

[0080] Figure 4 A schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0081] Figure 5 A schematic diagram of the composition of a pronunciation evaluation device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0082] Pronunciation assessment technology generally refers to a technique in which a computer scores the user's (e.g., student's) pronunciation quality (such as accuracy and fluency) after the user pronounces a reference text (e.g., a student reading a reference text). Existing pronunciation assessment methods primarily rely on regression models. Inappropriate selection of acoustic features can significantly reduce assessment effectiveness. Furthermore, regression schemes designed for specific assessment dimensions cannot be directly applied to other dimensions, resulting in poor reusability and poor assessment results.

[0083] Currently, the more commonly used pronunciation evaluation method is: first, pre-process the user's reading pronunciation and extract acoustic features; then align it with the preset reading text, and based on the segmented phoneme segment recognition results, extract high-quality evaluation features related to each dimension from multiple aspects such as pronunciation accuracy and fluency. Finally, use the support vector regression algorithm to achieve effective fusion of evaluation features of different dimensions, thereby giving a comprehensive evaluation that can fully reflect the overall quality of the user's reading pronunciation.

[0084] Taking accuracy as an example, existing evaluation methods primarily rely on GOP (goodness of pronunciation) features extracted from audio. After forcibly segmenting the speech according to the text being read, the accuracy of phoneme pronunciation is calculated based on the segmentation boundaries, and the correctness of phoneme pronunciation is determined using a threshold method.

[0085] However, in the aforementioned feature regression evaluation method, GOP features account for a significant portion of the score calculation for each dimension. Because the GOP calculation method is sensitive to segmentation boundaries, slight changes in segmentation boundaries can lead to significant changes in this feature, resulting in poor stability in the evaluation results for each dimension. Furthermore, in these feature regression evaluation methods, the calculation of evaluation scores for each dimension is typically based on the aggregation of local syllable feature information, and it is unable to use global information to distinguish subtle differences between similar pronunciations. For example, the first half of "in" and "ing" have similar pronunciation principles, which will reduce the feature differentiation between the two at the phoneme level.

[0086] Therefore, when the current regression evaluation method based on feature engineering is used for pronunciation evaluation, the accuracy of the evaluation results obtained is not high, and the pronunciation evaluation effect is poor.

[0087] To address the above-mentioned defects, the present application provides a pronunciation evaluation method, which first obtains the target speech to be evaluated and extracts the acoustic features of the target speech; then, the acoustic features of the target speech and the standard text corresponding to the target speech are segmented and aligned to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information; then, the effective audio features are subjected to feature fusion and encoding processing of adjacent phonemes to obtain the coding information matrix corresponding to the target speech; and then, the coding information matrix, as well as the phoneme sequence and / or phoneme boundary information can be fused and decoded according to the characteristics of the preset N evaluation dimensions, and the pronunciation evaluation of the target speech is performed based on the decoding results to obtain the evaluation results of the target speech.

[0088] It can be seen that the embodiment of the present application first encodes the global features of the target speech, and then uses the obtained global encoding features to simultaneously perform pronunciation evaluation in N different dimensions such as accuracy and fluency. This not only improves the accuracy of the evaluation results, but also enables the simultaneous performance of evaluation tasks in different dimensions, thereby solving the problem of poor reusability of the evaluation model in the prior art for different dimensions, and thereby improving the pronunciation evaluation effect of the target speech in different dimensions.

[0089] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0090] First embodiment

[0091] See also Figure 1 , is a flow chart of a pronunciation evaluation method provided in this embodiment, the method comprising the following steps:

[0092] S101: Obtain a target speech to be evaluated and extract acoustic features of the target speech.

[0093] In this embodiment, any speech for which pronunciation evaluation is required is defined as a target speech. It should be noted that this embodiment does not limit the language type of the target speech; for example, the target speech can be in Chinese or English. Furthermore, this embodiment does not limit the length of the target speech; for example, the target speech can be a sentence or a paragraph.

[0094] It is understood that the target speech can be obtained through recording or other means as needed. For example, the speech produced by a student reading a reference text, daily telephone conversations, or recordings of instant messaging software can all serve as the target speech. Furthermore, to improve the effectiveness of the target speech evaluation, it is necessary to utilize existing or future feature extraction methods to extract the acoustic features of the target speech for use in executing the subsequent step S102.

[0095] Specifically, when extracting the acoustic features of the target speech, the target speech needs to be framed first to obtain the corresponding speech frame sequence, and then the framed speech frame sequence is pre-emphasized; and then the acoustic features of each speech frame are extracted in turn, where the acoustic feature refers to the feature data used to characterize the acoustic information of the corresponding speech frame, for example, it can be Mel-scale Frequency Cepstral Coefficients (MFCC) features or Perceptual Linear Predictive (PLP) features.

[0096] It should be noted that the embodiments of the present application do not limit the method for extracting the acoustic features of the target speech, nor do they limit the specific extraction process. An appropriate extraction method can be selected according to actual conditions, and corresponding feature extraction operations can be performed.

[0097] S102: Perform segmentation and alignment processing on the acoustic features of the target speech and the standard text corresponding to the target speech to obtain valid audio features corresponding to the target speech, as well as phoneme sequences and / or phoneme boundary information.

[0098] In this embodiment, after obtaining the target speech to be evaluated and extracting the acoustic features of the target speech through step S101, in order to effectively improve the accuracy of the evaluation results of the target speech, the existing or future forced segmentation (Force Alignment) algorithm can be further used to perform time segmentation and alignment processing on the acoustic features of the target speech and the phoneme sequence of the standard text corresponding to the target speech, and obtain the phoneme sequence and phoneme boundary information (including but not limited to time boundary information, pronunciation interval, etc.) corresponding to the recognition result of the target language, and then obtain the speech segment containing context information corresponding to each phoneme based on the time boundary information, determine the effective audio with silence and noise removed from the target speech, and after extracting the acoustic features (such as MFCC) of the effective audio as the effective audio features, use it to execute the subsequent step S103 to realize global feature encoding, such as Figure 2 shown.

[0099] Specifically, in an optional implementation, it is first necessary to determine the standard phoneme sequence corresponding to the standard text, and based on the standard phoneme sequence, determine the incentive words. Then, an identification and matching algorithm with incentive words is used to perform segmentation and alignment processing on the acoustic features of the target speech and the standard text corresponding to the target speech. This allows for full consideration of situations such as skipping, omitting, and confusing mispronunciations that may occur when a user (such as a student) reads the standard text aloud. By using several syllables before and after the current pronounced syllable as incentive words, pronunciation defect syllables in the recognized audio can be optimized. The recognition result optimized with incentive words is closer to the pronunciation intention of the user (such as a student), thus making the judgment of additional readings, omitted readings, and re-readings more accurate. It can be seen that this matching algorithm gives judgments of additional readings and omitted readings at the syllable (or phoneme) level, judgments of re-readings at the sentence level, and information on the user's pronunciation clause segmentation based on the difference between the recognized result of the target language audio and the standard text.

[0100] In this implementation, when performing segmentation and alignment processing on the acoustic features of the target speech and the standard text corresponding to the target speech, first, the standard phoneme sequence corresponding to the standard text (such as Figure 2 "a, ai, g, uo") and the acoustic features of the target speech (such as MFCC) can be input into a preset acoustic model to perform segmentation and alignment processing on the audio frame sequence and phoneme sequence of the target speech, and the obtained processing result can be optimized and adjusted using incentive words. For example, an identification and matching algorithm with incentive words can be used to optimize and adjust the pronunciation "ai gui" to "ai guo", etc., to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information.

[0101] Among them, the specific structure of the preset acoustic model is not limited in this application and can be set according to actual situations and empirical values. A preferred implementation is to set the preset acoustic model as a Connectionist Temporal Classification (CTC for short) acoustic model. Using this model, the time boundary information corresponding to all syllables can be determined based on the recognition result. The CTC acoustic model can directly learn sequence data, adds a many-to-one spatial mapping between the input frame-level acoustic feature sequence and the output syllable sequence, and defines the loss function of CTC on this basis, so as to be able to achieve automatic alignment during the model training process and minimize the loss function to obtain an acoustic model with better recognition effect. The specific training process is the same as that of the existing model training process and will not be elaborated in detail here.

[0102] In this way, while inputting the acoustic features of the target speech (such as Fbank or MFCC) into the preset acoustic model (such as CTC) in units of frames, the standard text can be mapped into a phoneme sequence, and the stimulus words used to assist the recognition process can be used to optimize the recognition results of the model's pronunciation. After completing the phoneme sequence recognition prediction, the preset acoustic model such as CTC can be used to directly align the audio frame sequence and the recognized phoneme sequence to output the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information, such as Figure 2 shown.

[0103] S103: Perform feature fusion and encoding processing of adjacent phonemes on the effective audio features to obtain a coding information matrix corresponding to the target speech.

[0104] In this embodiment, after obtaining the effective audio features (such as Fbank or MFCC) corresponding to the target speech through step S102, in order to improve the pronunciation evaluation effect, the existing or future coding methods can be further used to perform feature fusion and coding processing of adjacent phonemes on the effective audio features (such as Fbank or MFCC) corresponding to the target speech, so that the acoustic features of the current speech phonemes of each frame contain the acoustic information of the speech phonemes of the previous and next frames, thereby realizing the fusion and coding of global acoustic features. For example, a deep neural network coding (ncoder) module can be used to perform acoustic information interaction on the effective audio features corresponding to the target speech, realize feature fusion and coding processing at the adjacent phoneme level, and obtain the coding information matrix corresponding to the target speech for executing the subsequent step S104.

[0105] Specifically, an optional implementation method is that after obtaining the effective audio features (such as Fbank or MFCC) corresponding to the target speech, the effective audio features can be first input into the pre-trained acoustic model to map them to a preset fixed-dimensional vector space (the specific value is not limited, such as 40, 60 or 120, etc.) to obtain a vector of the preset fixed dimension as the initial acoustic representation, such as Figure 2 Then, the pre-built encoder can be used to perform feature fusion and encoding of adjacent phonemes on the obtained initial acoustic representation to obtain the encoding information matrix corresponding to the target speech.

[0106] Among them, this application does not limit the specific structure of the preset acoustic model, which can be set according to actual conditions and experience. A preferred implementation method is to set the preset acoustic model to a word2vec model, etc., and use this model to map the effective audio features corresponding to the target speech (such as Fbank or MFCC) to a fixed-dimensional vector space of 40, 60 or 120, etc., as the initial acoustic representation. In addition, the initial acoustic representation usually refers to the fusion and interaction between the feature information in units of syllables or phonemes. The use of the initial acoustic representation can more accurately represent the global acoustic information of the target language.

[0107] In addition, the encoder pre-built in this application is trained using supervised data. Figure 3 As shown, the encoder may include but is not limited to a multi-head attention mechanism layer, a residual connection layer, a regularization layer, and a forward forwarding layer. In this way, a neural network encoding architecture based on a multi-head attention mechanism (such as a Transformer encoder) can fully integrate the input initial acoustic representation through linear transformation and output a relatively complete encoding information matrix for executing the subsequent step S104.

[0108] It should be noted that the specific construction process of the encoder is consistent with the existing model construction process, which will not be repeated here. Figure 3 The encoder structure shown is used to illustrate the encoding process as follows:

[0109] Figure 3 The multi-head attention layer of the encoder is based on the self-attention mechanism. The input of the self-attention mechanism comes from the initial acoustic representation, which is transformed into a Q, K, V triple through different linear transformations. The output vector is calculated as follows:

[0110]

[0111] in, dk Represents the dimensionality of the initial acoustic representation.

[0112] also, Figure 3 The residual connection layer (Add) of the encoder is used to solve the problem of multi-layer network training. The regularization layer (Normal) can accelerate convergence, and the forward forwarding layer (Position wise FFN) uses two layers of full connection. The activation function of the first layer is Relu, and the second layer does not use an activation function. The dimension of the final output encoding information matrix is ​​consistent with the initial acoustic representation of the input.

[0113] S104: Based on the characteristics of the preset N evaluation dimensions, the coding information matrix, as well as the phoneme sequence and / or phoneme boundary information are fused and decoded, and the pronunciation of the target speech is evaluated based on the decoding results to obtain the evaluation result of the target speech; wherein N is a positive integer greater than 0.

[0114] In this embodiment, after obtaining the phoneme sequence and / or phoneme boundary information corresponding to the target speech through step S102, and obtaining the coding information matrix corresponding to the target speech through step S103, in order to effectively improve the accuracy of the pronunciation evaluation results of the target speech, it is further possible to perform fusion decoding on the coding information matrix, and the phoneme sequence and / or phoneme boundary information in the corresponding evaluation dimensions according to the characteristics of the preset N (N is a positive integer greater than 0) evaluation dimensions (such as accuracy, fluency, etc. evaluation dimensions), and perform pronunciation evaluation on the target speech based on the decoding results to obtain the evaluation results of the target speech.

[0115] Among them, the specific contents of the preset N evaluation dimensions can be pre-set according to actual conditions, and this application does not limit this. For example, accuracy, fluency, and completeness can be used as the three pre-set evaluation dimensions, such as Figure 2 shown.

[0116] In addition, it should be noted that the characteristics of the preset N evaluation dimensions also vary according to the different evaluation dimensions, and are all related to the characteristics of each evaluation dimension. For example, for the evaluation dimension of fluency, its evaluation characteristics may be related to the phoneme boundary information such as the pronunciation duration and pronunciation interval of the phonemes in the speech. Then, according to this evaluation characteristic, when evaluating fluency, the coding information matrix and the phoneme boundary information can be fused and decoded, and then the pronunciation of the target speech can be evaluated for fluency based on the decoding results to obtain the corresponding evaluation results.

[0117] Specifically, an optional implementation method is to obtain the phoneme sequence and / or phoneme boundary information corresponding to the target speech, and the encoding information matrix corresponding to the target speech, first, according to the characteristics of the preset N evaluation dimensions, input the encoding information matrix, and the phoneme sequence and / or phoneme boundary information into the pre-constructed corresponding evaluation dimensions (such as Figure 2 The accuracy, fluency, and completeness shown in the figure are decoded by fusion to obtain complete acoustic features, such as Figure 2As shown. This complete acoustic feature can then be input into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic feature into the scoring space of the corresponding evaluation dimension, thereby obtaining the score of the target speech corresponding to the evaluation dimension. The scores of the target speech across these N evaluation dimensions can then be fused, such as by weighted summation, to obtain a processed score, which serves as the evaluation result of the target speech. A higher score indicates better pronunciation of the target speech, while a lower score indicates poorer pronunciation.

[0118] Among them, the decoders of each evaluation dimension pre-built in this application are obtained by training with supervised data. Figure 4 As shown, taking the fluency decoder as an example, the decoder may include but is not limited to a multi-head attention mechanism layer, a residual connection layer, a regularization layer, and a forward forwarding layer. The supplementary inputs of the decoders of each dimension vary depending on the characteristics of the evaluation dimension. In addition, the fully connected layer regressor of each dimension can be an M×1 linear layer, where M represents the size of the acoustic representation, and its function is to map the acoustic representation obtained by the decoders of different dimensions to the corresponding dimensional scoring space, that is, to determine the evaluation scores of each dimension. It can be understood that the structure of the decoder of each evaluation dimension is similar to that of the encoder, but the two add a Multi-Head Attention layer in the process of information interaction to fuse the intermediate results output by the encoder (i.e., the encoding information matrix) and the dimension-related supplementary information (such as phoneme boundary information related to fluency, such as Figure 4 shown).

[0119] It should be noted that the specific construction process of the decoder of each evaluation dimension is consistent with the existing model construction process, which will not be repeated here. Figure 4 The encoder structure shown is used to illustrate the decoding process as follows:

[0120] Figure 4 In addition to the encoding information matrix output by the encoder, the input of the fluency encoder also includes the phoneme boundary information obtained in step S102 as a supplementary input. Therefore, the decoding process consists of two parts: (1) obtaining information such as syllable pronunciation duration and pronunciation interval from the phoneme boundary information and performing sentence-level information fusion; (2) introducing the phoneme boundary information as a decoding reference and parsing the encoder output (i.e., the audio encoding information matrix) to output complete acoustic features for more accurate representation of the target speech.

[0121] in, Figure 4The main function of the newly added Multi-head attention layer in the upper middle section is to complete the information fusion of the encoding information matrix. The main difference between this part and the Multi-head attention layer in the encoder is that the K and V matrices of the Self-Attention are not calculated using the output of the previous Decoder block, but are calculated using the encoding information matrix output by the encoder. K and V are calculated using the output of the encoder, and Q is calculated based on the output of the previous Decoder block. The subsequent calculation method is consistent with the above formula (1). This decoding method ensures that the determination of each phoneme in the decoding process can utilize all the phoneme information output by the encoder.

[0122] In this way, after the fluency decoder outputs a complete acoustic representation, it can be mapped into the fluency score space through a linear layer, and the fluency prediction score can be output through a softmax layer. Similarly, the scores of the target speech in other evaluation dimensions can be determined.

[0123] Specifically, an optional implementation method is that when the N evaluation dimensions include accuracy and fluency, the encoding information matrix of the target language can be first input into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and the encoding information matrix of the target language and the phoneme boundary information can be input into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature.

[0124] Then, the first complete acoustic feature can be input into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space to obtain a score for the target speech accuracy; and the second complete acoustic feature can be input into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space to obtain a score for the target speech fluency.

[0125] Another optional implementation method is that when the N evaluation dimensions include fluency and completeness, the encoding information matrix and phoneme boundary information of the target language can first be input into a pre-built fluency decoder for decoding to obtain a second complete acoustic feature, and the encoding information matrix and phoneme sequence of the target language can be input into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature.

[0126] Then, the second complete acoustic feature can be input into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space to obtain a target speech fluency score; and the third complete acoustic feature can be input into a pre-built fully connected layer regressor for completeness to map the third complete acoustic feature into the completeness scoring space to obtain a target speech completeness score.

[0127] Another optional implementation method is that when the N evaluation dimensions include accuracy and completeness, the encoding information matrix of the target language can be first input into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and the encoding information matrix of the target language and the phoneme sequence can be input into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature.

[0128] Then, the first complete acoustic feature can be input into a pre-built fully connected layer regressor of accuracy to map the first complete acoustic feature into the accuracy scoring space to obtain a score of the target speech accuracy; and the third complete acoustic feature can be input into a pre-built fully connected layer regressor of completeness to map the third complete acoustic feature into the completeness scoring space to obtain a score of the target speech completeness.

[0129] Another optional implementation method is that when the N evaluation dimensions include accuracy, fluency and completeness, the coding information matrix of the target language can be first input into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature; and the coding information matrix of the target language and the phoneme boundary information can be input into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature; and the coding information matrix of the target language and the phoneme sequence can be input into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature.

[0130] Then, the first complete acoustic feature can be input into a pre-built fully connected layer regressor of accuracy to map the first complete acoustic feature into the accuracy scoring space to obtain the target speech accuracy score; and the second complete acoustic feature can be input into a pre-built fully connected layer regressor of fluency to map the second complete acoustic feature into the fluency scoring space to obtain the target speech fluency score; and the third complete acoustic feature can be input into a pre-built completeness fully connected layer regressor to map the third complete acoustic feature into the completeness scoring space to obtain the target speech completeness score, as shown in FIG. Figure 2 shown.

[0131] In summary, the present embodiment provides a pronunciation evaluation method, which first obtains the target speech to be evaluated and extracts the acoustic features of the target speech; then, the acoustic features of the target speech and the standard text corresponding to the target speech are segmented and aligned to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information; then, the effective audio features are subjected to feature fusion and encoding processing of adjacent phonemes to obtain the coding information matrix corresponding to the target speech; and then, the coding information matrix, as well as the phoneme sequence and / or phoneme boundary information can be fused and decoded according to the characteristics of the preset N evaluation dimensions, and the pronunciation evaluation of the target speech is performed based on the decoding results to obtain the evaluation results of the target speech.

[0132] It can be seen that the embodiment of the present application first encodes the global features of the target speech, and then uses the obtained global encoding features to simultaneously perform pronunciation evaluation in N different dimensions such as accuracy and fluency. This not only improves the accuracy of the evaluation results, but also enables the simultaneous performance of evaluation tasks in different dimensions, thereby solving the problem of poor reusability of the evaluation model in the prior art for different dimensions, and thereby improving the pronunciation evaluation effect of the target speech in different dimensions.

[0133] Second embodiment

[0134] This embodiment will introduce a pronunciation evaluation device. For related content, please refer to the above method embodiment.

[0135] See also Figure 5 , is a schematic diagram of the composition of a pronunciation evaluation device provided in this embodiment, the device 500 includes:

[0136] An acquisition unit 501 is configured to acquire a target speech to be evaluated and extract acoustic features of the target speech;

[0137] An alignment unit 502 is configured to segment and align the acoustic features of the target speech and the standard text corresponding to the target speech to obtain valid audio features, phoneme sequences, and / or phoneme boundary information corresponding to the target speech;

[0138] The encoding unit 503 is configured to perform feature fusion and encoding processing of adjacent phonemes on the effective audio features to obtain an encoding information matrix corresponding to the target speech;

[0139] The decoding unit 504 is used to perform fusion decoding on the coding information matrix, the phoneme sequence and / or the phoneme boundary information according to the characteristics of the preset N evaluation dimensions, and perform pronunciation evaluation on the target speech according to the decoding results to obtain the evaluation result of the target speech; N is a positive integer greater than 0.

[0140] In one implementation of this embodiment, the alignment unit 502 includes:

[0141] a determination subunit, configured to determine a standard phoneme sequence corresponding to the standard text, and determine an incentive word based on the standard phoneme sequence;

[0142] The alignment subunit is used to input the standard phoneme sequence and the acoustic features of the target speech into a preset acoustic model, perform segmentation and alignment processing on the audio frame sequence and phoneme sequence of the target speech, and optimize and adjust the processing results using the excitation words to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information.

[0143] In one implementation of this embodiment, the preset acoustic model is a connection-based temporal classification (CTC) acoustic model.

[0144] In one implementation of this embodiment, the encoding unit 503 includes:

[0145] A first input subunit is configured to input the valid audio features into a pre-trained acoustic model to map the audio features into a preset fixed-dimensional vector space, thereby obtaining a vector of the preset fixed dimension as an initial acoustic representation;

[0146] The encoding subunit is used to use a pre-built encoder to perform feature fusion and encoding processing on the initial acoustic representation of adjacent phonemes to obtain a coding information matrix corresponding to the target speech; the encoder is trained using supervised data; the encoder includes a multi-head attention mechanism layer, a residual connection layer, a regularization layer, and a forward forwarding layer.

[0147] In one implementation of this embodiment, the decoding unit 504 includes:

[0148] The decoding subunit is used to input the encoding information matrix and the phoneme sequence and / or phoneme boundary information into a pre-built decoder corresponding to the evaluation dimension for fusion decoding according to the characteristics of the preset N evaluation dimensions to obtain complete acoustic features; the decoder is trained using supervised data; the encoder includes a multi-head attention mechanism layer, a residual connection layer, a regularization layer, and a forward forwarding layer;

[0149] A second input subunit is configured to input the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension, so as to map the complete acoustic features into a scoring space corresponding to the evaluation dimension, and obtain a score of the target speech corresponding to the evaluation dimension;

[0150] The fusion subunit is used to perform fusion processing on the scores of the target speech in the N evaluation dimensions to obtain a processed score as the evaluation result of the target speech.

[0151] In one implementation of this embodiment, the N evaluation dimensions include accuracy and fluency; and the decoding subunit is specifically configured to:

[0152] Inputting the encoding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and inputting the encoding information matrix and phoneme boundary information into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature;

[0153] The second input subunit is specifically configured to:

[0154] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; and inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space, thereby obtaining a score for the fluency of the target speech.

[0155] In one implementation of this embodiment, the N evaluation dimensions include fluency and completeness; and the decoding subunit is specifically configured to:

[0156] Inputting the encoding information matrix and the phoneme boundary information into a pre-built fluency decoder for decoding to obtain a second complete acoustic feature, and inputting the encoding information matrix and the phoneme sequence into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature;

[0157] The second input subunit is specifically configured to:

[0158] Inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into a fluency scoring space to obtain a fluency score of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor for completeness to map the third complete acoustic feature into a completeness scoring space to obtain a completeness score of the target speech.

[0159] In one implementation of this embodiment, the N evaluation dimensions include accuracy and completeness; and the decoding subunit is specifically configured to:

[0160] Inputting the coding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and inputting the coding information matrix and the phoneme sequence into a pre-built integrity decoder for fusion decoding to obtain a third complete acoustic feature;

[0161] The second input subunit is specifically configured to:

[0162] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor of accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor of completeness to map the third complete acoustic feature into the completeness scoring space, thereby obtaining a score for the completeness of the target speech.

[0163] In one implementation of this embodiment, the N evaluation dimensions include accuracy, fluency, and completeness; and the decoding subunit is specifically configured to:

[0164] Inputting the coding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature; inputting the coding information matrix and the phoneme boundary information into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature; and inputting the coding information matrix and the phoneme sequence into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature;

[0165] The second input subunit is specifically configured to:

[0166] Inputting the first complete acoustic feature into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space, thereby obtaining a score for the fluency of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor for completeness to map the third complete acoustic feature into the completeness scoring space, thereby obtaining a score for the completeness of the target speech.

[0167] Furthermore, an embodiment of the present application also provides a pronunciation evaluation device, comprising: a processor, a memory, and a system bus;

[0168] The processor and the memory are connected via the system bus;

[0169] The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned pronunciation evaluation method.

[0170] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned pronunciation evaluation method.

[0171] Furthermore, an embodiment of the present application also provides a computer program product, which, when running on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned pronunciation evaluation method.

[0172] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments.

[0173] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the methods.

[0174] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0175] The above description of the disclosed embodiments is intended to enable one of ordinary skill in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one of ordinary skill in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A pronunciation evaluation method, characterized in that: include: Acquiring a target speech to be evaluated and extracting acoustic features of the target speech; Performing segmentation and alignment processing on the acoustic features of the target speech and the standard text corresponding to the target speech to obtain valid audio features, phoneme sequences, and / or phoneme boundary information corresponding to the target speech; Performing feature fusion and encoding processing of adjacent phonemes on the effective audio features to obtain a coding information matrix corresponding to the target speech; According to the characteristics of the preset N evaluation dimensions, the coding information matrix, and the phoneme sequence and / or phoneme boundary information are fused and decoded, and the pronunciation evaluation of the target speech is performed based on the decoding results to obtain the evaluation result of the target speech; N is a positive integer greater than 0.

2. The method according to claim 1, characterized in that The segmentation and alignment processing of the acoustic features of the target speech and the standard text corresponding to the target speech to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information, includes: Determining a standard phoneme sequence corresponding to the standard text, and determining a motivational word based on the standard phoneme sequence; The standard phoneme sequence and the acoustic features of the target speech are input into a preset acoustic model, the audio frame sequence and phoneme sequence of the target speech are segmented and aligned, and the processing results are optimized and adjusted using the excitation words to obtain the effective audio features corresponding to the target speech, as well as the phoneme sequence and / or phoneme boundary information.

3. The method according to claim 2, characterized in that The preset acoustic model is a connection-based temporal classification (CTC) acoustic model.

4. The method according to claim 1, wherein The performing feature fusion and encoding processing of adjacent phonemes on the effective audio features to obtain a coding information matrix corresponding to the target speech includes: Inputting the effective audio features into a pre-trained acoustic model to map them to a preset fixed-dimensional vector space to obtain a vector of the preset fixed dimension as an initial acoustic representation; The pre-built encoder is used to perform feature fusion and encoding processing of adjacent phonemes on the initial acoustic representation to obtain a coding information matrix corresponding to the target speech.

5. The method according to claim 1, wherein The method further comprises: performing fusion decoding on the encoding information matrix and the phoneme sequence and / or phoneme boundary information according to the characteristics of the preset N evaluation dimensions, and evaluating the target speech according to the decoding results to obtain the evaluation result of the target speech, including: According to the characteristics of the preset N evaluation dimensions, the encoding information matrix, and the phoneme sequence and / or phoneme boundary information are input into a pre-built decoder corresponding to the evaluation dimension for fusion decoding to obtain a complete acoustic feature; Inputting the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic features into the scoring space of the corresponding evaluation dimension, thereby obtaining a score of the target speech corresponding to the evaluation dimension; The scores of the target speech in the N evaluation dimensions are fused to obtain a processed score as the evaluation result of the target speech.

6. The method according to claim 5, characterized in that The N evaluation dimensions include accuracy and fluency. According to the characteristics of the preset N evaluation dimensions, the encoding information matrix and the phoneme sequence and / or phoneme boundary information are input into a pre-built decoder corresponding to the evaluation dimension for fusion decoding to obtain a complete acoustic feature, including: Inputting the encoding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature, and inputting the encoding information matrix and phoneme boundary information into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature; Inputting the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic features into the scoring space corresponding to the evaluation dimension to obtain the score of the target speech corresponding to the evaluation dimension includes: Inputting the first complete acoustic feature into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; and inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space, thereby obtaining a score for the fluency of the target speech.

7. The method according to claim 5, characterized in that The N evaluation dimensions include accuracy, fluency, and completeness. According to the characteristics of the preset N evaluation dimensions, the encoding information matrix and the phoneme sequence and / or phoneme boundary information are input into a pre-built decoder corresponding to the evaluation dimension for fusion decoding to obtain complete acoustic features, including: Inputting the coding information matrix into a pre-built accuracy decoder for decoding to obtain a first complete acoustic feature; inputting the coding information matrix and the phoneme boundary information into a pre-built fluency decoder for fusion decoding to obtain a second complete acoustic feature; and inputting the coding information matrix and the phoneme sequence into a pre-built completeness decoder for fusion decoding to obtain a third complete acoustic feature; Inputting the complete acoustic features into a pre-built fully connected layer regressor corresponding to the evaluation dimension to map the complete acoustic features into the scoring space corresponding to the evaluation dimension to obtain the score of the target speech corresponding to the evaluation dimension includes: Inputting the first complete acoustic feature into a pre-built fully connected layer regressor for accuracy to map the first complete acoustic feature into the accuracy scoring space, thereby obtaining a score for the accuracy of the target speech; inputting the second complete acoustic feature into a pre-built fully connected layer regressor for fluency to map the second complete acoustic feature into the fluency scoring space, thereby obtaining a score for the fluency of the target speech; and inputting the third complete acoustic feature into a pre-built fully connected layer regressor for completeness to map the third complete acoustic feature into the completeness scoring space, thereby obtaining a score for the completeness of the target speech.

8. A pronunciation evaluation device, characterized in that: include: an acquisition unit, configured to acquire a target speech to be evaluated and extract acoustic features of the target speech; an alignment unit, configured to segment and align the acoustic features of the target speech and the standard text corresponding to the target speech, to obtain valid audio features, phoneme sequences, and / or phoneme boundary information corresponding to the target speech; An encoding unit, configured to perform feature fusion and encoding processing of adjacent phonemes on the effective audio features to obtain an encoding information matrix corresponding to the target speech; A decoding unit is used to fuse and decode the coding information matrix, the phoneme sequence and / or the phoneme boundary information according to the characteristics of the preset N evaluation dimensions, and to perform pronunciation evaluation on the target speech according to the decoding results to obtain the evaluation result of the target speech; N is a positive integer greater than 0.

9. A pronunciation evaluation device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is configured to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor is enabled to perform the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 7.

Images