A method, device and electronic device for detecting pronunciation errors

Abstract

An embodiment of the present application discloses a method, apparatus, and electronic device for pronunciation error detection. The method includes: obtaining a standard speech and a user speech to be detected, where the standard speech and the user speech correspond to the same reading text; determining a first content feature vector for characterizing the content information of the standard speech, and determining a second content feature vector for characterizing the content information of the user speech; and performing pronunciation error detection on the user speech by using the first content feature vector and the second content feature vector to improve the accuracy of the error detection result.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular, to a method, apparatus, and electronic device for pronunciation error detection. Background Art

[0002] With the development of computer technology, the use of computer technology to process and analyze speech has been widely applied. Pronunciation error detection refers to detecting whether there are errors in the speech issued by a user, and pronunciation error detection has become a research hotspot in speech technology and has gradually attracted people's attention.

[0003] Currently, the method for pronunciation error detection is as follows: align the speech to be error-detected with the text to be read, obtain multiple speech segments and multiple corresponding phoneme texts, and input the speech segments and the corresponding phoneme texts into a speech error detection model to obtain the error detection results of each speech segment, thereby obtaining the error detection result of the speech to be error-detected.

[0004] Inputting the speech segment and the corresponding phoneme text into the speech error detection model is to perform pronunciation error detection using the content information of the speech. For a set of confusing phonemes that are easily confused, since the content information of the speech is relatively close, when using the above method for pronunciation error detection, the accuracy of the obtained error detection result is relatively low. Summary of the Invention

[0005] In view of this, the present application provides a method, apparatus, and electronic device for pronunciation error detection to improve the accuracy of the error detection result.

[0006] In a first aspect, the present application provides a method for pronunciation error detection, the method including:

[0007] Obtain a standard speech and a user speech to be detected, where the standard speech and the user speech correspond to the same text to be read;

[0008] Determine a first content feature vector for characterizing the content information of the standard speech, and determine a second content feature vector for characterizing the content information of the user speech;

[0009] Perform pronunciation error detection on the user speech using the first content feature vector and the second content feature vector.

[0010] In a possible implementation manner, determining a first content feature vector for characterizing the content information of the standard speech specifically includes:

[0011] Determine the Mel-frequency cepstral coefficients of the standard speech;

[0012] Input the Mel-frequency cepstral coefficients of the standard speech into a pre-trained variational autoencoder (VEA) to generate a first content feature vector.

[0013] In a possible implementation, determining a second content feature vector for characterizing the content information of the user's speech specifically includes:

[0014] Determine the Mel-frequency cepstral coefficients of the user's speech;

[0015] Input the Mel-frequency cepstral coefficients of the user's speech into the pre-trained variational autoencoder (VEA) to generate the second content feature vector.

[0016] In a possible implementation, using the first content feature vector and the second content feature vector to perform pronunciation error detection on the user's speech specifically includes:

[0017] Perform context enhancement on the first content feature vector to obtain a first enhanced vector, and perform context enhancement on the second content feature vector to obtain a second enhanced vector;

[0018] Determine the phoneme text corresponding to the read text, and determine the phoneme representation vector according to the phoneme text;

[0019] Use the phoneme representation vector, the first enhanced vector, and the second enhanced vector to perform pronunciation error detection on the user's speech.

[0020] In a possible implementation, performing context enhancement on the first content feature vector to obtain a first enhanced vector, and performing context enhancement on the second content feature vector to obtain a second enhanced vector specifically includes:

[0021] Input the first content feature vector into a preset semantic enhancement model to obtain a first enhanced vector. The preset semantic enhancement model is used to perform context enhancement to enhance the semantics in the content feature vector;

[0022] Input the second content feature vector into the preset semantic enhancement model to obtain a second enhanced vector.

[0023] In a possible implementation, using the phoneme representation vector, the first enhanced vector, and the second enhanced vector to perform pronunciation error detection on the user's speech specifically includes:

[0024] Use the first enhanced vector and the phoneme representation vector to determine a first representation matrix, where the first representation matrix is used to represent the correlation degree between each phoneme in the phoneme text and each frame of speech in the standard speech;

[0025] Use the second enhanced vector and the phoneme representation vector to determine a second representation matrix, where the second representation matrix is used to represent the correlation degree between each phoneme in the phoneme text and each frame of speech in the user's speech;

[0026] Use the phoneme representation vector, the first enhanced vector, and the first representation matrix to determine a first fusion vector, where the first fusion vector is the standard speech fusion vector for obtaining phoneme perception;

[0027] Determine a second fusion vector by using a phoneme representation vector, a second enhancement vector, and a second representation matrix, where the second fusion vector is a user speech fusion vector obtained by phoneme perception;

[0028] Perform pronunciation error detection by using the first fusion vector and the second fusion vector.

[0029] In a possible implementation, performing pronunciation error detection by using the first fusion vector and the second fusion vector specifically includes:

[0030] Obtain the difference value, sum value, and dot product of the first fusion vector and the second fusion vector;

[0031] Concatenate the first fusion vector, the second fusion vector, the difference value, the sum value, and the dot product to obtain a comparison representation vector;

[0032] Input the comparison identification vector into a preset error detection model to obtain the error detection result of each phoneme in the phoneme text.

[0033] In a second aspect, the present application provides a device for pronunciation error detection, where the device includes: an acquisition unit, a first determination unit, and a pronunciation error detection unit;

[0034] The acquisition unit is configured to acquire a standard speech and a user speech to be detected, where the standard speech and the user speech correspond to the same reading text;

[0035] The first determination unit is configured to determine a first content feature vector for characterizing the content information of the standard speech, and determine a second content feature vector for characterizing the content information of the user speech;

[0036] The pronunciation error detection unit is configured to perform pronunciation error detection on the user speech by using the first content feature vector and the second content feature vector.

[0037] In a third aspect, the present application provides an electronic device, where the electronic device includes a processor and a memory. Among them, the memory stores code, and the processor is configured to call the code stored in the memory to implement the pronunciation error detection method in any one of the above.

[0038] In a fourth aspect, the present application provides a computer-readable storage medium, where the computer-readable storage medium is configured to store a computer program, and the computer program is configured to execute the pronunciation error detection method in any one of the above.

[0039] Currently, when inputting a voice segment and the corresponding phoneme text into a voice error detection model, the content information of the voice is used for pronunciation error detection. For a set of easily confused phonemes, such as "in" and "ing", since the content information of the voice is relatively similar, it is difficult to obtain features for distinguishing the pronunciation differences between the confused phonemes when using the above method for pronunciation error detection, and the accuracy of the obtained error detection result is relatively low.

[0040] Adopting the solution of the present application, the standard voice and the user voice correspond to the same reading text. The standard voice can be used to obtain features for distinguishing pronunciation differences, and even in the presence of confused phonemes, the user voice can be accurately error-detected, thereby improving the accuracy of the error detection result.

[0041] Currently, the training data of the voice error detection model used for voice error detection includes a phoneme table and the phoneme text corresponding to each phoneme in the phoneme table. In the actual voice error detection process, various situations may occur for the phonemes included in the user voice. For example, the user voice may contain phonemes that do not exist in the phoneme table, that is, the user voice contains defective phonemes. At this time, since the training data of the voice error detection model does not include the phoneme b, for the voice error detection model, the test data has features that are not in the training data. Therefore, the error detection result obtained through this voice error detection model is inaccurate.

[0042] Adopting the solution of the present application, the standard voice and the user voice correspond to the same reading text, and the standard voice is used as the standard for judging the user voice, so the problem of defective phonemes will not occur. Therefore, the accuracy of pronunciation error detection can be improved.

[0043] Currently, the method of inputting a voice segment and the corresponding phoneme text into a voice error detection model to obtain an error detection result is highly dependent on the segmentation result of the user voice. For example, the pronunciation correctness of a phoneme is calculated based on the segmentation boundary, generally represented by GOP (goodness of pronunciation), and the discrimination of whether a phoneme pronunciation is correct or not is performed based on a threshold method to obtain the error detection result of the user voice. When the user voice contains large background noise or the user voice has abnormal truncation, since the segmentation result of the user voice may be abnormal, the accuracy of pronunciation error detection is relatively low.

[0044] In the solution of the present application, the correlation degree between a phoneme and each frame of voice data is characterized by a phoneme attention representation matrix, and the time boundary information of the phoneme does not need to be evaluated. Therefore, the accuracy of pronunciation error detection can be improved.

[0045] Currently, the method of inputting a speech segment and the corresponding phoneme text into a speech error detection model to obtain an error detection result only performs error detection on isolated speech segments in the user's speech, lacking the phoneme pronunciation information of the context. However, when a natural person performs speech error detection, the phoneme pronunciation information of the context is usually combined to obtain an error detection result for pronunciation. Therefore, the current method of pronunciation error detection is inconsistent with the way a natural person judges speech errors.

[0046] In the solution of this application, instead of performing error detection on the speech segment corresponding to a single phoneme in isolation, a semantic context enhancement process is performed on the speech, and the correlation between the phoneme and each frame of speech data is characterized by a phoneme attention representation matrix, so that the phonemes in the user's speech can be error-detected in combination with the phoneme pronunciation situation of the context, improving the consistency with the way a natural person judges speech errors and realizing end-to-end pronunciation error detection for the entire speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 is a flowchart of a method for pronunciation error detection provided by an embodiment of this application;

[0048] Figure 2 is a flowchart of a variational autoencoder (VEA) training method provided by an embodiment of this application;

[0049] Figure 3 is a schematic structural diagram of a semantic enhancement network layer provided by an embodiment of this application;

[0050] Figure 4 is a schematic structural diagram of a speech-phoneme model provided by an embodiment of this application;

[0051] Figure 5 is a schematic structural diagram of a model for a method for pronunciation error detection provided by an embodiment of this application;

[0052] Figure 6 is a schematic structural diagram of a device for pronunciation error detection provided by an embodiment of this application;

[0053] Figure 7 is a schematic structural diagram of an electronic device provided by an embodiment of this application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0054] Currently, a speech segment and the corresponding phoneme text are input into a speech error detection model, and the content information of the speech is used to obtain the result of pronunciation detection. For a set of easily confused phonetic confusions, since the content information of the speech is relatively close, when performing pronunciation error detection using the above method, the accuracy of the obtained error detection result is relatively low.

[0055] Based on this, in the embodiments of the present application provided by the applicant, standard speech and user speech to be detected are obtained, and the standard speech and the user speech correspond to the same reading text; a first content feature vector for characterizing the content information of the standard speech is determined, and a second content feature vector for characterizing the content information of the user speech is determined; the pronunciation of the user speech is detected using the first content feature vector and the second content feature vector.

[0056] Adopting the solution of the present application, the standard speech and the user speech correspond to the same reading text. By using the standard speech, features for distinguishing pronunciation differences can be obtained, and even in the presence of confusing sounds, the user speech can be detected for errors more accurately, thereby improving the accuracy of the error detection result.

[0057] To facilitate the understanding of the technical solution provided by the embodiments of the present application, the terms in the embodiments of the present application are first introduced.

[0058] Phoneme: The smallest speech unit divided according to the natural attributes of speech. Analyzing based on the pronunciation actions in a syllable, one pronunciation action constitutes one phoneme.

[0059] Timbre: Timbre is one of the attributes of sound. Due to different materials and structures, the timbres of the sounds emitted by sound-producing bodies are different.

[0060] When different people speak, the vibrations of their vocal cords have different characteristics, resulting in different characteristics of the sound in terms of waveform.

[0061] Speaker representation vector: Used to distinguish different speakers.

[0062] A voice signal contains various information, such as the speaker's voiceprint.

[0063] The voiceprint includes the voice features that can represent the speaker's identity in the voice information included, and the voice model established based on these voice features.

[0064] Voiceprint recognition refers to identifying the speaker's identity based on the voiceprint in the voice signal that can represent the speaker's personal information using identification technologies such as computers.

[0065] Traditional voiceprint recognition is mainly based on statistical models, such as the Gaussian Mixture Model-Universal Background Model (GMM-UBM) architecture.

[0066] To improve the ability to express speaker characteristics under limited data, multiple subspace models have been proposed one after another, such as the i-vector model. The i-vector model introduces the concept of a speaker embedding vector, which usually uses a continuous vector of a fixed length to characterize speaker characteristics. Through the speaker embedding vector, a space for describing speaker characteristics is constructed.

[0067] With the development of deep learning technology, speaker recognition models based on deep learning methods have been proposed one after another, such as the d-vector model, the x-vector model, etc. The speaker embedding vector obtained by the speaker recognition model based on the deep learning method can be called a deep speaker embedding.

[0068] Variational auto-encoder (VAE): A type of generative model.

[0069] To facilitate the understanding of the technical solutions provided by the embodiments of the present application, a method, device, and electronic device for pronunciation error detection provided by the embodiments of the present application will be described below with reference to the accompanying drawings.

[0070] Although the exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. Other embodiments obtained by those skilled in the art based on the embodiments of the present application without making creative contributions all fall within the protection scope of the present application.

[0071] In the claims, the specification, and the drawings of the present application, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion.

[0072] The present application provides a method for pronunciation error detection, which will be specifically described below with reference to the accompanying drawings.

[0073] Please refer to Figure 1 , Figure 1 which is a flowchart of the method for pronunciation error detection provided by the embodiments of the present application.

[0074] As Figure 1 shown, the method for pronunciation error detection in the embodiments of the present application includes S101 - S103.

[0075] S101. Obtain the standard speech and the user speech to be detected, where the standard speech and the user speech correspond to the same reading text.

[0076] The standard speech is used to determine whether there are errors in the user speech.

[0077] For example, for an oral proficiency test, the standard speech is the speech that meets the requirements of this oral proficiency test.

[0078] In some possible implementation manners, the standard speech may be manually input.

[0079] Specifically, for a Putonghua proficiency test, the standard speech is obtained by professional persons who meet the Putonghua test standards reading the reading text and through professional equipment under professional conditions. The standard speech is the speech that meets the Putonghua test standards of this test.

[0080] S102. Determine a first content feature vector for characterizing the content information of the standard speech, and determine a second content feature vector for characterizing the content information of the user speech.

[0081] The speech contains content information and speaker information.

[0082] The speaker information includes, for example, the voiceprint information of the speaker, which is used to represent the identity of the speaker.

[0083] The first content feature vector is used to characterize the content information of the standard speech and does not include the speaker information of the standard speech; the second content feature vector is used to characterize the content information of the user speech and does not include the speaker information of the user speech.

[0084] S103. Use the first content feature vector and the second content feature vector to perform pronunciation error detection on the user speech.

[0085] Since the first content feature vector is used to characterize the content information of the standard speech and the second content feature vector is used to characterize the content information of the user speech, by using the first content feature vector and the second content feature vector, it is possible to determine that there are differences between the content information of the standard speech and the content information of the user speech, so as to determine whether there are errors in the content information of the user's semantics.

[0086] Adopting the solution of the present application, the standard speech and the user speech correspond to the same reading text. By using the standard speech, features for distinguishing pronunciation differences can be obtained, and even in the case of confusing sounds, the user speech can be detected for errors more accurately, thereby improving the accuracy of the error detection result.

[0087] The present application also provides another method for pronunciation error detection.

[0088] The method for pronunciation error detection in the embodiments of the present application includes S201 - S213.

[0089] S201. Obtain the standard speech and the user speech to be detected.

[0090] The standard speech and the user speech correspond to the same read text.

[0091] The standard speech is used to determine whether there are errors in the user speech.

[0092] For example, for an oral proficiency test, the standard speech is the speech that meets the requirements of this oral proficiency test.

[0093] In some possible implementation manners, the standard speech can be manually input.

[0094] Specifically, for a Putonghua proficiency test, the standard speech is obtained by having professionals who meet the standards of this Putonghua test read the read text and recording it through professional equipment under professional conditions. The standard speech is the speech that meets the standards of this Putonghua test.

[0095] Specifically, for an oral learning application software, there is usually a learning goal. The standard speech is obtained by having professionals who have achieved the learning goal read the read text and recording it through professional equipment under professional conditions. The standard speech is the speech that meets this learning goal.

[0096] S202. Determine the Mel-frequency cepstral coefficients of the user speech and the Mel-frequency cepstral coefficients of the standard speech.

[0097] The Mel-frequency cepstral coefficients (mfcc) are used to characterize the frequency characteristics of speech and to describe the acoustic characteristics of speech.

[0098] The Mel-frequency is obtained based on the auditory characteristics of the human ear and has a non-linear correspondence with the Hertz frequency. The mfcc is the Hertz frequency characteristic obtained by using the correspondence between the Mel-frequency and the Hertz frequency, and is mainly used for feature extraction and reducing the operation dimension.

[0099] For example, in S202, obtain the Mel-frequency cepstral coefficients mfcc1 of the user speech and the Mel-frequency cepstral coefficients mfcc2 of the standard speech.

[0100] The dimensions of both mfcc1 and mfcc2 are T*d, where T is the number of frames of the speech data and d is the dimension of the mfcc feature. For example, d can take 40.

[0101] The Mel-frequency cepstral coefficients are the spectral feature vectors of speech and are used to represent the spectral characteristics of speech. In addition to using the Mel-frequency cepstral coefficients, the spectral characteristics of speech can also be represented in other ways.

[0102] The human ear's response to the sound spectrum is non - linear. Experience shows that if we can design a front - end processing algorithm to process audio in a way similar to the human ear, the performance of speech recognition can be improved. FilterBank (fbank) is such an algorithm, and the result obtained is used to represent the spectral features of speech.

[0103] In a possible implementation, a vector representing the spectral features of speech is obtained through the fbank algorithm.

[0104] S203: Input the Mel - Frequency Cepstral Coefficients (MFCCs) of the user's speech into the pre - trained Variational Auto - Encoder (VEA) to generate a content representation vector of the user's speech.

[0105] The content representation vector of the user's speech is used to characterize the content information in the user's speech.

[0106] Speech contains content information and speaker information.

[0107] The content representation vector of the user's speech obtained through S203 contains less or no speaker information.

[0108] VEA is pre - trained. The following is the implementation method for training VEA provided by the embodiments of this application.

[0109] Please refer to Figure 2 , Figure 2 which is the flowchart of the training method of the Variational Auto - Encoder (VEA) provided by the embodiments of this application.

[0110] The loss function is usually used during model training.

[0111] The loss function is used to measure the difference between the model output and the expected result, and is used to optimize the model so that the optimized model can make the model output close to the expected result.

[0112] During the process of training VEA, both the model input and the expected result of VEA are the spectral feature vectors of speech. That is, for VEA training, the training objective is that the model input and the model output of VEA are the same.

[0113] VEA includes a VEA encoding layer (encoder) and a VEA decoding layer (decoder).

[0114] After speech is used as the model input to the VEA encoding layer, a dimensionality - reduced speech feature representation vector xt is obtained.

[0115] For example, the original spectral feature vector s is used as the model input of the VEA, with a dimension of T*d. s is input into the VEA encoding layer to obtain the reduced-dimensional speech feature representation vector xt, with a dimension of 1*D, where D is the low-dimensional space feature of the VAE. For example, D can be set to 160.

[0116] xt is input into the VEA decoding layer to obtain the reconstructed spectral feature vector s'.

[0117] Since the training objective is that the model input and output of the VEA are the same, after the model training is completed, the reconstructed spectral feature vector s' and the original spectral vector s are approximately the same.

[0118] In a possible implementation, the loss function L is:

[0119] L(s, s', θ) = MSE(s, s', θ)

[0120] s is the original spectral feature vector, s' is the spectral feature vector reconstructed by the VEA, and θ is the coefficient for model training.

[0121] The mean squared error MSE refers to the expected value of the square of the difference between the parameter estimated value and the parameter value, and is usually a convenient method to measure the "average error".

[0122] During the training of the VEA, the model output s' in the loss function is replaced by the superposition of the content vector s1 and the speaker vector X-Vector to complete the training process.

[0123] The speaker vector X-Vector is obtained by inputting s into the x-vector model, and is used to represent the speaker information in the speech.

[0124] The x-vector model is a speaker recognition model based on deep learning methods. By performing speaker recognition on the speech, a speaker vector that can represent the speaker's identity is obtained.

[0125] The speaker vector can also be obtained by other means.

[0126] In a possible way, the original spectral vector s is input into the d-vector model to obtain the speaker vector.

[0127] During the training of the VEA, since the model output s' in the loss function is replaced by the superposition of the content representation vector s1 and the speaker vector X-Vector, during the process of reconstructing the speech feature representation vector xt in the VEA decoding layer, the model's learning of speaker-related features is relatively weak.

[0128] After the VEA training is completed, the model input s and the model output s' of the VEA are approximately the same. That is, the model input s, which is the superposition of the content representation vector s1 and the speaker vector X-Vector, is approximately the same.

[0129] After the VEA training is completed, s1 contains less speaker information in the speech and can be used to represent the content information in the speech.

[0130] For the spectral features of the reconstructed speech, s1 contains the content information in the speech, and X-Vector contains the speaker information in the speech. That is, after the original spectral feature vector s is input into the trained VEA, the speaker information in the spectral features can be mostly removed, and the content representation vector s1 that only contains content information can be obtained.

[0131] In S203, based on the pre-trained model, the speaker-related features in the user speech features are removed. In order to detect errors in the user speech, it is necessary to compare the user speech with the standard speech. Therefore, it is necessary to exclude the features of the speaker information in the user speech and the standard speech, and only retain the features related to the content in the speech.

[0132] In S203, mfcc1 is input into the pre-trained VEA. The dimension of mfcc1 is T*d, and the content representation vector con1 of the user speech is generated. The dimension of con1 is T*d.

[0133] S204: Input the mel-frequency cepstral coefficients of the standard speech into the pre-trained VEA to generate the content representation vector of the standard speech.

[0134] The process of generating the content representation vector of the standard speech in S204 is similar to the process of generating the content representation vector of the user speech in S203. The same pre-trained VEA can be used. The training process of the VEA has been described above and will not be elaborated here.

[0135] In S204, mfcc2 is input into the pre-trained VEA. The dimension of mfcc2 is T*d, and the content representation vector con2 of the user speech is generated. The dimension of con2 is T*d.

[0136] S205: Perform context enhancement on the content representation vector of the user speech to obtain the content-enhanced representation vector of the user speech.

[0137] Since the features representing the speech in mfcc are all at the frame level, usually a single frame of speech data contains less information and it is difficult to represent the content related to the speech. Therefore, it is necessary to perform semantic context enhancement on the content representation vector of the user speech to obtain the content-enhanced representation vector of the user speech.

[0138] In a possible implementation, the content representation vector of the input user speech is sent to the semantic enhancement network layer to obtain the content enhanced representation vector of the user speech, that is, the content representation vector of the user speech is encoded.

[0139] In a possible case, the structure of the semantic enhancement network layer is the superposition of 3 vgg blocks and 6 Transformer structures.

[0140] Please refer to Figure 3 , Figure 3 which is the schematic diagram of the structure of the semantic enhancement network layer provided by the embodiments of the present application.

[0141] The vgg block is the superposition of convolutional layers, used to strengthen the fusion of local features of speech; the Transformer is composed of a multi-head attention mechanism and multiple semantic perception layers, which can strengthen the long-distance dependence information of speech.

[0142] In the vgg block, the input passes through a two-dimensional convolutional layer, a layer normalization layer, and an activation function respectively;

[0143] In the Transformer, the input passes through a self-attention layer, a residual and normalization layer, a forward perception layer, and a residual and normalization layer respectively.

[0144] Due to the maxpooling downsampling process in the vgg block, generally, the dimension of the content enhanced representation vector will be reduced to 1 / 4 of the content representation vector.

[0145] For example, the content representation vector con1 of the input user speech is sent to the semantic enhancement model, the dimension of con1 is T*d, and the content enhanced representation vector At1 of the user speech is obtained, and the dimension of At1 is (T / 4)*d.

[0146] S206. Perform context enhancement on the content representation vector of the standard speech to obtain the content enhanced representation vector of the standard speech.

[0147] The process of obtaining the content enhanced representation vector of the standard speech in S206 is similar to the process of obtaining the content enhanced representation vector of the user speech in S205, and the same semantic enhancement model can be used, which will not be elaborated here.

[0148] For example, the content representation vector con2 of the input standard speech is sent to the semantic enhancement model, the dimension of con2 is T*d, and the content enhanced representation vector At2 of the standard speech is obtained, and the dimension of At2 is (T / 4)*d.

[0149] S207. Determine the representation vector of the phoneme text to obtain the phoneme representation vector, and the phoneme text is obtained according to the reading text.

[0150] The phoneme text contains the sequence of phonemes in the spoken text, that is, all the phonemes in the spoken text.

[0151] The phoneme representation vector Pt converts the discretized information of the phoneme text into a continuous vector representation.

[0152] In a possible implementation, a phoneme embedding matrix is determined, and the phoneme text is input into the phoneme embedding matrix to obtain the phoneme representation vector Pt. The embedding matrix includes the coefficients of the phoneme representation vector Pt.

[0153] Use the phoneme embedding matrix to perform vector representation on the phoneme text to be detected.

[0154] For example, it is determined that the dimension of the phoneme embedding matrix is 109*d, where 109 is the number of phonemes, d is the dimension of the embedding matrix. In the above description, the dimension of the content enhanced representation vector At1 of the user speech is (T / 4)*d. Usually, (T / 4)>Tp. The dimension of the phoneme representation vector Pt is Tp*d, where Tp is the number of phonemes corresponding to the spoken text.

[0155] S208. Obtain the phoneme attention representation matrix of the user speech according to the content enhanced representation vector of the user speech and the phoneme representation vector. The phoneme attention representation matrix of the user speech is used to represent the correlation degree between each phoneme in the phoneme text and each frame of speech data in the user speech.

[0156] That is, the phoneme attention representation matrix is used to represent the receptive field or importance of each phoneme in the phoneme text for each frame of speech data.

[0157] Speech is a frame-level feature, and it is necessary to implement phoneme perception and all related frame-level features. In the solution provided in this embodiment, this perception is a weighted perception mechanism based on the attention mechanism.

[0158] The following is the method provided in this embodiment for determining the above phoneme attention representation matrix.

[0159] Respectively input the content enhanced representation vector At1 of the user speech, the phoneme representation vector Pt, and the phoneme representation vector Pt into the first network FC1, the second network FC2, and the third network FC3 to obtain Q, K, and V.

[0160] The first network FC1, the second network FC2, and the third network FC3 are all fully connected layers.

[0161] Fully connected layers usually play the role of "classifier" in the entire convolutional neural network.

[0162] The phoneme attention representation matrix Attention of the user speech

[0163]

[0164] d k is the dimension of the content enhancement representation vector At1 of the user's speech. The softmax function contains is to prevent the inner product from being too large and avoid the phenomenon of spikes in the softmax function values.

[0165] What is obtained in the above process is the attention weight at the phoneme level.

[0166] The softmax function, also known as the normalized exponential function, is a generalization of the binary classification function sigmoid function to multi-classification. The purpose is to represent the results of multi-classification in the form of probabilities.

[0167] For example, the dimension of the content enhancement representation vector At1 of the user's speech is (T / 4)*d, that is, the number of frames of the user's speech data is T / 4; the dimension of the phoneme representation vector Pt is Tp*d, that is, the number of phonemes in the phoneme sequence of the read text is Tp.

[0168] The dimension of the phoneme attention representation matrix of the user's speech is (T / 4)*Tp.

[0169] The number of columns of the phoneme attention representation matrix of the user's speech is the number of phonemes in the phoneme sequence of the read text. Each column of data corresponds to the attention of each phoneme in the phoneme sequence.

[0170] There are (T / 4) data in each column of data. Each data is the attention of the phoneme to each frame of speech data. Due to the characteristics of the softmax function, for the (T / 4) data in each column of data, the sum is 1, that is, the sum of the attention of the phoneme to all frames of speech data is 1.

[0171] In S208, a weighted matrix of phonemes relative to each frame of speech is obtained. This weighted matrix is the receptive field and importance of the phoneme's attention to each frame-level feature.

[0172] S209: According to the content enhancement representation vector of the standard speech and the phoneme representation vector Pt, obtain the phoneme attention representation matrix of the standard speech. The phoneme attention representation matrix of the standard speech is used to represent the correlation between each phoneme in the phoneme text and each frame of speech data in the standard speech.

[0173] The process of obtaining the phoneme attention representation matrix of the standard speech in S209 is similar to the process of obtaining the phoneme attention representation matrix of the user's speech in S208 and can be implemented in the same way, which will not be elaborated here.

[0174] S210. Input the content enhancement representation vector, phoneme representation vector, and phoneme attention representation matrix of the user speech into the phoneme speech interaction layer to obtain the phoneme-aware fused representation vector of the user speech.

[0175] In a possible implementation, the phoneme speech interaction layer is a multi-layer Transformer structure.

[0176] For example, the phoneme speech interaction layer is a 3-layer Transformer structure.

[0177] S211. Input the content enhancement representation vector, phoneme representation vector, and phoneme attention representation matrix of the standard speech into the phoneme speech interaction layer to obtain the phoneme-aware fused representation vector of the standard speech.

[0178] The process of obtaining the phoneme-aware fused representation vector of the standard speech in S211 is similar to the process of obtaining the phoneme-aware fused representation vector of the user speech in S210 and can be implemented in the same way, which will not be elaborated here.

[0179] In a possible implementation, S205, S207, S208, and S210 are implemented through different network layers of the same model.

[0180] For example, S205, S207, S208, and S210 are implemented through a speech-phoneme model to obtain the phoneme-aware fused representation vector of the user speech.

[0181] Please refer to Figure 4 , Figure 4 which is a schematic structural diagram of the speech-phoneme model provided by the embodiments of the present application.

[0182] The speech-phoneme model includes a speech encoding layer (speech-encoder), a phoneme encoding layer (phoneme-encoder), an attention layer (Phone-acoustic attention layer), and an interaction layer (phoneme-speech interaction layer); among them, the speech encoding layer is used to implement S205 to obtain the content enhancement representation vector of the user speech; the phoneme encoding layer is used to implement S207 to obtain the phoneme representation vector; the attention layer is used to implement S208 to obtain the phoneme attention representation matrix of the user speech; the interaction layer is used to implement S210 to obtain the phoneme-aware fused representation vector of the user speech.

[0183] In the phoneme encoding layer (phoneme-encoder), " <cos>"zh", "ong", "g", and "uo" are phonemes respectively.

[0184] For the training of the speech coding layer, phoneme coding layer, attention layer, and interaction layer in the speech-phoneme model, it can be achieved by means of joint training.

[0185] The training data can be a Chinese speech-transcription parallel corpus. This model is trained by backpropagation based on large-scale pre-training data. After training, the parameters are fixed and it is only used as a feature extraction tool for the speech-phoneme model.

[0186] Since the implementation manners of S206, S209, and S211 are similar to those of S205, S208, and S210 respectively, in some possible cases, S206, S209, and S211 can be implemented through the above speech-phoneme model to obtain a phoneme-aware standard speech fusion representation vector.

[0187] S212. Construct a comparison representation vector between the phoneme-aware user speech fusion representation vector and the phoneme-aware standard speech fusion representation vector.

[0188] In a possible implementation manner, for the phoneme-aware user speech fusion representation vector and the phoneme-aware standard speech fusion representation vector, determine the difference, sum, and dot product of the two representation vectors respectively, and splice the difference, sum, and dot product of the two representation vectors to obtain a comparison representation vector.

[0189] For example, the comparison representation vector C fuse can be expressed as:

[0190] C fuse = concat(C1, C2, C1 + C2, C1 - C2, C1 * C2, )

[0191] C1 is the phoneme-aware user speech fusion representation vector, C2 is the phoneme-aware standard speech fusion representation vector, and concat is the splicing function.

[0192] S213. Input the comparison representation vector into the error detection model to obtain the error detection result of each phoneme in the phoneme text.

[0193] In a possible implementation manner, the error detection model is expressed as:

[0194] Y t = sigmoid(W2Relu(W1C fuse + b1) + b2)

[0195] Among them, b1, b2, W1, and W2 are parameters of the error detection model. Specifically, b1 and b2 are parameters in vector form, and W1 and W2 are parameters in matrix form.

[0196] Y t is the error detection result of each phoneme obtained through the error detection model.

[0197] The expression of the sigmoid function is:

[0198] σ(z) = 1 / (1 + e^(-z)) z

[0199] When the sigmoid function approaches positive infinity or negative infinity, the function value approaches a smooth state.

[0200] The output range of the sigmoid function is (0, 1). Therefore, the sigmoid function can be used for binary classification.

[0201] The ReLU function is the Rectified Linear Unit (ReLU),

[0202] ReLU(z) = max(0, z)

[0203] The ReLU function is a commonly used activation function in artificial neural networks. In a neural network, the rectified linear transformation defines the non-linear output result of the neuron after linear transformation.

[0204] The objective function of the error detection model is:

[0205]

[0206] Among them, represents the absolute value of, W is the set of parameters in the detection model, is the conversion function.

[0207] n is the number of phonemes in all training samples, and c is the error detection annotation for each sample.

[0208] For example, when the training samples contain 1000 voices and each voice contains 10 phonemes, the number of phonemes in the training samples is 1000 * 10;

[0209] The error detection annotation for each sample is whether each phoneme in each sample is correct or incorrect.

[0210] By comparing the two phoneme-aware speech fusion representations of the user's speech and the standard speech, a multi-layer perceptron linear classifier is used to obtain the final error detection result.

[0211] Please refer to Figure 5 , Figure 5 which is a schematic diagram of the model structure of the pronunciation error detection method provided by an embodiment of the present application.

[0212] The pronunciation sequence of the examinee is the user voice, and the standard pronunciation sequence is the standard voice.

[0213] S205 and S206 are implemented through the content feature extraction layer of the voice to obtain the content identification vector of the user voice and the content representation vector of the standard voice, that is Figure 5 the content feature vectors of multiple examinee voice frames in

[0214] The voice fusion layer implements S205 - S211 to obtain the phoneme-aware user voice fusion representation vector and the phoneme-aware standard voice fusion representation vector, that is Figure 5 the phoneme-aware voice fusion identification of the examinee and the standard phoneme-aware voice fusion identification in

[0215] The voice representation comparison layer and the error detection output layer respectively implement S212 and S213, and the final error detection result is obtained after passing through the error detection output layer.

[0216] For the possible implementation manners of each layer, please refer to the above description and will not be elaborated here.

[0217] The present application also provides a pronunciation error detection device.

[0218] Please refer to Figure 6 , Figure 6 which is a schematic diagram of the structure of the pronunciation error detection device provided by an embodiment of the present application.

[0219] As Figure 6 shown, the pronunciation error detection device 200 includes: an acquisition unit 201, a first determination unit 202, and a pronunciation error detection unit 203.

[0220] The acquisition unit is configured to acquire a standard voice and a user voice to be detected, and the standard voice and the user voice correspond to the same reading text;

[0221] The first determination unit is configured to determine a first content feature vector for characterizing the content information of the standard voice, and determine a second content feature vector for characterizing the content information of the user voice;

[0222] The pronunciation error detection unit is configured to perform pronunciation error detection on the user voice by using the first content feature vector and the second content feature vector.

[0223] The units included in the above pronunciation error detection device and the connection relationships between the units can achieve the same technical effects as the pronunciation error detection method in the above embodiments. To avoid repetition, they will not be elaborated here.

[0224] This application also provides an electronic device.

[0225] Please refer to Figure 7 , Figure 7 which is a schematic structural diagram of the electronic device provided by the embodiment of this application.

[0226] As Figure 7 shown, the electronic device 300 includes a processor 301 and a memory 302. Among them, the memory stores code, and the processor is used to call the code stored in the memory to implement any of the above pronunciation error detection methods.

[0227] The units included in the above electronic device and the connection relationships between the units can achieve the same technical effects as the pronunciation error detection method in the above embodiments. To avoid repetition, they will not be elaborated here.

[0228] In an embodiment of this application, a computer-readable storage medium is also provided. The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the above pronunciation error detection method and can achieve the same technical effects. To avoid repetition, they will not be elaborated here. Among them, the computer-readable storage medium is, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disc, etc.

[0229] The above description of the disclosed embodiments enables those skilled in the art to implement or use this application. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.< / cos>

Claims

1. A method for detecting pronunciation errors, characterized in that, The method includes: Obtaining a standard speech and a user speech to be detected, where the standard speech and the user speech correspond to the same reading text; Determining a first content feature vector for characterizing the content information of the standard speech, and determining a second content feature vector for characterizing the content information of the user speech; Performing pronunciation error detection on the user speech by using the first content feature vector and the second content feature vector; The performing pronunciation error detection on the user speech by using the first content feature vector and the second content feature vector specifically includes: Performing context enhancement on the first content feature vector to obtain a first enhanced vector, and performing context enhancement on the second content feature vector to obtain a second enhanced vector; Determining a phoneme text corresponding to the reading text, and determining a phoneme representation vector according to the phoneme text; Determining a first representation matrix by using the first enhanced vector and the phoneme representation vector, where the first representation matrix is used to represent the correlation between each phoneme in the phoneme text and each frame of speech in the standard speech; Determining a second representation matrix by using the second enhanced vector and the phoneme representation vector, where the second representation matrix is used to represent the correlation between each phoneme in the phoneme text and each frame of speech in the user speech; Determining a first fusion vector by using the phoneme representation vector, the first enhanced vector, and the first representation matrix, where the first fusion vector is a standard speech fusion vector for obtaining phoneme perception; Determining a second fusion vector by using the phoneme representation vector, the second enhanced vector, and the second representation matrix, where the second fusion vector is a user speech fusion vector for obtaining phoneme perception; Performing pronunciation error detection by using the first fusion vector and the second fusion vector.

2. The method for pronunciation error detection according to claim 1, wherein The determining a first content feature vector for characterizing the content information of the standard speech specifically includes: Determining the Mel frequency cepstral coefficients of the standard speech; Inputting the Mel frequency cepstral coefficients of the standard speech into a pre-trained variational autoencoder (VEA) to generate the first content feature vector.

3. The method for detecting pronunciation errors according to claim 1, wherein The determining a second content feature vector for characterizing the content information of the user speech specifically includes: Determining the Mel frequency cepstral coefficients of the user speech; Inputting the Mel frequency cepstral coefficients of the user speech into a pre-trained variational autoencoder (VEA) to generate the second content feature vector.

4. The method for pronunciation error detection according to claim 1, wherein The performing context enhancement on the first content feature vector to obtain a first enhanced vector, and performing context enhancement on the second content feature vector to obtain a second enhanced vector specifically includes: Inputting the first content feature vector into a preset semantic enhancement model to obtain the first enhanced vector, where the preset semantic enhancement model is used to perform context enhancement to enhance the semantics in the content feature vector; Inputting the second content feature vector into the preset semantic enhancement model to obtain the second enhanced vector.

5. The pronunciation error detection method according to claim 1, wherein The performing pronunciation error detection by using the first fusion vector and the second fusion vector specifically includes: Obtain the difference, sum, and dot product of the first fusion vector and the second fusion vector; Concatenate the first fusion vector, the second fusion vector, the difference, the sum, and the dot product to obtain a comparison representation vector; Input the comparison representation vector into a preset error detection model to obtain the error detection results of each phoneme in the phoneme text.

6. A device for detecting pronunciation errors, characterized in that, The device includes: an acquisition unit, a first determination unit, and a pronunciation error detection unit; The acquisition unit is configured to acquire a standard speech and a user speech to be detected, where the standard speech and the user speech correspond to the same reading text; The first determination unit is configured to determine a first content feature vector for characterizing the content information of the standard speech, and determine a second content feature vector for characterizing the content information of the user speech; The pronunciation error detection unit is configured to perform pronunciation error detection on the user speech by using the first content feature vector and the second content feature vector; Specifically, the pronunciation error detection unit is configured to: Perform context enhancement on the first content feature vector to obtain a first enhanced vector, and perform context enhancement on the second content feature vector to obtain a second enhanced vector; Determine the phoneme text corresponding to the reading text, and determine a phoneme representation vector according to the phoneme text; Use the first enhanced vector and the phoneme representation vector to determine a first representation matrix, where the first representation matrix is used to represent the correlation between each phoneme in the phoneme text and each frame of speech in the standard speech; Use the second enhanced vector and the phoneme representation vector to determine a second representation matrix, where the second representation matrix is used to represent the correlation between each phoneme in the phoneme text and each frame of speech in the user speech; Use the phoneme representation vector, the first enhanced vector, and the first representation matrix to determine a first fusion vector, where the first fusion vector is a standard speech fusion vector for obtaining phoneme perception; Use the phoneme representation vector, the second enhanced vector, and the second representation matrix to determine a second fusion vector, where the second fusion vector is a user speech fusion vector for obtaining phoneme perception; Perform pronunciation error detection by using the first fusion vector and the second fusion vector.

7. An electronic device, characterized in that, The electronic device includes a processor and a memory, where the memory stores code, and the processor is configured to call the code stored in the memory to implement the pronunciation error detection method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the pronunciation error detection method according to any one of claims 1 to 5.

Images