Pronunciation error detection method and device, and pronunciation error detection model training method and device

Abstract

The present invention provides a pronunciation error detection method and device, and a pronunciation error detection model training method and device. The pronunciation error detection method comprises: obtaining a speech signal to be detected and its corresponding read text; extracting acoustic features of the speech signal to be detected and converting the read text into a phoneme sequence; obtaining acoustic features of at least one pair of confused phoneme pairs; and using the pronunciation error detection model to perform pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence, thereby improving the accuracy of pronunciation error detection.

Description

Technical Field

[0001] The present invention relates to the technical field of pronunciation error detection, and in particular to a pronunciation error detection method and device, and a pronunciation error detection model training method and device. Background Art

[0002] With the development of computer technology and speech recognition technology, Computer Assisted Pronunciation Training (CAPT) has become a research hotspot in the field of intelligent speech technology. The CAPT system can automatically evaluate the learner's pronunciation level and provide feedback and guidance on pronunciation errors.

[0003] Pronunciation error detection, which detects errors in user pronunciation, is a crucial component of the CAPT system. However, existing methods rely on forced alignment of speech segments, which places high demands on the precision of the forced alignment technology. Errors introduced by forced alignment can significantly impact the effectiveness of subsequent steps, significantly reducing the accuracy of pronunciation error detection. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a pronunciation error detection method and device, and a pronunciation error detection model training method and device, which can improve the accuracy of pronunciation error detection.

[0005] According to a first aspect of an embodiment of the present invention, a pronunciation error detection method is provided, comprising: obtaining a speech signal to be detected and its corresponding read text; extracting acoustic features of the speech signal to be detected and converting the read text into a phoneme sequence; obtaining acoustic features of at least one pair of confused phoneme pairs; and performing pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence using a pronunciation error detection model.

[0006] In one embodiment of the present invention, the above-mentioned obtaining of acoustic features of at least one pair of confused phoneme pairs includes: obtaining acoustic features of a speech sample, wherein the speech sample includes multiple speech segments corresponding to each pair of confused phoneme pairs in at least one pair of confused phoneme pairs, wherein each pair of confused phoneme pairs includes a first phoneme and a second phoneme; obtaining a covariance matrix corresponding to each pair of confused phoneme pairs based on the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; and fusing the acoustic features of the speech sample with the covariance matrix corresponding to each pair of confused phoneme pairs to obtain the acoustic features of each pair of confused phoneme pairs.

[0007] In one embodiment of the present invention, the above-mentioned method of obtaining the covariance matrix corresponding to each pair of confused phoneme pairs based on the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs includes: segmenting the speech sample to obtain the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; extracting the acoustic features of the multiple speech segments corresponding to the first phoneme and clustering them to obtain N first-class center vectors; extracting the acoustic features of the multiple speech segments corresponding to the second phoneme and clustering them to obtain N second-class center vectors; and performing dimensionality reduction on the N first-class center vectors and the N second-class center vectors to obtain a covariance matrix.

[0008] In one embodiment of the present invention, the above-mentioned pronunciation error detection method also includes: determining whether the read text contains confusing phonemes; when the read text contains confusing phonemes, performing pronunciation error detection based on the output results of the pronunciation error detection model and the phoneme classification model corresponding to the confusing phonemes.

[0009] According to a second aspect of an embodiment of the present invention, a method for training a pronunciation error detection model is provided, comprising: obtaining training samples, wherein the training samples include speech signal samples and their corresponding text samples, wherein the speech signal samples include speech information formed by a reader reading aloud the text samples; extracting acoustic features of the speech samples, and converting the text samples into phoneme sequences; obtaining acoustic features of at least one pair of confused phoneme pairs; and performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal samples, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence.

[0010] In one embodiment of the present invention, the above-mentioned obtaining of acoustic features of at least one pair of confused phoneme pairs includes: obtaining acoustic features of a speech sample, wherein the speech sample includes multiple speech segments corresponding to each pair of confused phoneme pairs in at least one pair of confused phoneme pairs, wherein each pair of confused phoneme pairs includes a first phoneme and a second phoneme; obtaining a covariance matrix corresponding to each pair of confused phoneme pairs based on the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; and fusing the acoustic features of the speech sample with the covariance matrix corresponding to each pair of confused phoneme pairs to obtain the acoustic features of each pair of confused phoneme pairs.

[0011] In one embodiment of the present invention, the above-mentioned method of obtaining the covariance matrix corresponding to each pair of confused phoneme pairs based on the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs includes: segmenting the speech sample to obtain the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; extracting the acoustic features of the multiple speech segments corresponding to the first phoneme and clustering them to obtain N first-class center vectors; extracting the acoustic features of the multiple speech segments corresponding to the second phoneme and clustering them to obtain N second-class center vectors; and performing dimensionality reduction on the N first-class center vectors and the N second-class center vectors to obtain a covariance matrix.

[0012] In one embodiment of the present invention, before performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs and the phoneme sequence, the training method of the above-mentioned pronunciation error detection model also includes: using a mask to replace some phonemes in the phoneme sequence; based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs and the phoneme sequence after mask replacement, performing speech recognition training on the pronunciation error detection model, wherein the pronunciation error detection model recognizes and outputs the phonemes corresponding to the replaced positions.

[0013] In one embodiment of the present invention, the training method of the above-mentioned pronunciation error detection model also includes: constructing at least one phoneme classification model corresponding to at least one pair of confused phoneme pairs, so that when the text sample and / or speech signal sample contains confused phonemes, pronunciation error detection is performed according to the output results of the pronunciation error detection model and the phoneme classification model, wherein each pair of confused phoneme pairs corresponds to a phoneme classification model, each pair of confused phoneme pairs includes a first phoneme and a second phoneme, and the phoneme classification model is used to output the probability that the confused phoneme belongs to the first phoneme or the second phoneme.

[0014] In one embodiment of the present invention, the above-mentioned construction of at least one phoneme classification model corresponding to at least one pair of confused phoneme pairs includes: segmenting the speech sample to obtain multiple speech segments corresponding to each pair of confused phoneme pairs in at least one pair of confused phoneme pairs; obtaining a vector of each speech segment in the multiple speech segments corresponding to the first phoneme, and obtaining a vector of each speech segment in the multiple speech segments corresponding to the second phoneme; training the phoneme classification models corresponding to the first phoneme and the second phoneme based on the vectors of the multiple speech segments corresponding to the first phoneme and the vectors of the multiple speech segments corresponding to the second phoneme.

[0015] According to a third aspect of an embodiment of the present invention, a pronunciation error detection device is provided, comprising: a first acquisition module for acquiring a speech signal to be detected and its corresponding read-aloud text; an extraction module for extracting the acoustic features of the speech signal to be detected and converting the read-aloud text into a phoneme sequence; a second acquisition module for acquiring the acoustic features of at least one pair of confused phoneme pairs; and an error detection module for performing pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence using a pronunciation error detection model.

[0016] According to a fourth aspect of an embodiment of the present invention, a training device for a pronunciation error detection model is provided, comprising: a first acquisition module for acquiring training samples, wherein the training samples include speech signal samples and their corresponding text samples, wherein the speech signal samples include speech information formed by a reader reading aloud the text samples; an extraction module for extracting acoustic features of the speech samples and converting the text samples into phoneme sequences; a second acquisition module for acquiring acoustic features of at least one pair of confused phoneme pairs; and a training module for performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal samples, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence.

[0017] According to a fifth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which computer-executable instructions are stored, characterized in that when the executable instructions are executed by a processor, any of the methods described above is implemented.

[0018] According to a sixth aspect of an embodiment of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for executing any of the above methods.

[0019] According to the technical solution provided by the embodiment of the present invention, by obtaining the speech signal to be detected and its corresponding read text; extracting the acoustic features of the speech signal to be detected, and converting the read text into a phoneme sequence; obtaining the acoustic features of at least one pair of confused phoneme pairs; and using a pronunciation error detection model to perform pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence, it is possible to eliminate the errors caused by the forced alignment technology in the pronunciation error detection method, and improve the distinguishability of easily confused phonemes, thereby improving the accuracy of pronunciation error detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0021] Figure 1 FIG2 is a flow chart of a pronunciation error detection method provided by an embodiment of the present invention.

[0022] Figure 2 FIG2 is a schematic diagram showing a training method of a pronunciation error detection model provided by an embodiment of the present invention.

[0023] Figure 3 FIG2 is a schematic diagram of a process for obtaining acoustic features of at least one pair of confused phoneme pairs according to an embodiment of the present invention.

[0024] Figure 4 FIG2 is a flow chart of a method for training a pronunciation error detection model according to an embodiment of the present invention.

[0025] Figure 5 FIG2 is a flow chart of a method for training a pronunciation error detection model provided by another embodiment of the present invention.

[0026] Figure 6 FIG. 1 is a schematic diagram of a speech recognition training method according to an embodiment of the present invention.

[0027] Figure 7 FIG2 is a schematic diagram of a training method for performing error detection training on a pronunciation error detection model based on acoustic features of a speech sample and a phoneme sequence after replacement of erroneous phonemes, provided by an embodiment of the present invention.

[0028] Figure 8 FIG2 is a schematic diagram of a training method for performing error detection training on a pronunciation error detection model based on acoustic features and phoneme sequences of speech samples labeled with pronunciation error levels, provided by an embodiment of the present invention.

[0029] Figure 9 FIG2 is a flow chart of constructing at least one phoneme classification model corresponding to at least one pair of confused phonemes provided by an embodiment of the present invention.

[0030] Figure 10 FIG2 is a block diagram of a pronunciation error detection device provided by an embodiment of the present invention.

[0031] Figure 11 FIG2 is a block diagram of a training device for a pronunciation error detection model provided by an embodiment of the present invention.

[0032] Figure 12 FIG2 is a block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0034] Most existing pronunciation error detection methods rely on forced alignment of speech segments. The input is a read-aloud text and speech, and forced alignment is used to obtain the speech segment corresponding to each phoneme. A representation vector for the speech segment is constructed, and the error detection result is predicted using this vector. However, when performing phoneme-level error detection in a continuous speech stream, the forced alignment technology requires high precision due to the short pronunciation time of phonemes. The errors introduced by the forced alignment technology will have a significant impact on the results of subsequent steps, resulting in a significant decrease in the accuracy of pronunciation error detection. In addition, the existing pronunciation error detection methods have a low accuracy rate for easily confused sounds.

[0035] In response to the above problems, the embodiments of the present invention provide a pronunciation error detection method and device, and a pronunciation error detection model training method and device. By training the pronunciation error detection model based on the overall acoustic features of the speech sample, the errors caused by the forced alignment technology can be eliminated; in addition, by constructing the acoustic features of at least one pair of confusing phoneme pairs, the pronunciation error detection model can improve its ability to distinguish easily confused sounds, thereby improving the accuracy of pronunciation error detection.

[0036] Figure 1 FIG. 1 is a flow chart of a pronunciation error detection method provided by an embodiment of the present invention. The method can be executed by a computer device (eg, a server). Figure 1 As shown, the method includes the following contents.

[0037] S110: Obtain the speech signal to be detected for error and its corresponding reading text.

[0038] The above-mentioned speech signals to be detected for errors may be speech signals generated by people of different genders, different ages, and different dialects reading aloud the contents of the text.

[0039] Specifically, embodiments of the present invention can be applied to spoken language error detection scenarios. Given a text to be read aloud, the tester's speech is checked for phoneme-level errors. For example, in the case of Chinese, the initial consonant, final vowel, and tone of each character are checked for correct pronunciation. In another example, in the case of English, the vowels, consonants, and stress of each word are checked for correct pronunciation.

[0040] S120: Extracting acoustic features of the speech signal to be detected, and converting the read text into a phoneme sequence.

[0041] Specifically, acoustic feature parameters can be extracted from the speech signal to be detected, for example, filter bank feature parameter extraction or MFCC feature parameter extraction, to obtain its corresponding acoustic features. It should be understood that the above description is only exemplary, and the present invention does not specifically limit the method for extracting the acoustic features of the speech signal to be detected.

[0042] Specifically, the reading text can be converted into a phoneme sequence according to the characteristics of different languages, and the start and end time of each phoneme and other information can be further marked. Phoneme is the smallest unit in speech. It can be analyzed based on the pronunciation action in the syllables of a word, and one action constitutes a phoneme. For example, in Chinese, there are 32 phonemes, and phonemes can be divided into initial consonants and finals. For example, for the Chinese reading text, its corresponding phoneme sequence can be composed of the initial consonants and finals of each word in sequence. Taking the Chinese reading text "pronunciation error detection" as an example, its corresponding phoneme sequence can be "f, a, y, in, j, ian, c, uo". It should be understood that its corresponding phoneme sequence can also be "f, a, y, i, n, j, i, a, n, c, u, o". The present invention does not specifically limit the form of the phoneme sequence.

[0043] It should be noted that the pronunciation error detection method provided in the embodiments of the present invention is applicable to different languages. For example, English has 48 phonemes, which can be divided into two categories: vowels and consonants. The phoneme sequence in the embodiments of the present invention can be a sequence composed of various phonetic symbols. It should be understood that the embodiments of the present invention are described using Chinese as an example, and the embodiments of the present invention do not specifically limit the language of the speech signal and text to be detected.

[0044] S130: Acquire acoustic features of at least one pair of confused phoneme pairs.

[0045] For example, in Chinese, some phonemes are pronounced similarly or are highly confused in some dialects, which can easily cause certain difficulties in pronunciation error detection, such as the front and back nasal sounds in / ing, en / eng, etc., the tip and middle sounds l / n, d / t, etc., and the flat and retroflex sounds z / zh, s / sh, c / ch, etc.

[0046] S140: Performing pronunciation error detection using a pronunciation error detection model based on acoustic features of the speech signal to be detected, acoustic features of at least one pair of confused phoneme pairs, and a phoneme sequence.

[0047] Specifically, the acoustic features of the speech signal to be detected, the acoustic features of at least one confusing phoneme pair, and the phoneme sequence can be input into the pronunciation detection model. Based on the acoustic features and phoneme sequence, the pronunciation detection model determines whether the pronunciation of the speech signal to be detected matches the text to be read aloud, that is, whether the pronunciation of the speech signal to be detected is correct.

[0048] In one embodiment of the present invention, Figure 2 As shown, the pronunciation error detection model can be an encoder-decoder model. The acoustic features of the speech signal to be detected and the acoustic features of at least one pair of confused phonemes are fused as the input of the encoder, and the phoneme sequence is used as the input of the decoder. The decoder of the pronunciation error detection model outputs an error detection result on whether each phoneme is pronounced incorrectly.

[0049] For example, the targeted acoustic features of all the above-mentioned confused phoneme pairs can be concatenated with the acoustic features of the speech signal to be detected as the acoustic features of the encoder input in the pronunciation error detection model. For example, if the acoustic features of the speech signal to be detected are 10-dimensional, the acoustic features for the in / ing confused phoneme pair are 10-dimensional; the acoustic features for the en / eng confused phoneme pair are 10-dimensional; the acoustic features for the l / n confused phoneme pair are 10-dimensional, etc., the targeted acoustic features of 12 pairs of confused phoneme pairs can be concatenated with the acoustic features of the speech signal to be detected to obtain 130-dimensional acoustic features as the input features of the encoder in the pronunciation error detection model.

[0050] According to the technical solution provided by the embodiment of the present invention, by obtaining the speech signal to be detected and its corresponding read text; extracting the acoustic features of the speech signal to be detected, and converting the read text into a phoneme sequence; obtaining the acoustic features of at least one pair of confused phoneme pairs; and using a pronunciation error detection model to perform pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence, it is possible to eliminate the errors caused by the forced alignment technology in the pronunciation error detection method, and improve the distinguishability of easily confused phonemes, thereby improving the accuracy of pronunciation error detection.

[0051] In another embodiment of the present invention, in addition to converting the read-aloud text into a phoneme sequence, the tone sequence of the read-aloud text can also be obtained. That is, the above step S120 may include: converting the read-aloud text into a phoneme sequence and a tone sequence. Taking the Chinese read-aloud text "pronunciation error detection" as an example, its corresponding tone sequence is "1, 1, 3, 4". It should be noted that in order to make the tone sequence and the phoneme sequence "f, a, y, in, j, ian, c, uo" consistent in length, a null symbol can be added before each element of the tone sequence, that is, it becomes "null, 1, null, 1, null, 3, null, 4".

[0052] In this embodiment, the above step S140 may specifically include: using a pronunciation error detection model to perform pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence and tone sequence.

[0053] Specifically, the acoustic features of the speech signal to be detected and the acoustic features of at least one pair of confused phoneme pairs are fused as the input of the encoder, the phoneme sequence and tone sequence are used as the input of the decoder, and the decoder of the pronunciation error detection model outputs the error detection result on whether each phoneme is pronounced incorrectly.

[0054] According to the technical solution provided by the embodiment of the present invention, by utilizing a pronunciation error detection model, error detection training is performed based on the acoustic features of a speech sample, the acoustic features of at least one pair of confused phoneme pairs, and a phoneme sequence and a tone sequence, so that tone error detection can be performed, thereby further improving the accuracy of pronunciation error detection.

[0055] In order to improve the discrimination of the pronunciation error detection model for easily confused phonemes, in an embodiment of the present invention, a set of acoustic features can be constructed for each pair of confused phonemes mentioned above. Specifically, Figure 3 As shown, the above step S130 may include:

[0056] S131: Acquire acoustic features of a speech sample, wherein the speech sample includes a plurality of speech segments corresponding to each of at least one pair of confused phoneme pairs, wherein each pair of confused phoneme pairs includes a first phoneme and a second phoneme.

[0057] Specifically, large-scale speech recognition data can be used as speech samples for pronunciation error detection tasks.

[0058] S132: Obtain a covariance matrix corresponding to each pair of confused phonemes according to the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phonemes.

[0059] Specifically, the speech sample can be segmented to obtain multiple speech segments corresponding to the first phoneme and multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; the acoustic features of the multiple speech segments corresponding to the first phoneme are extracted and clustered to obtain N first-class center vectors; the acoustic features of the multiple speech segments corresponding to the second phoneme are extracted and clustered to obtain N second-class center vectors; the N first-class center vectors and the N second-class center vectors are reduced in dimensionality to obtain a covariance matrix.

[0060] The embodiment of the present invention takes the confusing phoneme pair of in / ing as an example.

[0061] Specifically, large-scale speech recognition data can be used as speech samples for pronunciation error detection tasks, and forced alignment technology can be used to obtain speech segments corresponding to all the first phonemes (in) and second phonemes (ing) in the language samples.

[0062] First, we can extract the filter bank features for all speech segments corresponding to in, and perform kmeans clustering on the filter bank features to obtain N cluster center vectors

[0063] Similarly, filter bank features are extracted for all speech segments corresponding to ing, and kmeans clustering is performed on the filter bank features to obtain N cluster center vectors

[0064] Secondly, you can and A total of 2N class center vectors are reduced in dimension using the principal component analysis (PCA) method to obtain their covariance matrix W.

[0065] It should be understood that other clustering methods can also be used to cluster the filter bank features, and the present invention does not specifically limit the clustering method. In addition, other methods can also be used to reduce the dimension of the above 2N class center vectors, and the present invention does not specifically limit the dimensionality reduction method.

[0066] S133: Fusing the acoustic features of the speech sample with the covariance matrix corresponding to each pair of confused phonemes to obtain the acoustic features of each pair of confused phonemes.

[0067] Specifically, the acoustic features of the speech sample may be multiplied by the covariance matrix corresponding to each pair of confused phonemes to obtain the acoustic features of each pair of confused phonemes.

[0068] For example, filter bank features of all speech samples are extracted, and the filter bank features of all speech samples are multiplied by the above covariance matrix W, thereby obtaining acoustic features for the in / ing confused phoneme pair.

[0069] It should be noted that, according to the above method, acoustic features for each pair of confusing phonemes such as en / eng, l / n, d / t, z / zh, s / sh, and c / ch can be constructed in sequence.

[0070] It should be noted that the acoustic features of the above-mentioned at least one pair of confused phoneme pairs can be constructed in advance (for example, the acoustic features of at least one pair of confused phoneme pairs used when training the pronunciation error detection model). During the pronunciation error detection process, they can be directly spliced with the acoustic features of the speech signal to be detected and used as the input of the encoder.

[0071] In addition, it should be understood that the above description is merely exemplary, and the present invention does not specifically limit the method for obtaining the acoustic features of at least one pair of confused phoneme pairs.

[0072] In one embodiment of the present invention, the above-mentioned pronunciation error detection method also includes: determining whether the read text contains confusing phonemes; when the read text contains confusing phonemes, performing pronunciation error detection based on the output results of the pronunciation error detection model and the phoneme classification model corresponding to the confusing phonemes.

[0073] It should be noted that the phoneme classification model can be trained by the following method: segmenting the speech sample to obtain multiple speech segments corresponding to at least one pair of confused phonemes, wherein each type of confused phoneme pair includes a first phoneme and a second phoneme; obtaining the vector of each speech segment in the multiple speech segments corresponding to the first phoneme, and obtaining the vector of each speech segment in the multiple speech segments corresponding to the second phoneme; training the phoneme classification model corresponding to the first phoneme and the second phoneme based on the vectors of the multiple speech segments corresponding to the first phoneme and the vectors of the multiple speech segments corresponding to the second phoneme, wherein the phoneme classification model is used to output the probability that the confused phoneme belongs to the first phoneme or the second phoneme.

[0074] In the embodiment of the present invention, the training of the phoneme classification model is described by taking the in / ing confused phoneme pair as an example.

[0075] Specifically, the speech samples can first be forced to align to obtain all isolated speech segments corresponding to the in and ing phonemes.

[0076] For each speech segment corresponding to "in" or "ing", the filter bank features of the speech segment can be extracted and forward-processed through the encoder of the trained pronunciation error detection model to obtain a set of output vectors. The set of output vectors is averaged to obtain the vector representation corresponding to the speech segment.

[0077] The vector representation corresponding to all the in speech segments is recorded as The vector representation corresponding to all ing speech segments is recorded as where N in and N ing Respectively represent the number of times in and ing appear in the corpus. Then according to and Train a phoneme classification model.

[0078] In one embodiment of the present invention, and The phoneme binary classification model is trained by the support vector machine (SVM) algorithm, denoted as SVM in / ing Repeat the above steps for each type of confused phoneme pair to obtain multiple SVM classification models, such as SVM z / zh , SVM c / ch , SVM s / shIt should be understood that the above-mentioned phoneme classification model can also be a neural network model such as CNN, and the present invention does not specifically limit the specific method of training the phoneme classification model.

[0079] In an embodiment of the present invention, when the text read aloud does not contain confusing phonemes, error detection can be performed directly based on the output result of the decoder of the trained pronunciation error detection model.

[0080] When the read text contains confusing phonemes, the results of the pronunciation error detection model and the phoneme classification model can be fused, that is, pronunciation error detection is performed based on the output results of the pronunciation error detection model and the phoneme classification model.

[0081] Taking the phoneme in as an example, we first obtain the probability p of misreading the position based on the output of the decoder of the trained pronunciation error detection model. ed-error , and then find the corresponding SVM model SVM in the phoneme classification model in / ing , from which we can obtain the probability that the SVM prediction result is ing, that is, the probability p that the SVM model believes that the position is read incorrectly svm-error , the final prediction result can be the average of the output results of the two models, that is, p error =(p ed-error +p svm-error ) / 2.

[0082] It should be understood that the above fusion of the results of the pronunciation error detection model and the phoneme classification model, in addition to finding the average of the output results of the two models, can also be the weighted addition of the output results of the two models, and the present invention does not make specific limitations on this.

[0083] According to the technical solution provided by the embodiment of the present invention, when the read-aloud text contains confusing phonemes, the phoneme classification model is used to classify the confused phonemes in the isolated speech segments obtained by forced alignment, and the output results of the pronunciation error detection model and the phoneme classification model are integrated to perform pronunciation error detection, which can improve the error detection effect of the pronunciation error detection model on confused phonemes; in addition, when the read-aloud text does not contain confusing phonemes, pronunciation error detection is performed according to the output results of the pronunciation error detection model. The technical solution provided by the embodiment of the present invention can ensure that it has a good error detection effect on both confused phonemes and non-confused phonemes.

[0084] Figure 4 FIG. 1 is a flow chart of a method for training a pronunciation error detection model according to an embodiment of the present invention. The method may be executed by a computer device (eg, a server). Figure 4 As shown, the method includes the following contents.

[0085] S410: Acquire training samples, wherein the training samples include speech signal samples and their corresponding text samples, wherein the speech signal samples include speech information generated when a reader reads aloud the text samples.

[0086] Specifically, the voice signal sample can be a voice recognition corpus formed by people of different genders, ages, and dialects reading the same or different text contents. Among them, the text sample can be understood as an annotation of the voice signal sample. Each voice signal sample can correspond to a text sample, and a text sample can correspond to multiple voice signal samples.

[0087] S420: Extracting acoustic features of the speech signal sample and converting the text sample into a phoneme sequence.

[0088] Specifically, acoustic feature parameters can be extracted from the speech signal sample, for example, filter bank feature parameter extraction or MFCC feature parameter extraction, to obtain the corresponding acoustic features. It should be understood that the above description is only exemplary, and the present invention does not specifically limit the method for extracting the acoustic features of the speech signal sample.

[0089] Specifically, the text sample can be converted into a phoneme sequence according to the characteristics of different languages, and the start and end time of each phoneme and other information can be marked. Phoneme is the smallest unit in speech. It can be analyzed based on the pronunciation action in the syllable of the word, and one action constitutes a phoneme. For example, in Chinese, there are 32 phonemes, and phonemes can be divided into initial consonants and finals. For example, for a Chinese text sample, its corresponding phoneme sequence can be composed of the initial consonants and finals of each word in sequence. Taking the Chinese text sample "pronunciation error detection" as an example, its corresponding phoneme sequence can be "f, a, y, in, j, ian, c, uo". It should be understood that its corresponding phoneme sequence can also be "f, a, y, i, n, j, i, a, n, c, u, o". The present invention does not specifically limit the form of the phoneme sequence.

[0090] It should be noted that the training method for the pronunciation error detection model provided in the embodiments of the present invention is applicable to different languages. For example, English has 48 phonemes, which can be divided into two major categories: vowels and consonants. The phoneme sequence in the embodiments of the present invention can be a sequence composed of various phonetic symbols. It should be understood that the embodiments of the present invention are described using the Chinese language as an example, and the embodiments of the present invention do not specifically limit the language of the speech signal samples and text samples.

[0091] S430: Acquire acoustic features of at least one pair of confused phoneme pairs.

[0092] For example, in Chinese, some phonemes are pronounced similarly or are highly confused in some dialects, which can easily cause certain difficulties in pronunciation error detection, such as the front and back nasal sounds in / ing, en / eng, etc., the tip and middle sounds l / n, d / t, etc., and the flat and retroflex sounds z / zh, s / sh, c / ch, etc.

[0093] S440: Perform error detection training on a pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence.

[0094] Specifically, the acoustic features of a speech signal sample, the acoustic features of at least one confused phoneme pair, and the phoneme sequence can be used as inputs to train the pronunciation error detection model. Based on the acoustic features of the speech signal sample, the acoustic features of at least one confused phoneme pair, and the phoneme sequence corresponding to the text sample, the pronunciation error detection model determines whether the pronunciation of the speech signal sample matches the text sample, that is, whether the pronunciation of the speech signal sample is correct.

[0095] In one embodiment of the present invention, the pronunciation error detection model may be an encoder-decoder model. The encoder inputs a mixture of acoustic features of a speech signal sample and acoustic features of at least one confused phoneme pair, and the phoneme sequence serves as input to the decoder. The decoder of the pronunciation error detection model outputs an error detection result indicating whether each phoneme is mispronounced.

[0096] For example, the targeted acoustic features of all the aforementioned confused phoneme pairs can be concatenated with the acoustic features of the speech signal sample to serve as the acoustic features input to the encoder in the pronunciation error detection model. For example, if the acoustic features of the speech signal sample are 10-dimensional, the acoustic features for the confused in / ing phoneme pair are 10-dimensional; the acoustic features for the confused en / eng phoneme pair are 10-dimensional; the acoustic features for the confused l / n phoneme pair are 10-dimensional, and so on. The targeted acoustic features of 12 pairs of confused phoneme pairs can be concatenated with the acoustic features of the speech signal sample to obtain 130-dimensional acoustic features, which serve as the input features of the encoder in the pronunciation error detection model. The phoneme sequence is then used as the input feature of the decoder to train the pronunciation error detection model.

[0097] It should be noted that the decoder in the embodiment of the present invention adopts a non-autoregressive bidirectional structure to enhance the modeling capability. For ease of description, the embodiment of the present invention and subsequent embodiments are described using a pronunciation error detection model as the encoder-decoder model. It should be understood that the present invention is not specifically limited to this.

[0098] According to the technical solution provided by the embodiment of the present invention, by utilizing the overall acoustic features of the speech signal sample to perform error detection training on the pronunciation error detection model, the errors caused by the forced alignment technology can be eliminated; in addition, by constructing acoustic features for at least one pair of confused phoneme pairs; based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs and the phoneme sequence, the pronunciation error detection model is trained for error detection, which can improve the discrimination of the pronunciation error detection model for easily confused sounds, thereby improving the accuracy of the pronunciation error detection model for easily confused phonemes.

[0099] In another embodiment of the present invention, in addition to converting the text sample into a phoneme sequence, the tone sequence of the text sample can also be obtained. That is, the above step S420 may include: converting the text sample into a phoneme sequence and a tone sequence. Taking the Chinese text sample "pronunciation error detection" as an example, its corresponding tone sequence is "1, 1, 3, 4". It should be noted that in order to make the tone sequence and the phoneme sequence "f, a, y, in, j, ian, c, uo" have the same length, a null symbol can be added before each element of the tone sequence, that is, it becomes "null, 1, null, 1, null, 3, null, 4".

[0100] In this embodiment, the above step S430 may specifically include: performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence and tone sequence.

[0101] Specifically, the acoustic features of the speech signal sample and the acoustic features of at least one pair of confused phoneme pairs are fused as the input of the encoder, the phoneme sequence and tone sequence are used as the input of the decoder, and the decoder of the pronunciation error detection model outputs the error detection result on whether each phoneme is pronounced incorrectly.

[0102] According to the technical solution provided in the embodiment of the present invention, by performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence and tone sequence, tone error detection can be performed, thereby further improving the accuracy of pronunciation error detection.

[0103] In order to improve the discrimination of easily confused phonemes by the pronunciation error detection model, in an embodiment of the present invention, a set of acoustic features may be constructed for each pair of confused phonemes. Specifically, the above step S430 may include:

[0104] S4310: Acquire acoustic features of a speech sample, wherein the speech sample includes a plurality of speech segments corresponding to each of at least one pair of confused phoneme pairs, wherein each pair of confused phoneme pairs includes a first phoneme and a second phoneme.

[0105] S4320: Obtain a covariance matrix corresponding to each pair of confused phonemes according to the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phonemes.

[0106] Specifically, first, the speech sample is segmented to obtain a plurality of speech segments corresponding to the first phoneme and a plurality of speech segments corresponding to the second phoneme in each pair of confused phonemes.

[0107] Secondly, the acoustic features of multiple speech segments corresponding to the first phoneme are extracted and clustered to obtain N first-category center vectors; and the acoustic features of multiple speech segments corresponding to the second phoneme are extracted and clustered to obtain N second-category center vectors.

[0108] Then, the N first-category center vectors and the N second-category center vectors are reduced in dimension to obtain the covariance matrix.

[0109] S4330: Fusing the acoustic features of the speech sample with the covariance matrix corresponding to each pair of confused phonemes to obtain the acoustic features of each pair of confused phonemes.

[0110] The embodiment of the present invention takes the confusing phoneme pair of in / ing as an example.

[0111] Specifically, large-scale speech recognition data can be used as a training sample set for the pronunciation error detection task, and forced alignment technology can be used to obtain all speech segments corresponding to the first phoneme (in) and the second phoneme (ing) in the training sample set.

[0112] First, we can extract the filter bank features for all speech segments corresponding to in, and perform kmeans clustering on the filter bank features to obtain N cluster center vectors

[0113] Similarly, filter bank features are extracted for all speech segments corresponding to ing, and kmeans clustering is performed on the filter bank features to obtain N cluster center vectors

[0114] Secondly, you can and A total of 2N class center vectors are reduced in dimension using the principal component analysis (PCA) method to obtain their covariance matrix W.

[0115] Then, the filter bank features of all speech samples in the training sample set can be extracted, and the filter bank features of all speech samples are multiplied by the above covariance matrix W to obtain the acoustic features for the in / ing confused phoneme pairs.

[0116] It should be noted that, according to the above method, acoustic features for each pair of confusing phonemes such as en / eng, l / n, d / t, z / zh, s / sh, and c / ch can be constructed in sequence.

[0117] It should be noted that the acoustic features of the above-mentioned at least one pair of confused phoneme pairs can be constructed before training the pronunciation error detection model. When training the pronunciation error detection model, they can be directly spliced with the acoustic features of the speech sample and used as the input of the encoder.

[0118] In addition, it should be understood that the above description is merely exemplary, and the present invention does not specifically limit the method for obtaining the acoustic features of at least one pair of confused phoneme pairs.

[0119] Figure 5 FIG. 1 is a flow chart of a method for training a pronunciation error detection model according to another embodiment of the present invention. Figure 4 The present invention is extended based on the embodiment shown Figure 5 The embodiment shown is described below in detail. Figure 5 The embodiment shown and Figure 4 The differences and similarities between the illustrated embodiments are not described in detail.

[0120] like Figure 5 As shown, in the method for training a pronunciation error detection model provided by an embodiment of the present invention, before performing error detection training on the pronunciation error detection model based on the acoustic features of a speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence, the method further includes:

[0121] Step S450: using a mask to replace some phonemes in a phoneme sequence; performing speech recognition training on a pronunciation error detection model based on the acoustic features of a speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence after mask replacement, wherein the pronunciation error detection model recognizes and outputs the phonemes corresponding to the replaced positions.

[0122] That is to say, the pronunciation error detection model can be trained for speech recognition first, and then the pronunciation error detection model can be trained for error detection.

[0123] Specifically, large-scale speech recognition data can be used as speech signal samples for training the pronunciation error detection model in an embodiment of the present invention, and the text samples of the large-scale speech recognition data can be converted into phoneme sequences and tone sequences. Some phonemes and tones are randomly deleted at the input end of the decoder, and the deleted parts can be replaced by masks. For the replaced positions, the corresponding phonemes and tones are predicted at the output end of the decoder, and for the positions that are not replaced, no prediction is made at the output end. Then, the loss value is calculated based on the prediction result and the labeled data (that is, the real phonemes and tones corresponding to the replaced positions), and the parameters of the pronunciation error detection model are updated by backpropagating the loss value until the loss value converges.

[0124] Take the word “error detection” as an example. Figure 6 As shown, the targeted acoustic features of the 12 pairs of confused phoneme pairs can be concatenated with the acoustic features of the speech signal sample with "error detection" speech information to serve as the acoustic features of the encoder input in the pronunciation error detection model. At the decoder input, the phonemes "j" and "ian" in the phoneme sequence and the tone "3" in the tone sequence are deleted, and the deleted positions are replaced with a mask [mask]. Based on the input acoustic features, the decoder can identify the phonemes "j" and "ian" and the tone "3" corresponding to the positions replaced by the mask [mask].

[0125] According to the technical solution provided in an embodiment of the present invention, by using a mask to replace some phonemes in a phoneme sequence; based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs and the phoneme sequence after mask replacement, the pronunciation error detection model is trained for speech recognition, which can improve the accuracy of pronunciation error detection of subsequent pronunciation error detection models.

[0126] In one embodiment of the present invention, the above-mentioned step S440 may include: replacing some phonemes in the phoneme sequence with incorrect phonemes; performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs and the phoneme sequence after the incorrect phonemes are replaced, wherein the pronunciation error detection model outputs whether the position of each phoneme is replaced.

[0127] Specifically, large-scale speech recognition data can be used as speech signal samples for training the pronunciation error detection model in the embodiments of the present invention. The text samples of the large-scale speech recognition data are converted into phoneme sequences and tone sequences. At the input of the decoder, some phonemes or tones in the phoneme sequence are randomly replaced with incorrect phonemes, so that the output of the decoder predicts whether each position has been replaced. For example, 0 can indicate no replacement, meaning the position is pronounced correctly, while 1 can indicate replacement, meaning the position is pronounced incorrectly.

[0128] Take the word “error detection” as an example. Figure 7As shown, the phoneme "c" is randomly replaced by the phoneme "d" at the input of the decoder. The pronunciation error detection model uses the phoneme sequence "j, ian, d, uo" as a benchmark and determines whether each position in the speech signal sample is read correctly based on the input acoustic features. For example, Figure 7 The output of the decoder predicts that the position of the phoneme "d" is read incorrectly, while the rest of the positions are read correctly.

[0129] According to the technical solution provided by the embodiment of the present invention, by replacing some phonemes in a phoneme sequence with erroneous phonemes; performing error detection training on a pronunciation error detection model based on the acoustic features of a speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence after replacement with erroneous phonemes, it is possible to provide pre-training for the pronunciation error detection model, solve more obvious misreading problems, and improve the error detection effect of the pronunciation error detection model.

[0130] In another embodiment of the present invention, the above-mentioned step S440 may include: performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal samples marked with pronunciation error levels, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence, wherein the pronunciation error detection model outputs the pronunciation error level classification of each phoneme position.

[0131] Specifically, the speech signal samples can be finely labeled manually. For example, according to the pronunciation of each phoneme and tone, they can be labeled as "correct", "wrong", or "defective". Among them, "defect" can be a situation where the error level is relatively minor, between "correct" and "wrong". Figure 8 As shown, the pronunciation error detection model is fine-tuned and trained based on the labeled data. The pronunciation error detection model can classify each phoneme and tone position into three categories: "correct," "incorrect," and "defective." For example, 0 represents correct, 1 represents incorrect, and 2 represents defective. It should be understood that the above "correct," "incorrect," and "defective" are merely exemplary descriptions, and the present invention does not specifically limit the pronunciation error classification level.

[0132] According to the technical solution provided by the embodiment of the present invention, by labeling the pronunciation error level of the speech signal samples, based on the acoustic features of the speech signal samples labeled with the pronunciation error level, the acoustic features of at least one pair of confused phoneme pairs and the phoneme sequence, the pronunciation error detection model is trained to detect the pronunciation error, so that the pronunciation error detection model outputs the pronunciation error level classification of each phoneme position, and the pronunciation error detection model can be fine-tuned and trained. Since the existing solutions are usually trained based on general large-scale speech recognition data, when the usage scenario is a specific scenario (such as a dialect scenario), the effect may decrease. If the effect of the specific scenario needs to be improved, it is necessary to collect large-scale speech recognition corpus for the corresponding scenario, which is costly. The embodiment of the present invention uses the labeled speech signal samples for fine-tuning training. For the purpose of improving the effect of the specific scenario, only a relatively small number of training samples need to be labeled, which is costly.

[0133] It should be noted that, in the case where the above-mentioned speech recognition training is first performed on the pronunciation error detection model, and then the pronunciation error detection training is performed on the pronunciation error detection model, the speech recognition module ( Figure 6 The left slash block in the figure) is replaced with the pronunciation error detection module ( Figure 7 or Figure 8 to facilitate pronunciation error detection training.

[0134] In another embodiment of the present invention, the above-mentioned step S440 includes: replacing some phonemes in a phoneme sequence with erroneous phonemes; performing error detection training on a pronunciation error detection model based on the acoustic features of a speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence after the erroneous phoneme replacement, wherein the pronunciation error detection model outputs whether the position of each phoneme is replaced; and performing error detection training on a pronunciation error detection model based on the acoustic features of a speech signal sample marked with a pronunciation error level, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence, wherein the pronunciation error detection model outputs a pronunciation error level classification for each phoneme position. According to the technical solution provided in an embodiment of the present invention, training the pronunciation error detection model by the above-mentioned two error detection training methods can enable the pronunciation error detection model to obtain a better error detection effect.

[0135] In one embodiment of the present invention, the training method of the above-mentioned pronunciation error detection model also includes: constructing at least one phoneme classification model corresponding to at least one pair of confused phoneme pairs, so that when the text sample and / or speech signal sample contains confused phonemes, pronunciation error detection is performed according to the output results of the pronunciation error detection model and the phoneme classification model, wherein each pair of confused phoneme pairs corresponds to a phoneme classification model, each pair of confused phoneme pairs includes a first phoneme and a second phoneme, and the phoneme classification model is used to output the probability that the confused phoneme belongs to the first phoneme or the second phoneme.

[0136] Specifically, if Figure 9 As shown, the above-mentioned construction of at least one phoneme classification model corresponding to at least one pair of confused phoneme pairs may include:

[0137] S910: Segment the speech sample to obtain multiple speech segments corresponding to each of at least one pair of confused phoneme pairs.

[0138] Specifically, taking the in / ing confused phoneme pair as an example, the speech samples can be forced to align to obtain all isolated speech segments corresponding to the in and ing phonemes.

[0139] S920: Obtain a vector of each speech segment in the plurality of speech segments corresponding to the first phoneme, and obtain a vector of each speech segment in the plurality of speech segments corresponding to the second phoneme.

[0140] In one embodiment of the present invention, for each speech segment corresponding to "in" or "ing", the filter bank features of the speech segment can be extracted, and a forward operation is performed on the encoder of the trained pronunciation error detection model to obtain a set of output vectors, which are then averaged to obtain the vector representation corresponding to the speech segment.

[0141] It should be understood that the vector of each speech segment may also be obtained in other ways, and the present invention does not impose any specific limitation on this.

[0142] S930: Training a phoneme classification model corresponding to the first phoneme and the second phoneme according to the vectors of the plurality of speech segments corresponding to the first phoneme and the vectors of the plurality of speech segments corresponding to the second phoneme.

[0143] For example, the vector representation corresponding to all the speech segments of in in step S920 is recorded as The vector representation corresponding to all ing speech segments is recorded as where N in and N ing Respectively represent the number of times in and ing appear in the corpus. Then according to and Train a phoneme classification model.

[0144] In one embodiment of the present invention, and The phoneme binary classification model is trained by the support vector machine (SVM) algorithm, denoted as SVM in / ing Repeat the above steps for each type of confused phoneme pair to obtain multiple SVM classification models, such as SVM z / zh , SVM c / ch , SVM s / shIt should be understood that the above-mentioned phoneme classification model can also be a neural network model such as CNN, and the present invention does not specifically limit the specific method of training the phoneme classification model.

[0145] In an embodiment of the present invention, when the text sample and / or the speech signal sample does not contain confusing phonemes, error detection can be performed directly based on the output result of the decoder of the trained pronunciation error detection model.

[0146] When the text sample and / or speech signal sample contains confusing phonemes, the results of the pronunciation error detection model and the phoneme classification model can be fused, that is, pronunciation error detection is performed based on the output results of the pronunciation error detection model and the phoneme classification model.

[0147] Taking the phoneme in as an example, we first obtain the probability p of misreading the position based on the output of the decoder of the trained pronunciation error detection model. ed-error , and then find the corresponding SVM model SVM in the phoneme classification model in / ing , from which we can obtain the probability that the SVM prediction result is ing, that is, the probability p that the SVM model believes that the position is read incorrectly svm-error , the final prediction result can be the average of the output results of the two models, that is, p error =(p ed-error +p svm-error ) / 2.

[0148] It should be understood that the above fusion of the results of the pronunciation error detection model and the phoneme classification model, in addition to finding the average of the output results of the two models, can also be the weighted addition of the output results of the two models, and the present invention does not make specific limitations on this.

[0149] According to the technical solution provided by the embodiments of the present invention, by training a phoneme classification model, when text samples and / or speech signal samples contain confused phonemes, the phoneme classification model is used to classify the confused phonemes in the isolated speech segments obtained by forced alignment, and the output results of the pronunciation error detection model and the phoneme classification model are fused to perform pronunciation error detection, which can improve the error detection effect of the pronunciation error detection model on confused phonemes.

[0150] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present invention, and will not be described in detail here.

[0151] The following are embodiments of the apparatus of the present invention, which can be used to implement the method embodiments of the present invention. For details not disclosed in the apparatus embodiments of the present invention, please refer to the method embodiments of the present invention.

[0152] Figure 10 FIG. 1 is a block diagram of a pronunciation error detection device provided by an embodiment of the present invention. Figure 10 As shown, the pronunciation error detection device 1000 includes:

[0153] The first acquisition module 1010 is used to obtain the speech signal to be detected and its corresponding reading text;

[0154] Extraction module 1020, for extracting acoustic features of the speech signal to be detected and converting the read text into a phoneme sequence;

[0155] A second acquisition module 1030 is configured to acquire acoustic features of at least one confused phoneme pair;

[0156] The error detection module 1040 is configured to utilize a pronunciation error detection model to perform pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence.

[0157] According to the technical solution provided by the embodiment of the present invention, by obtaining the speech signal to be detected and its corresponding read text; extracting the acoustic features of the speech signal to be detected, and converting the read text into a phoneme sequence; obtaining the acoustic features of at least one pair of confused phoneme pairs; and using a pronunciation error detection model to perform pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence, it is possible to eliminate the errors caused by the forced alignment technology in the pronunciation error detection method, and improve the distinguishability of easily confused phonemes, thereby improving the accuracy of pronunciation error detection.

[0158] In one embodiment of the present invention, the above-mentioned second acquisition module 1030 is used to obtain the acoustic features of the speech sample, wherein the speech sample includes multiple speech segments corresponding to each pair of confused phoneme pairs in at least one pair of confused phoneme pairs, wherein each pair of confused phoneme pairs includes a first phoneme and a second phoneme; based on the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs, a covariance matrix corresponding to each pair of confused phoneme pairs is obtained; the acoustic features of the speech sample are respectively fused with the covariance matrix corresponding to each pair of confused phoneme pairs to obtain the acoustic features of each pair of confused phoneme pairs.

[0159] In one embodiment of the present invention, the above-mentioned second acquisition module 1030 is used to segment the speech sample, obtain multiple speech segments corresponding to the first phoneme and multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; extract the acoustic features of the multiple speech segments corresponding to the first phoneme respectively and cluster them to obtain N first-class center vectors; extract the acoustic features of the multiple speech segments corresponding to the second phoneme respectively and cluster them to obtain N second-class center vectors; reduce the dimension of the N first-class center vectors and the N second-class center vectors to obtain a covariance matrix.

[0160] In one embodiment of the present invention, the above-mentioned pronunciation error detection device also includes a judgment module 1050, which is used to determine whether the read text contains confusing phonemes; when the read text contains confusing phonemes, the above-mentioned error detection module is used to perform pronunciation error detection based on the output results of the pronunciation error detection model and the phoneme classification model corresponding to the confusing phonemes.

[0161] The implementation process of the functions and effects of each module in the pronunciation error detection device 1000 is detailed in Figure 1 The implementation process of the corresponding steps in the embodiment will not be repeated here.

[0162] Figure 11 FIG. 1 is a block diagram of a training device for a pronunciation error detection model provided by an embodiment of the present invention. Figure 11 As shown, the training device 1100 of the pronunciation error detection model includes:

[0163] A first acquisition module 1110 is configured to acquire a training sample, wherein the training sample includes a speech signal sample and its corresponding text sample, wherein the speech signal sample includes speech information generated by a reader reading a text sample aloud;

[0164] Extraction module 1120, for extracting acoustic features of speech signal samples and converting text samples into phoneme sequences;

[0165] A second acquisition module 1130 is configured to acquire acoustic features of at least one confused phoneme pair;

[0166] The training module 1140 is configured to perform error detection training on the pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence.

[0167] According to the technical solution provided by the embodiment of the present invention, by utilizing the overall acoustic features of the speech signal sample to perform error detection training on the pronunciation error detection model, the errors caused by the forced alignment technology can be eliminated; in addition, by constructing acoustic features for at least one pair of confused phoneme pairs; based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs and the phoneme sequence, the pronunciation error detection model is trained for error detection, which can improve the discrimination of the pronunciation error detection model for easily confused sounds, thereby improving the accuracy of the pronunciation error detection model for easily confused phonemes.

[0168] In one embodiment of the present invention, the above-mentioned second acquisition module 1130 is used to obtain the acoustic features of a speech sample, wherein the speech sample includes multiple speech segments corresponding to each pair of confused phoneme pairs in at least one pair of confused phoneme pairs, wherein each pair of confused phoneme pairs includes a first phoneme and a second phoneme; based on the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs, a covariance matrix corresponding to each pair of confused phoneme pairs is obtained; the acoustic features of the speech sample are respectively fused with the covariance matrix corresponding to each pair of confused phoneme pairs to obtain the acoustic features of each pair of confused phoneme pairs.

[0169] In one embodiment of the present invention, the above-mentioned second acquisition module 1130 is used to segment the speech sample, obtain multiple speech segments corresponding to the first phoneme and multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; extract the acoustic features of the multiple speech segments corresponding to the first phoneme respectively and cluster them to obtain N first-class center vectors; extract the acoustic features of the multiple speech segments corresponding to the second phoneme respectively and cluster them to obtain N second-class center vectors; perform dimensionality reduction on the N first-class center vectors and the N second-class center vectors to obtain a covariance matrix.

[0170] In one embodiment of the present invention, before performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of at least one pair of confused phoneme pairs, and the phoneme sequence, the above-mentioned training module 1140 is also used to replace part of the phonemes in the phoneme sequence with a mask; based on the acoustic features of the speech signal sample and the phoneme sequence after mask replacement, the pronunciation error detection model is trained for speech recognition, wherein the pronunciation error detection model recognizes and outputs the phonemes corresponding to the replaced positions.

[0171] In one embodiment of the present invention, the training device of the above-mentioned pronunciation error detection model also includes a classification module 1150, which is used to construct at least one phoneme classification model corresponding to at least one pair of confused phoneme pairs, so that when the text sample and / or speech signal sample contains confused phonemes, pronunciation error detection is performed according to the output results of the pronunciation error detection model and the phoneme classification model, wherein each pair of confused phoneme pairs corresponds to a phoneme classification model, each pair of confused phoneme pairs includes a first phoneme and a second phoneme, and the phoneme classification model is used to output the probability that the confused phoneme belongs to the first phoneme or the second phoneme.

[0172] In one embodiment of the present invention, the above-mentioned classification module 1150 is used to segment the speech sample, obtain multiple speech segments corresponding to each pair of confused phonemes in at least one pair of confused phonemes; obtain the vector of each speech segment in the multiple speech segments corresponding to the first phoneme, and obtain the vector of each speech segment in the multiple speech segments corresponding to the second phoneme; train the phoneme classification model corresponding to the first phoneme and the second phoneme based on the vectors of the multiple speech segments corresponding to the first phoneme and the vectors of the multiple speech segments corresponding to the second phoneme.

[0173] In one embodiment of the present invention, the above-mentioned pronunciation error detection model includes an encoder-decoder model.

[0174] The implementation process of the functions and effects of each module in the device 1100 is detailed in Figures 4 to 9 The implementation process of the corresponding steps in the embodiment will not be repeated here.

[0175] Figure 12 FIG. 1 is a block diagram of an electronic device 1200 according to an embodiment of the present invention.

[0176] Reference Figure 12 The electronic device 1200 includes a processing component 1210, which further includes one or more processors, and a memory resource represented by a memory 1220 for storing instructions executable by the processing component 1210, such as an application. The application stored in the memory 1220 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1210 is configured to execute the instructions to perform the above-mentioned pronunciation error detection method or the pronunciation error detection model training method.

[0177] The electronic device 1200 may further include a power supply component configured to perform power management of the electronic device 1200, a wired or wireless network interface configured to connect the electronic device 1200 to a network, and an input / output (I / O) interface. The electronic device 1200 may operate based on an operating system stored in the memory 1220, such as Windows Server 200. TM , MacOS X TM , Unix TM , Linux TM , FreeBSD TM or similar.

[0178] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the above-mentioned electronic device 1200, enables the above-mentioned electronic device 1200 to execute a pronunciation error detection method or a pronunciation error detection model training method.

[0179] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present invention.

[0180] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0181] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0182] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0183] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0184] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program check codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0185] In addition, it should be noted that the combination of the various technical features in this case is not limited to the combination described in the claims of this case or the combination described in the specific embodiments. All technical features recorded in this case can be freely combined or combined in any way unless there is a contradiction between them.

[0186] It should be noted that the above examples are merely specific embodiments of the present invention. Obviously, the present invention is not limited to the above examples, and many similar variations are possible. All variations directly derived from or associating with the present invention by those skilled in the art are intended to fall within the scope of protection of the present invention.

[0187] It should be understood that the first, second, etc. qualifiers mentioned in the embodiments of the present invention are only used to more clearly describe the technical solutions of the embodiments of the present invention and cannot be used to limit the scope of protection of the present invention.

[0188] The above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A pronunciation error detection method, characterized in that: include: Obtain the speech signal to be detected and its corresponding reading text; Extracting acoustic features of the speech signal to be detected, and converting the read text into a phoneme sequence; obtaining acoustic features of at least one confused phoneme pair; Using a pronunciation error detection model, performing pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of the at least one pair of confused phoneme pairs, and the phoneme sequence; The method further comprises: determining whether the read-aloud text contains confusing phonemes; When the read text contains confusing phonemes, performing pronunciation error detection based on the output results of the pronunciation error detection model and the phoneme classification model corresponding to the confusing phonemes; Each of the confused phoneme pairs includes a first phoneme and a second phoneme; and the phoneme classification model is used to output a probability that the confused phoneme belongs to the first phoneme or the second phoneme.

2. The pronunciation error detection method according to claim 1, wherein The obtaining of acoustic features of at least one pair of confused phoneme pairs includes: Acquiring acoustic features of a speech sample, wherein the speech sample includes a plurality of speech segments corresponding to each of the at least one pair of confused phoneme pairs; Obtaining a covariance matrix corresponding to each pair of confused phoneme pairs according to the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; The acoustic features of the speech sample are fused with the covariance matrix corresponding to each pair of confused phonemes to obtain the acoustic features of each pair of confused phonemes.

3. The method according to claim 2, characterized in that The obtaining, based on the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs, a covariance matrix corresponding to each confused phoneme pair includes: Segmenting the speech sample to obtain a plurality of speech segments corresponding to the first phoneme and a plurality of speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; Extracting acoustic features of multiple speech segments corresponding to the first phoneme and clustering them to obtain N first-category center vectors; Extracting acoustic features of multiple speech segments corresponding to the second phoneme and clustering them to obtain N second-category center vectors; Dimensionality reduction is performed on the N first-category center vectors and the N second-category center vectors to obtain the covariance matrix.

4. A method for training a pronunciation error detection model, characterized in that: include: Acquire a training sample, wherein the training sample includes a speech signal sample and a corresponding text sample, wherein the speech signal sample includes speech information formed by a reader reading the text sample aloud; Extracting acoustic features of the speech signal sample and converting the text sample into a phoneme sequence; obtaining acoustic features of at least one confused phoneme pair; Based on the acoustic features of the speech signal sample, the acoustic features of the at least one pair of confused phoneme pairs and the phoneme sequence, the pronunciation error detection model is trained for error detection. The training method also includes: constructing at least one phoneme classification model corresponding to the at least one pair of confused phoneme pairs, so that when the text sample and / or the speech signal sample contains confused phonemes, pronunciation error detection is performed based on the output results of the pronunciation error detection model and the phoneme classification model, wherein each pair of confused phoneme pairs corresponds to a phoneme classification model, each pair of confused phoneme pairs includes a first phoneme and a second phoneme, and the phoneme classification model is used to output the probability that the confused phoneme belongs to the first phoneme or the second phoneme.

5. The training method according to claim 4, characterized in that The obtaining of acoustic features of at least one pair of confused phoneme pairs includes: Acquiring acoustic features of a speech sample, wherein the speech sample includes a plurality of speech segments corresponding to each of the at least one pair of confused phoneme pairs; Obtaining a covariance matrix corresponding to each pair of confused phoneme pairs according to the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; The acoustic features of the speech sample are fused with the covariance matrix corresponding to each pair of confused phonemes to obtain the acoustic features of each pair of confused phonemes.

6. The training method according to claim 5, characterized in that The obtaining, based on the multiple speech segments corresponding to the first phoneme and the multiple speech segments corresponding to the second phoneme in each pair of confused phoneme pairs, a covariance matrix corresponding to each confused phoneme pair includes: Segmenting the speech sample to obtain a plurality of speech segments corresponding to the first phoneme and a plurality of speech segments corresponding to the second phoneme in each pair of confused phoneme pairs; Extracting acoustic features of multiple speech segments corresponding to the first phoneme and clustering them to obtain N first-category center vectors; Extracting acoustic features of multiple speech segments corresponding to the second phoneme and clustering them to obtain N second-category center vectors; Dimensionality reduction is performed on the N first-category center vectors and the N second-category center vectors to obtain the covariance matrix.

7. The training method according to any one of claims 4 to 6, characterized in that: Before performing error detection training on the pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of the at least one confused phoneme pair, and the phoneme sequence, the method further includes: replacing some phonemes in the phoneme sequence using a mask; Based on the acoustic features of the speech signal sample and the phoneme sequence after mask replacement, the pronunciation error detection model is trained for speech recognition, wherein the pronunciation error detection model recognizes and outputs the phoneme corresponding to the replaced position.

8. The training method according to claim 4, characterized in that The constructing of at least one phoneme classification model corresponding to the at least one pair of confused phoneme pairs includes: Segmenting the speech sample to obtain a plurality of speech segments corresponding to each of the at least one pair of confused phoneme pairs; Obtaining a vector for each of the plurality of speech segments corresponding to the first phoneme, and obtaining a vector for each of the plurality of speech segments corresponding to the second phoneme; According to the vectors of the plurality of speech segments corresponding to the first phoneme and the vectors of the plurality of speech segments corresponding to the second phoneme, a phoneme classification model corresponding to the first phoneme and the second phoneme is trained.

9. A pronunciation error detection device, characterized in that: include: The first acquisition module is used to obtain the speech signal to be detected and its corresponding reading text; An extraction module, configured to extract acoustic features of the speech signal to be detected and convert the read text into a phoneme sequence; a second acquisition module, configured to acquire acoustic features of at least one pair of confused phoneme pairs; An error detection module is used to use a pronunciation error detection model to perform pronunciation error detection based on the acoustic features of the speech signal to be detected, the acoustic features of the at least one pair of confused phoneme pairs, and the phoneme sequence; and to determine whether the read text contains confused phonemes; when the read text contains confused phonemes, pronunciation error detection is performed based on the output results of the pronunciation error detection model and the phoneme classification model corresponding to the confused phonemes; wherein each pair of confused phoneme pairs includes a first phoneme and a second phoneme; and the phoneme classification model is used to output the probability that the confused phoneme belongs to the first phoneme or the second phoneme.

10. A training device for a pronunciation error detection model, characterized in that: include: A first acquisition module is configured to acquire a training sample, wherein the training sample includes a speech signal sample and its corresponding text sample, wherein the speech signal sample includes speech information generated by a reader reading the text sample aloud; An extraction module, configured to extract acoustic features of the speech signal sample and convert the text sample into a phoneme sequence; a second acquisition module, configured to acquire acoustic features of at least one pair of confused phoneme pairs; A training module is used to perform error detection training on a pronunciation error detection model based on the acoustic features of the speech signal sample, the acoustic features of the at least one pair of confused phoneme pairs, and the phoneme sequence; and to construct at least one phoneme classification model corresponding to the at least one pair of confused phoneme pairs, so that when the text sample and / or the speech signal sample contains confused phonemes, pronunciation error detection is performed according to the output results of the pronunciation error detection model and the phoneme classification model, wherein each pair of confused phoneme pairs corresponds to a phoneme classification model, each pair of confused phoneme pairs includes a first phoneme and a second phoneme, and the phoneme classification model is used to output the probability that the confused phoneme belongs to the first phoneme or the second phoneme.

11. A computer-readable storage medium having computer-executable instructions stored thereon, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the method according to any one of claims 1 to 8.

Images