Pronunciation correction method for deaf-mute

Abstract

The invention belongs to the technical field of pronunciation correction methods, and particularly discloses a deaf-mute pronunciation correction method, which comprises the following steps of: marking pronunciation units in standard audio data, extracting Mel-frequency cepstral coefficients and filter bank characteristics in the standard audio data, and establishing an acoustic model of a deep neural network hidden Markov model; acoustic features in standard audio data can be effectively extracted, user audio is converted into a text through an acoustic model, and time axis alignment with the standard audio data is carried out through a dynamic time warping algorithm, so that accurate scores are provided for the user audio, a user is helped to know own pronunciation ability more clearly, and the user experience is improved. And more efficient correction and learning are realized.

Description

Technical Field

[0001] The invention belongs to the technical field of pronunciation correction methods, and in particular relates to a pronunciation correction method for deaf-mute people. Background Art

[0002] Deaf-mute people generally have difficulty in pronunciation, mainly because they cannot hear their own voices and the speech of others, and therefore cannot accurately imitate and master the pronunciation of speech. Deaf-mute people lack auditory feedback on speech and cannot adjust and correct their pronunciation. In addition, since they cannot hear the subtle differences in speech, it is also difficult for them to distinguish and master the differences between different phonemes, resulting in inaccurate or confusing pronunciation. Therefore, deaf-mute people usually need to communicate through mouth shapes, gestures, and other auxiliary tools, and their pronunciation ability is limited to a certain extent.

[0003] In order to help the deaf-mute learn pronunciation, the existing technology has proposed a variety of ways to evaluate the pronunciation accuracy of the deaf-mute and make targeted correction suggestions. However, the existing technology generally only evaluates the pronunciation accuracy through a simple comparison between the deaf-mute's vocal audio and the reference audio, and lacks a comprehensive evaluation method for pronunciation accuracy, resulting in poor correction effect. Summary of the invention

[0004] The technical problem to be solved by the present invention is to overcome the defect that the pronunciation accuracy assessment of the pronunciation correction method in the prior art is not comprehensive enough, resulting in poor correction effect, thereby providing a pronunciation correction method for the deaf-mute.

[0005] A method for correcting pronunciation of a deaf-mute comprises the following steps:

[0006] Step S1: marking pronunciation units in standard audio data, extracting Mel-frequency cepstral coefficients and filter bank features in the standard audio data, and establishing an acoustic model of a deep neural network hidden Markov model based on the Mel-frequency cepstral coefficients and filter bank features;

[0007] Step S2: Display the follow-up text and record the speaker's follow-up audio;

[0008] Step S3: converting the audio into user audio data through an analog-to-digital converter;

[0009] Step S4: inputting the user audio data into the acoustic model to obtain the text of the pronunciation unit;

[0010] Step S5: aligning the user audio data and the corresponding standard audio data on a time axis by using a dynamic time warping algorithm;

[0011] Step S6: Score the audio, including accuracy score, fluency score, completeness score and rhythm score.

[0012] Furthermore, the method further includes step S7: comparing the user audio data with the corresponding standard audio data, identifying pronunciation errors, determining the causes of the errors, and generating correction suggestions.

[0013] Furthermore, in step S7, the correction suggestion includes: using different colors to distinguish and display the correctly pronounced part and the incorrectly pronounced part in the waveform diagram of the user audio data.

[0014] Furthermore, in step S7, the correction suggestion includes: displaying a dynamic mouth shape model according to the corresponding pronunciation position, showing the position of the lips and tongue when the corresponding part is correctly pronounced, and the direction of the airflow when pronouncing.

[0015] Furthermore, in step S7, the correction suggestion includes: converting the sound frequency of the correct pronunciation corresponding to the incorrect pronunciation into vibration, and transmitting it to the speaker by using the vibration of the bracelet.

[0016] Furthermore, in step S5, the user audio data and the corresponding standard audio data are aligned in time axis by finding the best time axis alignment path through calculation.

[0017] Furthermore, in step S6, the accuracy score is calculated as follows: Where C represents the number of correctly pronounced pronunciation units in the user audio data, and N represents the number of pronunciation units in the standard audio data.

[0018] Furthermore, in step S6, the fluency score is calculated as follows: Where U={u1,u2,...,u n} represents the feature vector of the user audio data, S = {s1, s2, ..., s m} represents the feature vector of the standard audio data, and D(U,S) represents the distance between the feature vector U of the user audio data and the feature vector S of the standard audio data.

[0019] Furthermore, in step S6, the calculation method of the integrity score is: Where L U Indicates the number of valid pronunciation units of the user's audio data, L S Indicates the number of pronunciation units in the standard audio data.

[0020] Furthermore, in step S6, the prosody score is calculated as follows: Among them, x i and iIt represents the value of the two waveforms of the user audio data and the corresponding standard audio data at the same time point i, n represents the number of data points, and RMSE can evaluate the similarity between the user audio data and the standard audio data at the same time point as an indicator for judging the prosody score.

[0021] Beneficial effects: The present invention belongs to the technical field of pronunciation correction methods, and specifically discloses a pronunciation correction method for the deaf-mute. The present invention can effectively extract the acoustic features in the standard audio data by marking the pronunciation units in the standard audio data, extracting the Mel-frequency cepstral coefficients and filter group features in the standard audio data, and establishing an acoustic model of a deep neural network hidden Markov model. The user audio is converted into text through the acoustic model, and the time axis is aligned with the standard audio data through a dynamic time warping algorithm, thereby providing an accurate score for the user audio, helping the user to understand his or her pronunciation ability more clearly, and achieving more efficient correction and learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0023] Figure 1 The figure is a schematic block diagram of the method flow of the present invention. DETAILED DESCRIPTION

[0024] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0025] In the description of the present application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present application, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0026] Reference Figure 1 As shown, the present invention provides a method for correcting pronunciation of a deaf-mute, comprising the following steps:

[0027] Step S1: marking pronunciation units in standard audio data, extracting Mel-frequency cepstral coefficients and filter bank features in the standard audio data, and establishing an acoustic model of a deep neural network hidden Markov model based on the Mel-frequency cepstral coefficients and filter bank features;

[0028] Step S2: Display the follow-up text and record the speaker's follow-up audio;

[0029] Step S3: converting the audio into user audio data through an analog-to-digital converter;

[0030] Step S4: inputting the user audio data into the acoustic model to obtain the text of the pronunciation unit;

[0031] Step S5: aligning the user audio data and the corresponding standard audio data on a time axis by using a dynamic time warping algorithm (DTW);

[0032] Step S6: scoring the audio, including accuracy score, fluency score, completeness score and rhythm score;

[0033] Step S7: Compare the user audio data with the corresponding standard audio data, identify pronunciation errors, determine the causes of the errors, and generate correction suggestions.

[0034] Specifically, in step S1, Praat is used to annotate the pronunciation units in the standard audio data to ensure that each pronunciation unit has an accurate timestamp. After the audio signal is divided into short frames and a window function is added, the spectrum is obtained by fast Fourier transform conversion, and the spectrum is filtered through a Mel filter bank to simulate the auditory characteristics of the human ear, and the Mel frequency cepstral coefficient (MFCC) and filter bank (FBank) features in the standard audio data are extracted. According to the annotated pronunciation units, the hidden Markov model (HMM) is initialized, the state transition probability and initial probability of each pronunciation unit are defined, and the structure of the deep neural network (DNN) is defined, which usually includes an input layer, a hidden layer, and an output layer. The extracted MFCC and FBank features are used as the input of the DNN, and the annotated pronunciation units are used as output labels, and the iteration is repeated until the model converges. When faced with new speech data, the same features are extracted, and the HMM state probability of each frame feature is calculated using the trained DNN-HMM model, and the optimal state sequence is converted into the corresponding pronunciation unit (phoneme or word), thereby realizing the conversion of speech into text;

[0035] Step S3 also includes converting the digital sound signal into a visual waveform diagram, so that the speaker can intuitively see the characteristics and differences of the pronunciation.

[0036] Specifically, two time series are defined: the user audio data time series X = [x1, x2, ..., xN ] and the corresponding standard audio data time series Y = [y1, y2, ..., y M ], containing N and M data points respectively; initialize an N×M matrix D, whose size is the product of the lengths of the two time series; where D(i,j) represents the i-th data point x in the time series X. i and the jth data point y in the time series Y j The distance between them is expressed using the Euclidean distance D(i,j)=(x i -y j ) 2 ; Construct an N×M cumulative distance matrix C, where C(i,j) represents the distance from the first data point x1 to the i-th data point x in the time series X. i , and the first data point y1 to the jth data point y in the time series Y j The minimum cumulative distance.

[0037] Find the best timeline alignment path:

[0038] C(i,j)=D(i,j)+min{C(i-1,j),C(i,j-1),C(i-1,j-1)}#;

[0039] In step S6, the accuracy score is calculated as follows: Where C represents the number of correctly pronounced pronunciation units in the user audio data, and N represents the number of pronunciation units in the standard audio data.

[0040] The fluency score is calculated as follows: Where U={u1,u2,...,u n} represents the feature vector of the user audio data, S = {s1, s2, ..., s m} represents the feature vector of the standard audio data, and D(U,S) represents the distance between the feature vector U of the user audio data and the feature vector S of the standard audio data. In this embodiment, the feature vector is Mel Frequency Cepstral Coefficient (MFCC).

[0041] The completeness score is calculated as follows: Where L U Indicates the number of valid pronunciation units of the user's audio data, L S Indicates the number of pronunciation units in the standard audio data.

[0042] The calculation method of the prosody score is: Among them, x i and iIt represents the value of the two waveforms of the user audio data and the corresponding standard audio data at the same time point i, n represents the number of data points, and RMSE can evaluate the similarity between the user audio data and the standard audio data at the same time point as an indicator for judging the prosody score.

[0043] Mapping the four scoring results in the range of 0 to 100 respectively can display the scoring results more intuitively.

[0044] In step S7, the correction suggestion includes: using different colors to distinguish and display the correctly pronounced part and the incorrectly pronounced part in the waveform diagram of the user's audio data.

[0045] In step S7, the correction suggestion includes: displaying a dynamic mouth shape model according to the corresponding pronunciation position, showing the position of the lips and tongue when the corresponding part is correctly pronounced, and the direction of the airflow when pronouncing.

[0046] In step S7, the correction suggestion includes: converting the sound frequency of the correct pronunciation corresponding to the incorrect pronunciation into vibration, and transmitting it to the speaker by using the vibration of the bracelet.

[0047] In this embodiment, the correction suggestion method is: mapping the frequency range of the sound to the frequency range that the bracelet vibration motor can handle. The frequency range of human voice is roughly between 85Hz and 255Hz, while the frequency of the bracelet vibration motor is between 10Hz and 100Hz. The sound frequency can be converted to the vibration frequency by linear mapping or logarithmic mapping.

[0048] f Vib =map(f Sound ,f minSound ,f maxSound ,f minVib ,f maxVib )#;

[0049] Among them, f Sound is the frequency of human voice, f minSound 、f maxSound is the minimum and maximum value of the human voice frequency range, f Vib is the vibration motor frequency of the bracelet, f minVib 、f maxVib It is the minimum and maximum value of the bracelet vibration motor frequency range.

[0050] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0051] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A method for correcting pronunciation of a deaf-mute, characterized in that: The following steps are involved: Step S1: marking pronunciation units in standard audio data, extracting Mel-frequency cepstral coefficients and filter bank features in the standard audio data, and establishing an acoustic model of a deep neural network hidden Markov model based on the Mel-frequency cepstral coefficients and filter bank features; Step S2: Display the follow-up text and record the speaker's follow-up audio; Step S3: converting the audio into user audio data through an analog-to-digital converter; Step S4: inputting the user audio data into the acoustic model to obtain the text of the pronunciation unit; Step S5: aligning the user audio data and the corresponding standard audio data on a time axis by using a dynamic time warping algorithm; Step S6: Score the audio, including accuracy score, fluency score, completeness score and rhythm score.

2. A method for correcting pronunciation of a deaf-mute according to claim 1, characterized in that: The method also includes step S7: comparing the user audio data with the corresponding standard audio data, identifying pronunciation errors, determining the causes of the errors, and generating correction suggestions.

3. A method for correcting pronunciation of a deaf-mute according to claim 2, characterized in that: In step S7, the correction suggestion includes: using different colors to distinguish and display the correctly pronounced part and the incorrectly pronounced part in the waveform diagram of the user's audio data.

4. A method for correcting pronunciation of a deaf-mute according to claim 2, characterized in that: In step S7, the correction suggestion includes: displaying a dynamic mouth shape model according to the corresponding pronunciation position, showing the position of the lips and tongue when the corresponding part is correctly pronounced, and the direction of the airflow when pronouncing.

5. A method for correcting pronunciation of a deaf-mute according to claim 2, characterized in that: In step S7, the correction suggestion includes: converting the sound frequency of the correct pronunciation corresponding to the incorrect pronunciation into vibration, and transmitting it to the speaker by using the vibration of the bracelet.

6. A method for correcting pronunciation of a deaf-mute according to claim 1, characterized in that: The best time axis alignment path is found by calculation, and the user audio data and the corresponding standard audio data are aligned in time axis.

7. A method for correcting pronunciation of a deaf-mute according to claim 1, characterized in that: In step S6, the accuracy score is calculated as follows: Where C represents the number of correctly pronounced pronunciation units in the user audio data, and N represents the number of pronunciation units in the standard audio data.

8. A method for correcting pronunciation of a deaf-mute according to claim 1, characterized in that: In step S6, the fluency score is calculated as follows: Where U={u1,u2,...,u n } represents the feature vector of the user audio data, S = {s1, s2, ..., s m } represents the feature vector of the standard audio data, and D(U,S) represents the distance between the feature vector U of the user audio data and the feature vector S of the standard audio data.

9. A method for correcting pronunciation of a deaf-mute according to claim 1, characterized in that: In step S6, the calculation method of the integrity score is: Where L U Indicates the number of valid pronunciation units of the user's audio data, L S Indicates the number of pronunciation units in the standard audio data.

10. A method for correcting pronunciation of a deaf-mute according to claim 1, characterized in that: In step S6, the prosody score is calculated as follows: Among them, x i and i It represents the value of the two waveforms of the user audio data and the corresponding standard audio data at the same time point i, n represents the number of data points, and RMSE can evaluate the similarity between the user audio data and the standard audio data at the same time point as an indicator for judging the prosody score.

Images