Language learning assisting method and device
The language learning auxiliary device, which uses deep neural networks and DNN-HMM models, solves the problems of high cost, easy distraction and noise interference in existing technologies, realizes real-time ear return and high-definition pronunciation feedback, and improves the effect and efficiency of language learning.
Patent Information
- Application Number
- CN202511026335.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-07-24
- Publication Date
- 2025-09-26
AI Technical Summary
Existing language learning assistive devices are expensive, easily distracting, produce non-standard pronunciations, lack real-time ear monitoring, and are easily disturbed by external noise, which affects learning outcomes.
It uses a noise reduction model based on deep neural networks and a DNN-HMM phoneme alignment model to collect user pronunciation through a circular microphone array, performs real-time noise reduction and matches it with a standard pronunciation library, generates a multi-dimensional pronunciation report, and uses wireless headphones for real-time ear monitoring and pronunciation feedback.
It realizes real-time ear monitoring in noisy environments, provides high-definition pronunciation feedback, isolates external noise, improves learning effects, and enhances memory reinforcement.
Smart Images
Figure CN120708650A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of language learning assistance technology, and in particular to a language learning assistance method and device. Background Art
[0002] Current language learning assistance devices primarily use tablets, smartphones combined with apps, or standalone word memorization devices with screens. These devices are expensive, and parents must manage their use to avoid distractions.
[0003] Another problem with using a word memorizer is that its functions are relatively simple, and it uses local sound, without real-time ear feedback, and cannot strengthen memory in the brain.
[0004] The local sound generation method is limited by the quality of the loudspeakers in different devices, and the sound generation may be non-standard or distorted, affecting the final learning quality; in some places with ambient noise, users are easily distracted by external sounds and cannot concentrate on learning. Summary of the Invention
[0005] In order to address the problems existing in the prior art, the present invention aims to provide a language learning assistance method that can provide real-time ear monitoring and real-time analysis and comparison of pronunciation accuracy. Another object of the present invention is to provide a language learning assistance device that implements the above method.
[0006] To achieve the above object, the present invention provides a language learning assistance method, comprising:
[0007] S1. Collect user pronunciation audio through several microphones on the device body;
[0008] S2. Process the audio using a noise reduction model based on a deep neural network;
[0009] S3. Send the noise-canceled audio to the noise-canceling headphones via a low-latency wireless transmission protocol, achieving real-time in-ear feedback.
[0010] S4. extracting MFCC feature vectors of the denoised audio;
[0011] S5. Use the pre-trained phoneme alignment model to perform phoneme-level dynamic time warping matching between the user's pronunciation and the standard pronunciation library;
[0012] S6. Calculate phoneme accuracy, intonation deviation, and fluency scores based on the acoustic model to generate a multi-dimensional pronunciation report.
[0013] Furthermore, the training step of the noise reduction model includes:
[0014] 1) Construct a training dataset: clean speech and noise are mixed at an SNR of 0-30dB. Noise types include white noise and human voice noise.
[0015] 2) Data training is performed using a loss function, where the loss function is a SI-SNR (scale-invariant signal-to-noise ratio) loss function, which is used to measure the similarity between the target signal and the estimated signal.
[0016] Furthermore, the phoneme alignment model is a DNN-HMM model, which combines the powerful acoustic modeling capabilities of DNN with the sequence modeling capabilities of HMM (Hidden Markov Model). The DNN accurately determines "which phoneme the current sound resembles," while the HMM is responsible for rationally connecting and organizing these judgments on the timeline based on pronunciation rules and text constraints, ultimately outputting phoneme boundaries.
[0017] Furthermore, the multi-dimensional pronunciation report generating step includes:
[0018] 1) Calculate the phoneme error rate and mark three types of errors: substitution ( / θ / → / s / ), deletion (missing vowel), and insertion (superfluous phoneme), and mark the error location;
[0019] 2) extracting the fundamental frequency trajectory and calculating the intonation deviation value;
[0020] 3) Detect the proportion of silent segments, the location of silent segments, compare with standard pause points, and evaluate fluency.
[0021] Furthermore, it also includes:
[0022] 1) Dynamically retrieve the corresponding word's demonstration audio from the standard pronunciation library based on the pronunciation score results;
[0023] 2) Play the demonstration audio through the headphones and highlight the incorrect phonemes on the display simultaneously.
[0024] Furthermore, the process of processing audio by the noise reduction model includes:
[0025] 1) Perform time-frequency conversion on the audio to generate a Mel-spectrogram;
[0026] 2) Predicting the ideal float mask IRM through the encoder-decoder of the U-Net structure;
[0027] 3) Multiply the IRM with the noisy spectrum and output the noise-reduced audio through inverse transformation.
[0028] A language learning assistance device for implementing the above-mentioned language learning assistance method comprises:
[0029] Device body: built-in processor, memory, battery and charging port;
[0030] Ring microphone array: a number of digital microphones arranged symmetrically around the device body;
[0031] Touch screen: displays the operation interface and pronunciation feedback information;
[0032] Physical button group: includes word switching keys and recording control keys;
[0033] Headphone compartment: built-in pair of wireless headphones that support active noise reduction;
[0034] Wireless communication module: used for headphone connection and data synchronization.
[0035] Furthermore, the processor is configured as follows:
[0036] Execute the denoising model: using the U-Net architecture, the input is a 256-frame Mel-spectrogram, and the output is a 128-dimensional IRM vector;
[0037] Run the DNN-HMM phoneme alignment model; convert the audio into MFCC acoustic features, construct an HMM state sequence based on the transcript and pronunciation dictionary, use the trained DNN to calculate each possible HMM state, and use the Viterbi algorithm to calculate the optimal HMM state sequence path;
[0038] Real-time in-ear monitoring control: audio processing delay ≤ 50ms.
[0039] Furthermore, the earphone compartment and the device body adopt a magnetic coupling structure, and metal contacts are provided in the compartment for charging the earphones, with a charging current of 100mA.
[0040] Furthermore, an ambient light sensor is provided on the side wall of the device body for automatically adjusting the backlight intensity of the display screen according to the ambient brightness.
[0041] The beneficial effects of the present invention are as follows:
[0042] The present invention provides a language learning assistance method and device, which realizes real-time ear monitoring during the language learning process (such as memorizing words), can effectively isolate external noise, create a good learning environment, record in real time and high definition, and analyze and compare pronunciation accuracy in real time. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a schematic structural diagram of the present invention;
[0044] Figure 2 It is a schematic diagram of the internal structure of the present invention;
[0045] Figure 3 The phoneme alignment model structure. DETAILED DESCRIPTION
[0046] The following will clearly and completely describe the technical solution of the present invention in conjunction with the accompanying drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.
[0047] In the description of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicating orientations or positional relationships, are based on the orientations or positional relationships shown in the accompanying drawings and are intended solely to facilitate and simplify the description of the present invention. They are not intended to indicate or imply that the devices or components referred to must have, be constructed, or operate in a specific orientation, and therefore should not be construed as limitations on the present invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0048] In the description of the present invention, it should be noted that, unless otherwise expressly specified or limited, the terms "mounted," "connected," and "connected" should be understood in a broad sense. For example, they may refer to fixed, detachable, or integral connections; mechanical or electrical connections; direct or indirect connections through an intermediate medium; and internal communication between two components. Those skilled in the art will understand the specific meanings of the above terms in the present invention based on specific circumstances.
[0049] The following combination Figure 1 、 Figure 2 The specific embodiments of the present invention are described in detail. It should be understood that the specific embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.
[0050] The present invention provides a language learning assistance method, comprising:
[0051] S1. Collect user pronunciation audio through several microphones on the device body;
[0052] S2. Process the audio using a noise reduction model based on a deep neural network;
[0053] S3. Send the noise-canceled audio to the noise-canceling headphones via a low-latency wireless transmission protocol, achieving real-time in-ear feedback.
[0054] S4. extracting MFCC feature vectors of the denoised audio;
[0055] S5. Use the pre-trained phoneme alignment model to perform phoneme-level dynamic time warping matching between the user's pronunciation and the standard pronunciation library;
[0056] S6. Calculate phoneme accuracy, intonation deviation, and fluency scores based on the acoustic model to generate a multi-dimensional pronunciation report.
[0057] The training steps of the denoising model include:
[0058] 1) Construct a training dataset: clean speech and noise are mixed at an SNR of 0-30dB. Noise types include white noise and human voice noise.
[0059] 2) Data training is performed using a loss function, where the loss function is a SI-SNR (scale-invariant signal-to-noise ratio) loss function, which is used to measure the similarity between the target signal and the estimated signal.
[0060] The loss function is calculated as:
[0061] SISNR=10*log10(||s_target||2 / ||e_noise||2)
[0062] Where ||s_target||2 is the energy of the target signal after projection, and ||e_noise||2 is the energy of the residual noise.
[0063] The phoneme alignment model is a DNN-HMM model, which combines the powerful acoustic modeling capabilities of DNN with the excellent sequence modeling capabilities of HMM (Hidden Markov Model). DNN is responsible for accurately determining which phoneme (or part) the current sound resembles, while HMM is responsible for rationally connecting and organizing these judgments on the timeline based on pronunciation rules and text constraints, ultimately outputting phoneme boundaries. The phoneme alignment model specifically includes:
[0064] Input layer: 39-dimensional MFCC features;
[0065] Hidden layer: 3-layer 512-unit BiLSTM;
[0066] Output layer: The softmax layer outputs the phoneme state probability;
[0067] The optimal phoneme sequence is decoded by the Viterbi algorithm.
[0068] The phoneme alignment model structure is as follows Figure 3 As shown, where:
[0069] HMM: Each phoneme is modeled as a small HMM (typically with three states: start, middle, and end). These phoneme HMMs are connected according to the phoneme sequence of the words in the pronunciation dictionary to form a composite HMM for the entire sentence or utterance. The HMM defines the transition probability between states (P(S_t|S_{t-1})), that is, the probability of jumping from one phoneme state to itself or the next state (typically constrained by a language model or forced path).
[0070] DNN part:
[0071] Input: acoustic feature vectors (such as MFCCs, PLP, FBANK, etc.) of the current time t and its context (several frames before and after).
[0072] Output: A probability distribution that represents the probability that the feature belongs to each possible HMM state (usually all states of all phonemes) given the current acoustic feature (and its context). That is, P(S_t|O_t).
[0073] Function: DNN acts as a powerful pattern recognizer / classifier, learning to more accurately distinguish different phoneme states from complex acoustic features. It can capture more complex acoustic patterns and contextual dependencies than GMM.
[0074] Alignment process (forced alignment):
[0075] step:
[0076] Feature extraction: Convert the audio into a sequence of acoustic features (one vector per frame).
[0077] Construct a composite HMM: Based on the transcription text and pronunciation dictionary, construct an HMM state sequence (a finite state machine) representing this sentence.
[0078] Calculate observation likelihood: For each frame of acoustic feature O_t, use the trained DNN to calculate the probability P(S_i|O_t) that it belongs to each possible HMM state S_i.
[0079] Viterbi Algorithm: Under the mandatory path constraint (the path must follow the HMM state path corresponding to the text), the Viterbi algorithm is applied. This algorithm dynamically finds an optimal HMM state sequence path that maximizes the joint probability of generating an observation feature sequence along this path. This optimal path determines: which phoneme each frame belongs to (through its corresponding HMM state) and the starting and ending frames of each phoneme.
[0080] In step S2, the process of the noise reduction model processing the audio includes:
[0081] 1) Perform time-frequency conversion on the audio to generate a Mel-spectrogram;
[0082] 2) Predicting the ideal float mask IRM through the encoder-decoder of the U-Net structure;
[0083] 3) Multiply the IRM with the noisy spectrum and output the noise-reduced audio through inverse transformation.
[0084] The steps for generating a multi-dimensional pronunciation report include:
[0085] 1) Calculate the phoneme error rate and mark three types of errors: substitution ( / θ / → / s / ), deletion (missing vowel), and insertion (superfluous phoneme), and mark the error location;
[0086] 2) extracting the fundamental frequency trajectory and calculating the intonation deviation value;
[0087] 3) Detect the proportion of silent segments, the location of silent segments, compare with standard pause points, and evaluate fluency.
[0088] The present invention provides a language learning assistance method, further comprising:
[0089] 1) Dynamically retrieve the corresponding word's demonstration audio from the standard pronunciation library based on the pronunciation score results;
[0090] 2) Play the demonstration audio through the headphones and highlight the incorrect phonemes on the display simultaneously.
[0091] A language learning assisting device comprising:
[0092] Device body: built-in processor, memory, battery and charging port;
[0093] Ring microphone array: a number of digital microphones arranged symmetrically around the device body;
[0094] Touch screen: displays the operation interface and pronunciation feedback information;
[0095] Physical button group: includes word switching keys and recording control keys;
[0096] Headphone compartment: built-in pair of wireless headphones that support active noise reduction;
[0097] Wireless communication module: used for headphone connection and data synchronization.
[0098] The present invention provides a language learning auxiliary device, which mainly includes a device body, which includes a processor, a built-in memory, a touch screen, several buttons, one or more microphones, and an openable compartment cover for storing a pair of wireless headphones for monitoring, a battery, a charging data interface, etc.
[0099] The earphone compartment and the device body are magnetically coupled, with metal contacts inside for charging the earphones at 100mA. An ambient light sensor on the side of the device automatically adjusts the display backlight intensity based on ambient brightness. The device body can be charged using the charging data port, as well as for downloading content and device updates. A wireless connection allows access to a smartphone app for uploading and downloading learning data and device updates.
[0100] The processor is connected to the display to display the operating interface and content, and the buttons are used for auxiliary input, such as the previous word and the next word. The processor has wireless connection capabilities and can be connected to a pair of wireless headphones. When in use, the processor can output sound to the user's ears through the wireless headphones. To enhance the user experience, sound transmission uses low-latency wireless audio transmission technology. The wireless headphones use an active noise reduction algorithm to reduce ambient noise during user use, creating a good learning environment for users.
[0101] Using a denoising algorithm, a neural network algorithm is used to input a noisy spectrum and predict an ideal binary mask (IBM) or floating value mask (IRM). The specific real-time method is as follows:
[0102] 1. Preliminary training: using a mixture of various clear voices and background noise to form training sets and data sets to train the neural network model for noise suppression
[0103] 2. Preprocess the recordings using time-frequency conversion to calculate the spectrum of the real-time speech. Using the trained network model, we calculate the floating value mask of the frequency domain data and perform noise reduction on the speech to obtain clear user speech.
[0104] The processor is connected to one or more microphones to collect user audio. The processor's built-in digital signal processing unit (DSP) performs noise reduction on the collected user audio before outputting it to the headphones, providing the user with pure sound in noisy environments. The collected audio is compared with standard audio stored in the device's internal memory. The pre-trained Wav2Vec model is used to extract deep feature vectors of the user and standard audio. The cosine similarity or Euclidean distance between the feature vectors is then calculated to determine if the user's voice is standard.
[0105] The processor configuration is:
[0106] 1. Execute the denoising model: using the U-Net architecture, the input is a 256-frame Mel-spectrogram, and the output is a 128-dimensional IRM vector;
[0107] 2. Run the DNN-HMM phoneme alignment model; convert the audio into MFCC acoustic features, construct an HMM state sequence based on the transcript and pronunciation dictionary, use the trained DNN to calculate each possible HMM state, and use the Viterbi algorithm to calculate the optimal HMM state sequence path;
[0108] 3. Realize real-time in-ear monitoring control: audio processing delay ≤ 50ms.
[0109] Using speech recognition technology to align text and phonemes, using acoustic models to calculate matching probabilities or distances, and combining deep learning methods to learn pronunciation patterns, we ultimately quantify the differences between the user's pronunciation and the standard pronunciation in multiple dimensions, such as phonemes, words, stress, intonation, and fluency. We also diagnose specific error types and provide a comprehensive score and detailed feedback. The specific method is as follows:
[0110] 1. Feature extraction: Convert the original speech waveform into a numerical vector that can represent the pronunciation characteristics.
[0111] 2. Acoustic Model Distance: This model uses a deep neural network to directly learn the mapping from user speech features to pronunciation scores or error categories. The model is trained on a large amount of labeled data (including standard pronunciation and various mispronunciations) to learn the relationship between complex acoustic patterns and pronunciation quality.
[0112] 3. Error detection and diagnosis: Based on information such as phoneme alignment, posterior probability, and distance, the algorithm can detect specific types of pronunciation errors.
[0113] 4. Comprehensive Scoring and Feedback: Scores are given for phoneme accuracy, stress, intonation, fluency, and other aspects. The student will be given the specific word or phoneme that was mispronounced, the type of error, and a demonstration of standard pronunciation (recording or phonetic transcription) will be provided.
[0114] Technical advantages of the present invention:
[0115] 1. Using wireless headphones for sound reproduction provides accurate sound quality. Traditional solutions often use local speakers, which can easily cause sound distortion. Furthermore, if the sound is too low, the user may not be able to hear clearly, while if it is too loud, it may be distracting to others. Using headphones eliminates this problem.
[0116] 2. The device uses the microphone and the processor's built-in CODEC to capture the user's voice in high definition, which is then transmitted to the earphones via low-latency wireless. Users can monitor their pronunciation in real time and can also play back the recording. A built-in algorithm compares the sound and provides prompts or corrections for incorrectly pronounced words.
[0117] 3. The headphones themselves have noise reduction function, which can block external noise and create an immersive learning environment.
[0118] 4. Enhance memory with low-latency wireless ear monitoring. Isolating external noise creates an environment for deep learning and promptly identifies pronunciation issues. The display, microphone, and earphones all work together to achieve three-in-one viewing, listening, and speaking. The earphones can also be used as regular Bluetooth headphones to connect to mobile phones, expanding the product's versatility.
[0119] Any process or method described in the flowchart of the present invention or in other ways herein can be understood as representing a module, segment or portion of code including one or more executable instructions for implementing specific logical functions or process steps, which can be implemented in any computer-readable medium for use by an instruction execution system, device or apparatus. The computer-readable medium can be any medium that stores, communicates, propagates or transmits a program for use by an execution system, device or apparatus, including read-only memory, magnetic disk or optical disk, etc.
[0120] Throughout this specification, reference to terms such as "embodiment" and "example" indicates that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, those skilled in the art may combine or integrate different embodiments or examples described in this specification, as well as features therein, without creating any inconsistency.
[0121] Although the above content has shown and described the embodiments of the present invention, it can be understood that the above embodiments are exemplary and cannot be understood as limitations of the present invention. Ordinary technicians in this field can perform update operations such as changes, modifications, replacements and variations on the above embodiments within the scope of the present invention.
Claims
1. A language learning assistance method, characterized in that: The method comprises: S1. Collect user pronunciation audio through several microphones on the device body; S2. Process the audio using a noise reduction model based on a deep neural network; S3. Send the noise-canceled audio to the noise-canceling headphones via a low-latency wireless transmission protocol, achieving real-time in-ear feedback. S4. extracting MFCC feature vectors of the denoised audio; S5. Use the pre-trained phoneme alignment model to perform phoneme-level dynamic time warping matching between the user's pronunciation and the standard pronunciation library; S6. Calculate phoneme accuracy, intonation deviation, and fluency scores based on the acoustic model to generate a multi-dimensional pronunciation report.
2. The language learning assistance method according to claim 1, characterized in that: The training steps of the denoising model include: 1) Construct a training dataset: clean speech and noise are mixed at an SNR of 0-30dB. Noise types include white noise and human voice noise. 2) Use loss function for data training.
3. The language learning assistance method according to claim 1, wherein the phoneme alignment model is a DNN-HMM model, which combines the acoustic modeling capabilities of a DNN with the sequence modeling capabilities of an HMM hidden Markov model. The DNN is responsible for determining "which phoneme the current sound resembles," and the HMM is responsible for concatenating and organizing these judgments on a timeline based on pronunciation rules and text constraints, ultimately outputting phoneme boundaries.
4. The language learning assistance method according to claim 1, wherein: The multi-dimensional pronunciation report generating step comprises: 1) Calculate the phoneme error rate; 2) extracting the fundamental frequency trajectory and calculating the intonation deviation value; 3) Detect the proportion of silent segments, the location of silent segments, compare with standard pause points, and evaluate fluency.
5. The language learning assistance method according to claim 1, wherein: Also includes: 1) Dynamically retrieve the corresponding word's demonstration audio from the standard pronunciation library based on the pronunciation score results; 2) Play the demonstration audio through the headphones and highlight the incorrect phonemes on the display simultaneously.
6. The language learning assistance method according to claim 1, characterized in that: The process of the noise reduction model processing audio includes: 1) Perform time-frequency conversion on the audio to generate a Mel-spectrogram; 2) Predicting the ideal float mask IRM through the encoder-decoder of the U-Net structure; 3) Multiply the IRM with the noisy spectrum and output the noise-reduced audio through inverse transformation.
7. A language learning assistance device for implementing the language learning assistance method according to any one of claims 1 to 6, characterized in that: include: Device body: built-in processor, memory, battery and charging port; Ring microphone array: a number of digital microphones arranged symmetrically around the device body; Touch screen: displays the operation interface and pronunciation feedback information; Physical button group: including word switching keys and recording control keys; Headphone compartment: built-in pair of wireless headphones that support active noise reduction; Wireless communication module: used for headphone connection and data synchronization.
8. The language learning assisting device according to claim 7, wherein: The processor is configured as follows: Execute the denoising model: using the U-Net architecture, the input is a 256-frame Mel-spectrogram, and the output is a 128-dimensional IRM vector; Run the DNN-HMM phoneme alignment model; convert the audio into MFCC acoustic features, construct an HMM state sequence based on the transcript and pronunciation dictionary, use the trained DNN to calculate each possible HMM state, and use the Viterbi algorithm to calculate the optimal HMM state sequence path; Real-time in-ear monitoring control: audio processing delay ≤ 50ms.
9. The language learning assisting device according to claim 7, wherein: The earphone compartment and the device body adopt a magnetic coupling structure, and metal contacts are provided in the compartment for charging the earphones, with a charging current of 100mA.
10. The language learning assisting device according to claim 7, wherein: An ambient light sensor is provided on the side wall of the device body for automatically adjusting the backlight intensity of the display screen according to the ambient brightness.