Tone evaluation rehabilitation training device and system

Abstract

A tone evaluation rehabilitation training device and system, the device comprises a flexible cap body and a bandage, the cap body is integrated with a 10-channel electrode plate to accurately cover language brain areas on both sides, electroencephalogram signals are transmitted in real time through Bluetooth 5.0, and a noise reduction earphone is used for providing sound output for a patient. The microphone is used for collecting audio of a patient and transmitting the audio to the processing module for analysis, the system dynamically regulates acoustic stimulation based on the neural activation feedback module, high-precision tone recognition is achieved in combination with audio preprocessing, SpecAugment data enhancement, a CNN-Transform mixed model and an attention mechanism, and training efficiency is optimized through pre-training model fine tuning and Warmup learning rate scheduling. An initial consonant-vowel-tone three-dimensional confusion probability table is introduced to divide interference intensity grades, high and low interference task paths are dynamically switched through a semantic error rate, and an evaluation module generates a four-tone accuracy rate, an F0 curve comparison graph and a personalized rehabilitation scheme. The recognition precision is improved, the training period of special crowds is shortened, and full-period self-adaptive rehabilitation is provided for preschool children to speech disorder patients.

Description

Technical Field

[0001] The present invention relates to the technical field of tone assessment rehabilitation training, in particular to a tone assessment rehabilitation training device and system. Background Art

[0002] Tones are one of the core elements of Mandarin, but for non-native speakers and some people with speech disorders, mastering tones often presents significant challenges, leading to inaccurate pronunciation and communication difficulties. Tone abnormalities are particularly prominent when foreign language speakers learn Mandarin, impacting not only their fluency but also their ability to understand and express language. Furthermore, people with language disorders such as aphasia and autism often experience severe impairments in pronunciation and language expression, particularly in the articulation of tones, which presents a key challenge in language recovery. Currently, technological advancements in speech recognition, language processing, and tone recognition are rapid, particularly in speech recognition and natural language processing. Technologies like iFlytek's have achieved high speech recognition accuracy and multilingual support. Furthermore, the Jingyun Rehabilitation Training Platform, a cloud-based speech rehabilitation training system, has also achieved certain applications in the rehabilitation of speech and language disorders. However, its application in tone perception training is still immature and has certain limitations.

[0003] Existing tone recognition systems have low accuracy for users such as children, those with speech disorders, and non-native speakers. They are particularly affected by factors such as background noise and non-standard pronunciation in continuous speech. Traditional feature engineering methods or shallow neural networks struggle to effectively model the pitch curve (F0 contour) and speech context of Chinese tones, and lack the ability to deeply model temporal information. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the above technical defects and provide a tone assessment rehabilitation training device and system.

[0005] In order to solve the above problems, the technical solution of the present invention is: a tone assessment rehabilitation training device, comprising: A cap body 1, wherein a binding strap 2 is connected to the lower end surface of the cap body 1; Electrode pads 3, several of which are mounted on the cap body 1. The EEG signal transmission of the electrode pads 3 is achieved through Bluetooth 5.0 connection, and is used to collect EEG signals of bilateral brain regions in real time. The bilateral brain regions include Oz, Pz, Cz, Fz, C3, C4, F3, F4, T3, and T4 regions, and the electrode pads 3 transmit the signals to the processing module via Bluetooth or a wired interface; Noise-cancelling headphones, with two noise-cancelling headphone circuits in the cap body, for providing sound output for the patient; The microphone is connected to one side of the cap body and is used to collect the patient's audio and transmit it to the processing module for analysis.

[0006] Furthermore, a tone assessment rehabilitation training system includes an audio acquisition module, a display interaction module and a database, and further includes: S1, Neural Activation Feedback Module: Generates acoustic stimulation signals based on the bilateral brain region synergy mechanism, monitors the intensity of high-frequency neural activity in the user's bilateral brain regions through real-time EEG, and dynamically adjusts the speech rate and repetition frequency of rising and falling tone samples; S2, audio preprocessing module, used for sampling, normalizing, endpoint detection and acoustic feature extraction of user input voice signals; S3, data enhancement module, uses the SpecAugment method based on time masking and frequency masking to enhance speech features; S4, the tone recognition model module, uses an end-to-end deep neural network combining CNN and Transformer encoders to perform tone recognition on the enhanced speech features; S5, learning rate scheduling module, adaptively adjusts the learning rate during training based on the Warmup strategy; S6, a pre-trained model replacement module, is used to replace the encoding part of the model with a model that has been pre-trained on large-scale speech and fine-tune it to adapt to the tone recognition task; S7, dynamic path generation module: This module divides interference intensity levels based on the initial-final-tone three-dimensional confusion probability table and drives the generation and path switching of situational dialogue tasks through the semantic error rate. S8, the evaluation and feedback module, generates quantitative scores and speech rehabilitation suggestions based on the recognition results.

[0007] Furthermore, the neural activation feedback module includes: Acoustic stimulation unit: synthesizes slow rising tone samples to strengthen the non-lateralized response of bilateral brain areas, and fast falling tone samples to train the attention stage response; Closed-loop control unit: When EEG monitoring shows a decrease in neural activity, it automatically increases the proportion of slow samples and compares the F0 curve deviation between the user's pronunciation and the target tone through a hybrid model.

[0008] Furthermore, the two-dimensional convolutional network (CNN) extracts local time-frequency features, and the calculation formula is: F1=ReLU(BN(Conv2D(X))).

[0009] Furthermore, the Transformer encoder models the temporal context through a multi-head attention mechanism, and the attention calculation formula is: .

[0010] Furthermore, the learning rate scheduling module adopts the Warmup strategy, and the scheduling function is: .

[0011] Furthermore, the data enhancement module performs time masking and frequency masking operations, where the masking formula is: as well as .

[0012] Furthermore, the pre-trained model replacement module uses a speech encoder of HuBERT, Wav2Vec2.0 or other Transformer architecture as a feature extractor.

[0013] Furthermore, the dynamic path generation module includes: Confusion Matrix Analysis Unit: Analyzes user tone perception data using a pre-trained language model, records the probability of a user misjudging a word as another, sets tone differences as low interference, tone + final differences as medium interference, and tone + final + initial differences as high interference, thereby generating a three-dimensional confusion probability table of initials, finals, and tones, and classifying the interference intensity levels accordingly.

[0014] Path switching unit: Set the threshold for semantic error rate switching according to the situation. The calculation formula for semantic error rate is: When the semantic error rate is too high, a low-interference task is enabled to amplify the tone difference. When the error rate is lower than the preset threshold, a high-intensity interference task is enabled to insert homophone pairs and shorten the tone duration.

[0015] Furthermore, the evaluation and feedback module can generate four-tone recognition accuracy, pronunciation curve comparison chart and personalized rehabilitation training plan.

[0016] The advantages of the present invention compared with the existing technology are: The present invention overcomes the defects of incomplete electrode coverage and delayed neural response monitoring of traditional equipment by precisely locating bilateral language brain areas through the cap and electrode sheets, combining Bluetooth 5.0 to transmit EEG signals in real time and dynamic acoustic regulation of the neural activation feedback module. At the same time, the collaborative processing of SpecAugment enhancement and CNN-Transformer hybrid model, combined with HuBERT pre-training fine-tuning and Warmup learning rate scheduling, significantly improves the accuracy of tone recognition in noisy environments, constructs a dynamic path generation module based on the three-dimensional confusion probability table of initials-finals-tones, and intelligently switches between high and low interference tasks through the semantic error rate threshold to achieve adaptive evolution of rehabilitation difficulty; finally, with the triple closed-loop architecture of neural feedback-speech processing-path regulation, it drives the real-time comparison of F0 curves and the quantitative evaluation of four tones to generate a visual rehabilitation plan, shortening the training cycle of special populations and forming a full-cycle, high-precision, adaptive rehabilitation support system for preschool children to patients with speech disorders, which completely solves the core technical bottlenecks of one-way training, functional fragmentation and single scenario of traditional systems. BRIEF DESCRIPTION OF THE DRAWINGS Figure 1 It is a three-dimensional schematic diagram of the device of the present invention; Figure 2 This is a flow chart of the neural activation feedback module of the present invention; Figure 3 Schematic diagram of the audio processing and tone recognition process of the present invention; Figure 4 This is a schematic diagram of the dynamic path generation module flow of the present invention; Figure 5 This is a flow chart of the evaluation and feedback module of the present invention; Figure 6 This is a flow chart of the pre-training model fine-tuning module of the present invention; As shown in the figure: 1. Cap body; 2. Strap; 3. Electrode; 4. Noise-canceling headphones; 5. Microphone. DETAILED DESCRIPTION

[0017] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0018] Example 1: This embodiment provides a tone assessment rehabilitation training device. When the device and system of the present invention are used in a preschool child's Mandarin tone learning scenario, the child wears a cap 1 equipped with 10-channel electrodes 3. The cap 1 is made of flexible material. The electrodes collect real-time EEG signals from ten points in the bilateral brain regions Oz, Pz, Cz, Fz, C3, C4, F3, F4, T3, and T4. The signals are transmitted to the processing module via Bluetooth 5.0. The noise-canceling headphones 4 issue instructions to the child to read the four tones of the initial consonant-final-tone combination "ma". The signals are transmitted to the audio acquisition module via microphone 5 for voice recording. The neural activation feedback module dynamically generates acoustic stimulation based on the real-time EEG data. When the activity of the right brain regions F4, C4, and T4 is detected to be attenuated, the proportion of slow rising tone samples is automatically increased. The child's voice is enhanced by time and frequency masking using the SpecAugment method. The masking formula is as follows: when or , otherwise keep the original value.

[0019] The enhanced Mel spectrum is input into the CNN+Transformer model CNN layer to perform feature extraction: F1=ReLU(BN(Conv2D(X))), and the Transformer encoder calculates the attention weight: .

[0020] Use the Warmup strategy to adjust the learning rate: , output the four-tone recognition results, generate F0 curve comparison chart and personalized rehabilitation plan.

[0021] Example 2: A tone assessment rehabilitation training device of this embodiment, when the device and system of the present invention are used in the rehabilitation scenario of patients with speech disorders, the user's voice is input into the pre-trained model replacement module, Wav2Vec2.0 is used as the feature extractor, and the learning rate scheduling module is enabled in the fine-tuning stage to dynamically adjust the learning rate according to the warmup function.

[0022] Start the dynamic training path to generate the confusion matrix analysis unit to build a three-dimensional probability table of initials, finals and tones based on the patient's historical data: Set the semantic error rate detection value according to the semantic error rate calculation formula: ,When the semantic error rate is a detection value, the path switching unit ,starts the low-interference task, and enables the high-interference task when the error rate is ,less than the detection value.

[0023] Finally, the evaluation module outputs tone accuracy statistics, F0 curve deviation values, and phased rehabilitation recommendations.

[0024] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0025] It will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

[0026] The present invention and its embodiments are described above. This description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if a person skilled in the art is inspired by this and, without departing from the purpose of the present invention, designs structures and embodiments similar to this technical solution without inventiveness, they shall fall within the scope of protection of the present invention.

Claims

1. A tone assessment rehabilitation training device, characterized in that include: A cap body (1), wherein a binding strap (2) is connected to the lower end surface of the cap body (1); Electrode sheets (3), wherein a plurality of the electrode sheets (3) are mounted on the cap body (1), and the EEG signal transmission of the electrode sheets (3) is realized through a Bluetooth 5.0 connection, and is used for real-time collection of EEG signals of bilateral brain regions, wherein the bilateral brain regions include Oz, Pz, Cz, Fz, C3, C4, F3, F4, T3, and T4 regions, and the electrode sheets (3) transmit the signals to the processing module through a Bluetooth or wired interface; Noise-cancelling headphones (4), two Bluetooth headphones (4) with circuits in the cap body (1) for providing sound output for the patient; A microphone (5) is connected to one side of the cap body (1) and is used to collect the patient's audio and transmit it to the processing module for analysis.

2. A tone assessment rehabilitation training system, comprising an audio acquisition module, a display interaction module and a database, characterized in that: Also includes: S1, Neural Activation Feedback Module: Generates acoustic stimulation signals based on the bilateral brain region synergy mechanism, monitors the intensity of high-frequency neural activity in the user's bilateral brain regions through real-time EEG, and dynamically adjusts the speech rate and repetition frequency of rising and falling tone samples; S2, audio preprocessing module, used for sampling, normalizing, endpoint detection and acoustic feature extraction of user input voice signals; S3, data enhancement module, uses the SpecAugment method based on time masking and frequency masking to enhance speech features; S4, the tone recognition model module, uses an end-to-end deep neural network combining CNN and Transformer encoders to perform tone recognition on the enhanced speech features; S5, learning rate scheduling module, adaptively adjusts the learning rate during training based on the Warmup strategy; S6, a pre-trained model replacement module, is used to replace the encoding part of the model with a model that has been pre-trained on large-scale speech and fine-tune it to adapt to the tone recognition task; S7, dynamic path generation module: This module divides interference intensity levels based on the initial-final-tone three-dimensional confusion probability table and drives the generation and path switching of situational dialogue tasks through the semantic error rate. S8, the evaluation and feedback module, generates quantitative scores and speech rehabilitation suggestions based on the recognition results.

3. The tone assessment rehabilitation training system according to claim 2, characterized in that: The neural activation feedback module includes: Acoustic stimulation unit: synthesizes slow rising tone samples to strengthen the non-lateralized response of bilateral brain areas, and fast falling tone samples to train the attention stage response; Closed-loop control unit: When EEG monitoring shows a decrease in neural activity, it automatically increases the proportion of slow samples and compares the F0 curve deviation between the user's pronunciation and the target tone through a hybrid model.

4. The tone assessment rehabilitation training system according to claim 2, characterized in that: The two-dimensional convolutional network (CNN) extracts local time-frequency features, and the calculation formula is: F1=ReLU(BN(Conv2D(X))).

5. The tone assessment rehabilitation training system according to claim 2, characterized in that: The Transformer encoder models temporal context through a multi-head attention mechanism, and the attention calculation formula is: 。 6. The tone assessment rehabilitation training system according to claim 2, characterized in that: The learning rate scheduling module adopts the Warmup strategy, and the scheduling function is: 。 7. The tone assessment rehabilitation training system according to claim 2, characterized in that: The data enhancement module performs time masking and frequency masking operations, where the masking formula is: as well as 。 8. The tone assessment rehabilitation training system according to claim 2, characterized in that: The pre-trained model replacement module uses a speech encoder of HuBERT, Wav2Vec2.0 or other Transformer architecture as a feature extractor.

9. The tone assessment rehabilitation training system according to claim 2, characterized in that: The dynamic path generation module includes: Confusion Matrix Analysis Unit: Analyzes user tone perception data using a pre-trained language model, records the probability of a user misjudging a word as another, sets tone differences as low interference, tone + final differences as medium interference, and tone + final + initial differences as high interference, thereby generating a three-dimensional confusion probability table of initials, finals, and tones, and classifying the interference intensity levels accordingly. Path switching unit: Set the threshold for semantic error rate switching according to the situation. The calculation formula for semantic error rate is: When the semantic error rate is too high, a low-interference task is enabled to amplify the tone difference. When the error rate is lower than the preset threshold, a high-intensity interference task is enabled to insert homophone pairs and shorten the tone duration.

10. The tone assessment rehabilitation training system according to claim 2, characterized in that: The evaluation and feedback module can generate four-tone recognition accuracy, pronunciation curve comparison chart and personalized rehabilitation training plan.

Images