A method for tone evaluation and improvement for Chinese learners

Abstract

The present application relates to the technical field of speech recognition, in particular to a tone evaluation and improvement method for Chinese learners, comprising: extracting the fundamental frequency F0 curve and vocal tract parameters of the speech signal, constructing a voicing self-adaptive fusion model, and fusing the fundamental frequency related features and the fundamental frequency independent features; constructing a tone error corpus, performing tone classification annotation, generating a multi-level tone feature set through the voicing self-adaptive fusion model, aligning the speech signal and the reference speech in time domain, and extracting and decoupling the tone shape feature vector and the tonal domain feature vector; based on the decoupled feature vector, establishing a double-channel evaluation path, calculating the tonal type distance and the tonal domain distance of the tone in layers, and performing weighted fusion to generate the final evaluation score, and finally generating feedback information for Chinese learners. The present application quantitatively diagnoses the pronunciation problems of learners by constructing a complete method from feature decoupling to double-channel evaluation, thereby improving the pertinence and efficiency of Chinese tone learning.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, specifically to a method for tone assessment and improvement for Chinese learners. Background Technology

[0002] In the field of speech recognition for Chinese learners, automatic tone evaluation is mainly used to identify learners' language. Existing technologies have evolved from early template matching algorithms to advanced methods that use deep learning models for comprehensive evaluation. For example, some methods combine deep neural networks with hidden Markov models, integrating multi-level features such as acoustics and prosody for tone recognition and evaluation; other methods combine phonetics knowledge to improve the accuracy of tone recognition by constructing models with specific topological structures.

[0003] However, despite the increasing complexity of existing speech recognition methods, they still have fundamental limitations in applications aimed at Chinese language learners. The core objective of these methods is to output a classification result or a global similarity score, failing to deconstruct learners' random and diverse pronunciation errors into different linguistic dimensions. Specifically, they cannot identify and quantify whether pronunciation deviations stem from structural errors in tone contours or from overall shifts in pitch range. This ambiguity in the assessment results prevents the system from providing accurate and diagnostic feedback, thus failing to meet the need for actionable guidance in language teaching.

[0004] Therefore, this invention proposes a method for tone assessment and improvement for Chinese learners. Summary of the Invention

[0005] The purpose of this invention is to provide a method for tone assessment and improvement for Chinese learners. This method uses a dual-channel assessment by constructing a voiced-voiced adaptive fusion model combined with a tone error corpus to achieve tone diagnosis and personalized feedback for Chinese learners.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for assessing and improving the tone of Chinese learners, including: The speech signal input by Chinese learners is preprocessed to extract the fundamental frequency F0 curve and vocal tract parameters of the speech signal, and a voiced-unvoiced adaptive fusion model is constructed. The voiced-unvoiced adaptive fusion model fuses the fundamental frequency-related features generated by the fundamental frequency F0 curve and the fundamental frequency-independent features generated by the vocal tract parameters. A corpus of tone errors is constructed, typical random error patterns are classified and labeled, and a multi-level tone feature set is generated based on the aforementioned voiced / voiced adaptive fusion model. Based on the multi-level tone feature set, the speech signal is aligned with the standard reference speech at the vowel level in the time domain, and the tone shape feature vector and tone domain feature vector are extracted and decoupled. Based on the decoupled tone shape feature vector and tone domain feature vector, a dual-channel evaluation path is established, the tone shape distance and tone domain distance of the tone are calculated in layers, and the tone shape distance and tone domain distance are weighted and fused according to the preset tone shape weight and tone domain weight to generate the final evaluation score. Based on the calculated values ​​of tone distance and tone range distance, personalized diagnostic feedback information is generated for Chinese learners.

[0007] Preferably, the preprocessing of the speech signal input by the Chinese learner includes: digitally sampling the speech signal, noise suppression, amplitude normalization, and pre-emphasis filtering to generate a standardized speech signal.

[0008] Preferably, the construction of the turbidity-clarity adaptive fusion model, which fuses the fundamental frequency F0 curve to generate fundamental frequency-related features and the duct parameters to generate fundamental frequency-independent features, includes: The fundamental frequency F0 curve is extracted using a fundamental frequency extraction algorithm, and harmonic correction and spline interpolation are performed to generate fundamental frequency-related features. In parallel, vocal tract parameters, including linear prediction model gain, are extracted to generate fundamental frequency-independent features. Based on the voiced characteristics of the speech segment, the voiced / unvoiced adaptive fusion model switches between and fuses fundamental frequency-related and fundamental frequency-independent features. Voiced segments use fundamental frequency-related features, unvoiced segments use fundamental frequency-independent features, and the transition segments between voiced and unvoiced segments undergo weighted fusion.

[0009] Preferably, the construction of the tone error corpus includes: Speech data from non-native Chinese learners is collected, including pronunciation samples of different tones. The collected data is labeled, including correct tones and typical random error patterns. The typical random error patterns include mispronouncing the second tone as the third tone, insufficient lowering of the third tone, and a flat lowering of the fourth tone. A corpus of tone errors is constructed. The speech samples in the corpus of tone errors are processed by the voiced-voiced adaptive fusion model to generate corresponding multi-level tone feature sets. The feature sets, along with the speech samples and labels, are stored in the corpus. The corpus of tone errors is used to verify the accuracy and stability of the evaluation results.

[0010] Preferably, the multi-level tone feature set includes: Frame-level features include Mel frequency cepstral coefficients, fundamental frequency F0 curve, and its first and second derivatives; Segment-level features include the fundamental frequency F0 curve fitting parameters of the syllable, syllable duration, mean fundamental frequency, and mean energy. Articulation features are extracted from Mel frequency cepstral coefficients using a pre-trained deep neural network to characterize vocal tract articulation actions; Prosodic context features are composed of acoustic information of adjacent syllables and characterize coarticulation and tone sandhi.

[0011] Preferably, the speech signal is aligned with the standard reference speech at the vowel level in the time domain, and the tone shape feature vector and tone domain feature vector are extracted and decoupled, including: Using a dynamic time warping algorithm, the speech signal is time-domain aligned with the standard reference speech at the boundaries of initials and finals to determine the speech segments to be compared; tone shape feature vectors are extracted, including normalized start pitch, end pitch, lowest pitch, highest pitch and their relative time positions, average slope of rising and falling segments and polynomial fitting coefficients; tone domain feature vectors are extracted, including fundamental frequency mean, fundamental frequency dynamic range and energy mean.

[0012] Preferably, the above-mentioned dual-channel evaluation path is established, and the tone shape distance and tone range distance of the tone are calculated hierarchically. The tone shape distance and tone range distance are then weighted and fused according to preset tone shape weights and tone range weights to generate a final evaluation score, including: In the first evaluation channel, the Euclidean distance between the learner's tone shape feature vector and the shape feature vectors of all standard tone models is calculated, and the distance to the target tone is taken as the tone shape distance. In the second evaluation channel, based on the result of the first channel: if the tone shape distance is lower than a preset tone shape similarity threshold, the difference between the learner's tone domain feature vector and the standard reference tone domain feature vector is calculated as the tone domain distance; if the tone shape distance is higher than the threshold, a preset penalty value is assigned to the tone domain distance. Based on the principle of perceived importance of tone shape over tone domain in Chinese linguistics, the output distances of the two channels are weighted and summed using tone shape weight and tone domain weight to generate the total tone error distance, which is mapped to the final evaluation score. The tone shape weight and tone domain weight are preset by statistical analysis of the tone error corpus and in combination with the principle of perceived importance of tone shape over tone domain in Chinese linguistics, and the tone shape weight is higher than the tone domain weight.

[0013] Preferably, the personalized diagnostic feedback information includes: Based on the values ​​of pitch shape distance and pitch range distance, feedback information is generated on the learning platform system. When the pitch shape distance is greater than a preset threshold, the system provides feedback on pitch shape errors and differences between the target pitch and the actual pitch. When the pitch shape distance is less than a preset threshold and the pitch range distance is greater than a preset threshold, the system provides suggestions that the pitch shape is correct but the pitch range fluctuation is insufficient. The system provides real-time pitch curve visualization, superimposing and comparing the learner's fundamental frequency F0 curve with the standard pitch fundamental frequency F0 curve. Based on the values ​​of pitch shape distance and pitch range distance, targeted practice content is recommended, including pitch shape identification exercises and exaggerated pitch fluctuation imitation exercises.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention decomposes and decouples tone information into two evaluation dimensions: "tone shape" and "tone range," thereby achieving precise quantification and attribution of pronunciation deviations, improving the accuracy and depth of diagnosis, and overcoming the limitations of existing technologies that can only provide fuzzy, single similarity scores.

[0015] 2. This invention establishes a dual-channel assessment path and diagnosis-practice mapping mechanism based on decoupling dimensions, which enables the direct transformation of quantitative assessment results into actionable personalized learning guidance. This makes the feedback no longer a general "right" or "wrong", but a specific and actionable improvement suggestion, thereby improving the efficiency of Chinese language teaching and self-study.

[0016] 3. This invention constructs an adaptive fusion model for voiced and unvoiced sounds, dynamically integrating fundamental frequency-related features and fundamental frequency-independent features such as duct parameters. This achieves continuity and stability of feature extraction in unvoiced, weak, and fundamental frequency-free scenarios, significantly enhancing the robustness of the evaluation and its applicability to real-world environments. It also solves the problem of traditional methods over-relying on high-quality fundamental frequency signals.

[0017] 4. This invention constructs and applies a corpus of tone errors specifically labeled with typical error patterns of Chinese learners (such as "insufficient tone sinking in the third tone"), enabling the assessment model to identify and quantify these specific, non-standard pronunciation patterns, making the assessment results closer to real teaching scenarios, and improving the diagnostic accuracy of common difficulties for learners. Attached Figure Description

[0018] Figure 1 This is a flowchart of a method for assessing and improving the tone of Chinese learners according to the present invention; Figure 2 This is a flowchart illustrating the internal logic of the turbidity-clarity adaptive fusion model in an embodiment of the present invention. Figure 3 This is a flowchart of the dual-channel conditional evaluation logic according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the diagnosis-practice mapping mechanism in an embodiment of the present invention. DETAILED DESCRIPTION OF EMBODIMENTS

[0019] To make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely preferred examples of the present invention, which are intended to explain the technical principles of the present invention, rather than limiting the protection scope of the present invention in any form. Based on the content disclosed herein, any insubstantial modification, equivalent substitution or improvement made by any person skilled in the art within the spirit and principle of the present invention shall be deemed to fall within the protection scope of the present invention.

[0020] Please refer to Figures 1 to 4 , the present invention provides a tone evaluation and improvement method for Chinese language learners.

[0021] Embodiment 1: Referring to Figure 1 , a tone evaluation and improvement method for Chinese language learners provided in this embodiment includes: preprocessing a voice signal input by a Chinese learner, extracting a fundamental frequency F0 curve and vocal tract parameters of the voice signal, and constructing a voiced-unvoiced adaptive fusion model, wherein the voiced-unvoiced adaptive fusion model fuses fundamental frequency-related features generated from the fundamental frequency F0 curve and fundamental frequency-independent features generated from the vocal tract parameters; constructing a tone error corpus, classifying and labeling typical random error patterns, and generating a multi-level tone feature set based on the voiced-unvoiced adaptive fusion model; performing initial and final level time-domain alignment between the voice signal and standard reference voice based on the multi-level tone feature set, and extracting and decoupling a tone contour feature vector and a tone range feature vector; establishing a dual-channel evaluation path based on the decoupled tone contour feature vector and tone range feature vector, calculating the tone contour distance and the tone range distance in a layered manner, and performing weighted fusion on the tone contour distance and the tone range distance according to preset tone contour weights and tone range weights to generate a final evaluation score; generating personalized diagnostic feedback information for Chinese language learners based on the respectively calculated values of the tone contour distance and the tone range distance.

[0022] Further, the preprocessing of the voice signal input by the Chinese learner includes: when a Chinese learner inputs a segment of voice through a microphone, for example, a target word "你好(nǐ hǎo)", the method of the present invention first preprocesses the input voice.

[0023] digital sampling: sampling the analog voice signal with a sampling rate of 16kHz and a quantization precision of 16 bits to ensure that sufficient voice details are captured; Noise suppression: Algorithms such as spectral subtraction or wavelet transform are used to filter out environmental background noise; Amplitude normalization: The amplitude of the sampled signal is linearly scaled to the range of [-1.0, 1.0] to eliminate the influence of different recording volumes; Pre-emphasis filtering: The signal is processed by a first-order high-pass filter to enhance the energy of the high-frequency part, compensate for the natural attenuation of high-frequency energy during the sound production process, and facilitate subsequent feature extraction.

[0024] This embodiment reduces interference from non-linguistic factors such as recording equipment, environment, and volume through standardized preprocessing, ensuring the stability and reliability of subsequent acoustic feature analysis.

[0025] Further, preferably, the construction of the turbidity-clarity adaptive fusion model, which fuses the fundamental frequency F0 curve to generate fundamental frequency-related features and the duct parameters to generate fundamental frequency-independent features, includes: This embodiment employs a fundamental frequency extraction algorithm, specifically the YIN algorithm, to extract the original fundamental frequency (F0) curve from the preprocessed speech signal. First, this method uses a harmonic correction module to correct common octave errors (misclassifying the actual fundamental frequency as a second harmonic) and half-frequency errors (misclassifying the second harmonic as the fundamental frequency) in the original F0 curve by detecting the energy of the harmonic structure. A cubic spline interpolation algorithm is then applied to the corrected F0 curve. This algorithm mathematically guarantees the continuity of the first and second derivatives at the interpolation points, generating a smooth and natural transition curve in the F0 interruption region caused by voiceless consonants (such as the initials / s / , / f / , / ch / ). Finally, a fundamental frequency-related feature that exists continuously throughout the entire syllable duration is generated as the first input.

[0026] While performing the above steps, the same speech signal is processed in parallel. The signal is divided into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. For each frame, the 12th-order linear prediction coefficients are calculated using the autocorrelation method. The energy of the prediction error signal of this linear prediction model is the linear prediction model gain, and its physical meaning is related to the energy of the excitation source of the speech. The linear prediction model gains of each frame are constructed into a continuous sequence. This sequence is independent of the presence or absence of the fundamental frequency and can effectively reflect the energy changes of the speech signal. This sequence serves as the fundamental frequency-independent feature of the second input.

[0027] For each speech analysis frame, low-level acoustic parameters such as short-time energy, zero-crossing rate and spectral flatness are calculated to jointly estimate the voicing probability of the frame. The probability is a continuous value between 0 and 1, and the voicing probability value is used as a dynamic fusion weight, wherein the method for jointly estimating the voicing probability of the frame specifically comprises: using a logistic regression classifier pre-trained on speech data annotated with voiced / unvoiced information, the classifier takes the short-time energy, zero-crossing rate and spectral flatness as input features, and the output of the classifier is the voicing probability of the frame.

[0028] Finally, the weight is applied to perform a frame-level weighted linear combination of the two feature streams. Specifically, the final fused feature of each frame is calculated by the following deterministic formula: Fused feature = (voicing probability × F0-dependent features) + ((1 - voicing probability) × F0-independent features) Taking the character "去 (qù)" as an example, its initial consonant / q / is unvoiced. In the initial pronunciation stage, the calculated voicing probability is close to 0, and the value of the fused feature is mainly determined by F0-independent features at this time; when the pronunciation transitions to the final / ü / , which is voiced, the calculated voicing probability rises rapidly and approaches 1, and the value of the fused feature is mainly determined by F0-dependent features (F0). The detailed internal logic flow of the model is shown in Figure 2 .[__END__]

[0029] To further improve the fusion quality, the method also introduces the consideration of fundamental frequency quality confidence when calculating the dynamic fusion weight. Specifically, firstly, the fundamental frequency quality confidence of each frame in the F0-dependent features is calculated, and the confidence characterizes the periodic intensity of the frame's fundamental frequency signal and the clarity of the harmonic structure; the dynamic fusion weight is generated based on the voicing probability and the fundamental frequency quality confidence; wherein, when the fundamental frequency quality confidence is lower than a preset quality threshold, the proportion of the F0-dependent features in the weighted fusion is reduced, which reduces the interference of low-quality fundamental frequency signals on the fused features and significantly enhances the robustness of the voiced / unvoiced adaptive fusion model.

[0030] The voiced / unvoiced adaptive fusion model of the method dynamically fuses F0-dependent and F0-independent features by extracting them in parallel according to the physical characteristics of speech, which fundamentally solves the problem of poor robustness caused by the excessive dependence on fundamental frequency in existing methods, and ensures the stability and reliability of the method when processing complex syllables including unvoiced sounds, aspirated sounds and the like. In addition, through correction and smoothing processing on the fundamental frequency curve, the accuracy and smoothness of F0-dependent features are greatly improved, which lays a high-quality data foundation for subsequent tone shape analysis.

[0031] Further, the constructing a tone error corpus comprises: Systematic speech data collection is conducted for non-native Chinese learner groups. To ensure the broad representativeness of the corpus, the native language backgrounds of the collected subjects cover multiple typical languages, including but not limited to English, Japanese, Korean, French and Russian. The collected speech content fully covers all combinations of initials, finals and tones in Mandarin Chinese, specifically including: Monosyllabic characters: including single characters with all four tones and the neutral tone, such as “妈(mā)”, “麻(má)”, “马(mǎ)”, “骂(mà)”, “吗(ma)”; Disyllabic words: covering all theoretically possible tone combinations, such as “飞机(fēi jī)”, “学习(xué xí)”, “你好(nǐ hǎo)”, “再见(zài jiàn)”; Specific short sentences: sentences containing easily confused tone combinations and typical tone sandhi phenomena (such as the tone sandhi of "一" and "不", and the tone sandhi of consecutive third tones).

[0032] Technicians with professional backgrounds in teaching Chinese as a foreign language or phonetics knowledge conduct independent cross-annotation on each collected speech. The annotation content includes not only standard Chinese Pinyin and correct tones, but also systematically classifies and annotates pronunciation errors. This classification system summarizes the random errors of learners into several typical patterns, for example: Tone pattern confusion error: for example, "the second tone is incorrectly pronounced as the third tone", whose objective judgment criterion is that the fundamental frequency (F0) curve of the actual pronunciation presents a complete "fall first then rise" shape; Tone range execution error: for example, "insufficient falling of the third tone", whose objective judgment criterion is that after speaker normalization processing is performed on the F0 curve of the pronunciation, the value of its actual valley point is higher than the 30% percentile position from low to high of the speaker's personal pitch range; for example, "the falling of the fourth tone is gentle, constructing a wrong tone corpus", whose objective judgment criterion is that the absolute value of the average falling slope from the inflection point to the end point of the F0 curve is less than 50% of the average slope of all correct fourth tone samples of the speaker; Consistency check is adopted in the annotation process to ensure the objectivity and reliability of the annotation results.

[0033] For each speech sample in the corpus, it is pre-processed by a voiced-unvoiced adaptive fusion model to extract its multi-level tone feature set; finally, each record in the corpus structurally includes at least four parts: the original speech signal, manually annotated text and error labels, the corresponding standard reference pronunciation sample, and the pre-calculated acoustic feature set.

[0034] The corpus will verify the accuracy and stability of the evaluation method described in this invention. For example, all samples labeled as "insufficient tone drop" can be selected from the corpus and input into the method of this invention in batches for evaluation. The accuracy of the diagnosis on this specific error type can be quantified by statistically analyzing the proportion of diagnoses that give "correct tone pattern but incorrect tone range".

[0035] This embodiment constructs a tone error corpus, which systematically and structurally classifies and quantifies the fuzzy and random tone errors of non-native language learners, providing a high-quality validation dataset for the field of tone teaching and assessment. At the same time, this corpus provides a solid and quantifiable validation foundation for the effectiveness of the assessment method.

[0036] Furthermore, the multi-level tone feature set includes: First level: Frame-level features. This level of features can capture the instantaneous, low-level acoustic characteristics of the speech signal. After dividing the signal into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds, the following features are extracted from each frame: 13. Vimer frequency cepstral coefficients (MFCC): used to characterize the spectral envelope of the signal in this frame, i.e. the timbre characteristics of the sound. Fundamental frequency F0 value: The fused F0 value in this frame, output by the turbidity-clarity adaptive fusion model; First and second derivatives: Calculate the difference and the difference coefficient of the above MFCC and F0 characteristics to characterize their dynamic change trend; The final frame-level feature vector is (13+1)×3=42 dimensions.

[0037] The second level: segmental level features, which characterize the overall acoustic pattern of a complete syllable (such as "nǐ" or "hǎo"), specifically include: F0 curve fitting parameters: The F0 curve within the syllable is fitted using a third-order polynomial with the least squares method, and the four coefficients of the polynomial are extracted as features; Duration characteristic: The duration of the syllable (in milliseconds); Statistical characteristics: mean F0, standard deviation of F0, and mean energy of all frames within the syllable; These features together describe the macroscopic tonal profile and energy distribution of a syllable.

[0038] Third level: pronunciation features. In this embodiment, the features are implemented by a deep neural network (DNN) containing a bottleneck layer that is pre-trained on a large amount of general speech data. A frame-level MFCC sequence is input into the DNN, and the output of its intermediate layer (i.e., the bottleneck layer) is extracted. The dimension of the bottleneck layer is usually low (e.g., 40 dimensions), and it has been proven that its output can effectively represent more compact and robust acoustic information that is highly correlated with specific articulatory movements.

[0039] Fourth level: prosodic context features. When analyzing the tone of the current target syllable, this method simultaneously extracts the segment-level features of its immediately preceding syllable and immediately succeeding syllable, and concatenates these features with the features of the current syllable. For example, when analyzing "nǐ" in the word "你好 (nǐ hǎo)", the segment-level features of the following "hǎo" are introduced as prosodic context features, which is crucial for accurately determining whether "nǐ" has undergone tone change from the third tone to the second tone.

[0040] In this embodiment, by constructing an information-rich multi-level tone feature set with comprehensive dimensions and fusing information from different levels, the micro dynamics (frame-level), macro contour (segment-level), intrinsic articulation (pronunciation features) and context dependence (prosodic context) of tones are captured simultaneously, which makes the description of tones more complete and accurate; secondly, the introduction of pronunciation features and prosodic context features enables the model to better handle complex tone change phenomena and has stronger tolerance to local errors in F0 extraction.

[0041] Further, performing time alignment at the initial and final level between the speech signal and a standard reference speech, extracting and decoupling a tone shape feature vector and a tone range feature vector, comprising: Processing the speech signal input by the learner through a preset speech recognition module or a forced alignment algorithm, so as to accurately determine the start and end time points of the initial part and final part of the target syllable therein, that is, the initial-final boundaries, and accurately place the analysis focus on the main part carrying tone information, that is, the final segment; then applying a dynamic time warping algorithm (DTW) to perform non-linear time axis alignment between the feature sequence of the learner's pronounced final segment and the feature sequence of the corresponding final segment of the standard reference pronunciation stored in the corpus. The DTW algorithm establishes the optimal corresponding relationship of each feature frame between two time sequences by finding a path with minimal energy dissipation.

[0042] After obtaining the aligned and duration-normalized final segment, two feature vectors respectively intended to characterize different perceptual dimensions of tones, that is, a tone shape feature vector and a tone range feature vector, are extracted in parallel.

[0043] To extract tone shape feature vectors, this method first traverses the fundamental frequency (F0) curves within the aligned vowel segments to find the fundamental frequency values ​​at the start, end, highest, and lowest points. Next, the fundamental frequency values ​​at these four key points are normalized. A preferred normalization method is T-Norm, which involves subtracting the speaker's average fundamental frequency over a longer speech segment (e.g., a complete practice session) from the current fundamental frequency value, and then dividing the difference by the speaker's fundamental frequency standard deviation within the same segment. The normalized value obtained through this process... The shape feature vector reflects only the relative pitch of the key point with respect to the speaker's personal vocal range, and no longer includes its absolute pitch information; the shape feature vector also includes other parameters describing the shape, such as the time points of the highest and lowest points, expressed as the percentage of the total vowel duration; the average slope of the main rising and falling segments, used to describe the speed of tone change; and four polynomial coefficients obtained by least-squares fitting of the entire F0 curve with a third-order polynomial, which can characterize the curvature of the entire curve in a compact way.

[0044] Simultaneously, this method also directly extracts and calculates the absolute physical quantities within the aligned vowel segments, without any normalization. These quantities specifically include: the arithmetic mean of the fundamental frequency values ​​of all voiced frames within the syllable, i.e., the fundamental frequency mean, measured in Hertz (Hz); the difference between the highest and lowest fundamental frequency values ​​within the syllable, i.e., the fundamental frequency dynamic range, also measured in Hertz (Hz), used to characterize the actual amplitude of pitch fluctuations; and the arithmetic mean of the energy values ​​of all frames within the syllable, i.e., the energy mean, used to characterize the loudness of the sound.

[0045] Furthermore, to assess the smoothness of pronunciation, this method introduces a third dimension in addition to extracting tone shape feature vectors. Specifically, after fitting the F0 curve with a third-order polynomial, the method calculates the root mean square of the sum of squares of the differences between each point on the original F0 curve and the corresponding point on the fitted curve. This value is defined as the fitting residual and directly used as the "smoothness distance." A smooth, natural pronunciation will have an F0 curve that closely matches the smooth polynomial curve, resulting in a small fitting residual (smoothness distance). Conversely, a shaky, unstable pronunciation will have an F0 curve that deviates from the fitted curve, leading to a large fitting residual. By introducing a novel "smoothness" evaluation dimension related to pronunciation stability, this method can quantify subtle defects such as undetectable pronunciation shakiness, thereby greatly improving the accuracy of the assessment and the ability to perceive the true quality of pronunciation.

[0046] This embodiment extracts "shape" features by normalizing the fundamental frequency and extracts "tonality" features by retaining its absolute value. It successfully separates a complex, coupled acoustic signal into two dimensions that are significantly different in perception and can be independently evaluated: the shape dimension, which mainly reflects the tone contour, and the tone dimension, which mainly reflects the pitch range. This makes it possible to conduct independent and accurate quantitative evaluations of "whether the tone is correct" and "whether the pitch range is full," which is a fundamental prerequisite for achieving accurate diagnosis and feedback.

[0047] Furthermore, the aforementioned dual-channel evaluation path calculates the tone pattern distance and tone range distance in layers, and weights and fuses the tone pattern distance and tone range distance according to preset tone pattern weights and tone range weights to generate a final evaluation score, including: In the first evaluation channel, the correctness of the learner's pronunciation tonal contour morphology is assessed. This method calculates the Euclidean distance between the learner's pronunciation tonal shape feature vector and the shape feature vectors of four pre-established standard tone models (obtained by taking the arithmetic mean of shape vectors from a large number of standard pronunciations in the tone error corpus). The method uses the calculated distance to the target tone model (e.g., the standard fourth tone model if the target tone is the fourth tone) as the final tone distance.

[0048] Subsequently, in the second evaluation channel, the accuracy of the physical implementation of tone is evaluated, and its calculation process is based on the evaluation results of the first channel. If the tone distance calculated in the previous step is lower than a preset tone similarity threshold, the method for determining the preset tone similarity threshold includes: calculating the intra-class distance distribution of all positive samples (i.e., correct pronunciation) and the inter-class distance distribution of negative samples (i.e., incorrect pronunciation) on the tone error corpus, and selecting the distance value that minimizes the classification error rate of the two classes of samples as the threshold, for example, by calculating the equal error rate point. This method determines that the learner's tone is basically correct. Under this condition, the method further calculates the normalized difference between the learner's tone domain feature vector and the tone domain feature vector of the standard reference pronunciation, and uses this as the tone domain distance; conversely, if the tone distance is higher than the threshold, the method determines that the learner has fundamentally confused tone. At this time, the accuracy of the tone domain is no longer meaningful for evaluation, and the method directly assigns a preset penalty value, such as 1.0, to the tone domain distance, which is sufficient to dominate the evaluation result.

[0049] Finally, to generate the total distance of tone errors, the method applies a preset tone type weight to the tone type distance and a preset tone range weight to the tone range distance, and then sums the two weighted distances. The tone type weight and tone range weight are not fixed but adaptively adjustable. Specifically, a learner proficiency profile is created for each Chinese learner, which is dynamically updated based on the learner's historical assessment data, considering the average tone type distance and average tone range distance. Before weighted fusion, the tone type weight and tone range weight are adaptively adjusted according to the learner proficiency profile. Specifically, when the profile indicates the learner is at a beginner level, the method automatically increases the tone type weight; when the profile indicates the learner is at an advanced level, the method automatically increases the tone range weight. The specific rules for adaptive adjustment are as follows: if the historical average tuning distance is greater than 1.5 times the historical average tuning range distance, the tuning weight will be temporarily increased by 0.1 and the tuning range weight will be decreased by 0.1; conversely, if the historical average tuning range distance is greater than 1.5 times the historical average tuning distance, the tuning range weight will be temporarily increased by 0.1 and the tuning weight will be decreased by 0.1.

[0050] This method upgrades the static assessment system into a personalized coaching model that can dynamically adjust the assessment focus based on the learner's proficiency, realizing intelligent assessment that is tailored to individual needs and significantly improving the relevance and efficiency of learning.

[0051] The detailed logic flow of this dual-channel evaluation path is as follows: Figure 3 As shown, the weights are pre-set based on statistical analysis of the tone error corpus, which found that tone pattern errors affect speech intelligibility more severely than tone range errors, and in accordance with this linguistic principle. For example, a typical setting is a tone pattern weight of 0.7 and a tone range weight of 0.3. Finally, the method maps the total error distance to an intuitive score of 0 to 100 using a non-linear function, such as subtracting the total distance from 1 and then multiplying by 100.

[0052] Before generating the final evaluation score, this method also considers the "smoothness distance" calculated above. If the smoothness distance is greater than a preset threshold (e.g., 0.5), it means that the learner's pronunciation has obvious instability or trembling. In this case, regardless of the results of the tone pattern and tone range evaluation, the final score will be deducted an additional fixed penalty score, such as 10 points.

[0053] This embodiment establishes a dual-channel, conditional assessment path that simulates the diagnostic logic of a language expert, assessing tone pattern first and then tone range, thus achieving a high degree of interpretability in the assessment process and results. This method can clearly distinguish the nature and severity of errors, making the assessment score no longer a vague number, but a transparent result that comprehensively quantifies the learner's pronunciation problems across two orthogonal dimensions.

[0054] Furthermore, the personalized diagnostic feedback information includes: First, the system generates diagnostic feedback information in text form. This method accesses a pre-set feedback template library and selects and fills the corresponding template based on whether the tone distance and pitch range distance exceed their respective preset thresholds. If the calculated tone distance is greater than its threshold, the system will select a template indicating an incorrect tone shape. The content of this template will explain the specific differences between the learner's pronunciation tone and the target tone, such as pointing out that a rising tone was mistakenly pronounced as a flat tone. Conversely, if the tone distance is less than its threshold but the pitch range distance is greater than its threshold, the system will select a template indicating an incorrect pitch range execution. The content of this template will explain that the learner's pronunciation shape is correct, but the amplitude or range of its pitch fluctuations is insufficient.

[0055] Secondly, the system provides real-time pitch curve visualization and comparison. On the user interface, the system overlays and compares the learner's fundamental frequency (F0) curve with the fundamental frequency (F0) curve of the standard reference speech. This method renders two graphical fundamental frequency curves on the user interface. The first curve is drawn based on the fundamental frequency data of the standard reference speech and is assigned a first visual attribute, such as being displayed as a solid green line, representing the correct target. The second curve is drawn in real-time based on the fundamental frequency data extracted from the learner's current pronunciation and is assigned a second visual attribute, such as being displayed as a dashed red line, representing the actual pronunciation. The overlay display of the two curves in the same coordinate system allows learners to intuitively and instantly compare the differences between their pronunciation and the standard pronunciation in terms of pitch variation trajectory.

[0056] To recommend targeted practice content, this recommendation process is also performed based on the values ​​of tone shape distance and tone range distance. This embodiment implements this function through a diagnostic-practice mapping mechanism, which can be implemented as a lookup table or a set of conditional rules. It takes the contribution of tone shape distance and tone range distance to the total distance of tone errors as input. If the mechanism determines that tone shape distance is the primary contributor to errors, it automatically retrieves and recommends exercises related to "tone shape discrimination" from a preset practice content library; if it determines that tone range distance is the primary contributor to errors, it accordingly retrieves and recommends exercises related to "exaggerated pitch variation imitation." The detailed logical flow of this diagnostic-practice mapping mechanism is as follows: Figure 4 As shown, this mechanism precisely maps the quantified evaluation distance to specific practice recommendations through a cascaded decision-making process.

[0057] The specific logical steps of this mechanism are as follows: Execution of Rule 1: The system first determines whether the tone distance is greater than the preset threshold A, which represents the critical value at which the tone is fundamentally confused.

[0058] If so, i.e., the tone distance is greater than the threshold A, then the incorrect attribution is determined to be "tone confusion", and the system will directly output and recommend practice content related to "tone shape discrimination".

[0059] Execution of Rule 2: If the conditions of Rule 1 are not met (i.e. the pitch pattern is basically correct), the process will proceed to the next level of judgment. The system will then judge whether the pitch range distance is greater than the preset threshold B. Threshold B represents the critical value for whether the pitch fluctuation is sufficient.

[0060] If so, i.e., the pitch range distance is greater than the threshold B, then the error attribution is determined to be "insufficient pitch range execution", and the system will output and recommend practice content related to "exaggerated imitation of pitch fluctuations".

[0061] Execution of Rule 3: If the conditions of Rule 2 are not met (i.e., the tone domain is also basically qualified), the process continues to the final level of judgment. The system judges whether the smoothness distance is greater than the preset threshold C. The threshold C represents the critical value for whether the pronunciation is stable and smooth.

[0062] If so, i.e., the smoothness distance > threshold C, then the error attribution is determined to be "poor pronunciation stability", and the system will output and recommend practice content related to "stable maintenance of long vowels".

[0063] Final judgment: If none of the "yes" paths of the above rules are triggered, the system will ultimately judge the pronunciation as "basic standard" and recommend "comprehensive consolidation exercises".

[0064] This embodiment establishes a "diagnosis-practice" mapping mechanism, which attributes errors based on quantified tone distance and tone range distance, thereby transforming assessment results into personalized and actionable guidance. This method can not only clearly identify the learner's error dimensions (tone or tone range), but also provide a three-in-one solution including visual comparison and targeted practice recommendations, thus forming an efficient learning loop and significantly improving learners' self-correction ability and learning efficiency.

[0065] This embodiment constructs a complete technical process integrating voiced / voiced adaptive fusion, a tone error corpus, feature decoupling, dual-channel evaluation, and diagnostic feedback. This results in a robust, accurate, and highly interpretable method for evaluating and improving Chinese tone patterns. This method not only overcomes the poor robustness of existing methods when dealing with complex speech, but also solves the fundamental problem of traditional evaluation methods' inability to accurately attribute errors by quantitatively evaluating tone errors as two independent dimensions: "shape" and "tone range." Ultimately, it provides learners with actionable and personalized learning guidance, significantly improving the efficiency of teaching and learning Chinese tone patterns.

[0066] Example 2: One specific embodiment of the present invention is applied in the International Chinese Language Teaching Center of a university to learners whose native language is a tonal language (such as Thai or Vietnamese). This embodiment aims to solve the problem of specific pronunciation errors caused by the negative transfer of the native language's tonal system to these learners.

[0067] First, a targeted corpus of tone errors is constructed. Specifically, a large amount of Chinese pronunciation data of learners with this background is collected and recorded, covering all initials, finals and tone combinations. Then, professionals with backgrounds in teaching Chinese as a foreign language and phonetics cross-label the collected speech. The labeling content includes not only the correct pinyin and tone, but also the classification of typical error patterns, such as: "the first tone (55) is pronounced as a middle tone (33)", "the fourth tone (falling tone) has insufficient falling momentum", and "the third tone (rising tone) has tone sandhi error". Finally, the labeled speech samples, text, error labels and corresponding standard pronunciation samples are stored in the database to form a corpus specifically for model fine-tuning and evaluation verification.

[0068] When a learner records the target word "teacher (lǎo shī)" through a client, the method of this invention begins to execute. The system first preprocesses the input speech signal by digital sampling, noise suppression, and amplitude normalization. Based on this, the system extracts a stable and reliable fundamental frequency (F0) curve from the preprocessed signal using a built-in voiced / voiced adaptive fusion model. Next, the system uses a vowel boundary recognition algorithm to focus the analysis on the vowel segments of "lǎo" and "shī" that carry tone information, and extracts two sets of independent feature vectors in parallel: one is the tone shape feature vector, representing the relative contour of the tone, obtained after normalizing the fundamental frequency curve; the other is the unnormalized tone range feature vector, representing the absolute physical properties of pitch.

[0069] The system then initiates a dual-channel evaluation path. In the first evaluation channel, the system calculates the Euclidean distance between the tone shape feature vector of the learner's pronunciation of "lǎo" and the shape feature vector of the standard third tone model. This distance is defined as the "tone shape distance." The calculation in the second evaluation channel is based on the result of the first channel. The system will determine whether the calculated "tone shape distance" is lower than a preset tone shape similarity threshold. If it is lower than the threshold, it indicates that the tone shape of the learner's pronunciation is basically correct. The system will then continue to calculate the difference between the tone feature vector of the learner's pronunciation and the tone feature vector of the standard reference pronunciation, and define this difference as the "tone shape distance." Conversely, if the "tone shape distance" is higher than the threshold, it indicates that the learner has suffered from fundamental tone shape confusion, and the system will directly assign a preset penalty value to the "tone shape distance." Finally, the system uses a weighted sum of "tone distance" and "tone range distance" based on a pre-set weight according to linguistic principles (tone type weight is 0.7, tone range weight is 0.3) to generate a total distance of tone error, which is then mapped to an intuitive evaluation score from 0 to 100.

[0070] After the assessment is completed, the system will convert the results into personalized diagnostic and practice recommendations. If the assessment results show that the "tone shape distance" is small but the "tone range distance" is large, the system will select and fill the built-in feedback template library to generate text feedback such as "The tone shape of the third tone (falling then rising) is basically correct, but the pitch range fluctuation is insufficient, and the amplitude of the sound sinking and rising is not enough".

[0071] Meanwhile, on the user interface, the system will overlay the learner's fundamental frequency F0 curve (represented by a red dashed line) with the standard pronunciation's fundamental frequency F0 curve (represented by a green solid line), allowing learners to intuitively see that the turning point of their curve is significantly smaller than that of the standard curve. Based on the judgment that "tonal range distance" is the main contributor to errors, the system will automatically retrieve and recommend exercises related to "exaggerated imitation of pitch fluctuations" from the exercise content library, thereby guiding learners to make targeted improvements.

[0072] Example 3: Another specific embodiment of the present invention is applied to the teaching of learners whose native language is a non-tonal language (such as English or French) in the same teaching environment. This embodiment aims to solve the problem that such learners have difficulty distinguishing and pronouncing the four basic tone shapes due to a lack of tone concept.

[0073] In this embodiment, to address the fundamental challenges faced by learners whose mother tongues are non-tonal languages, the method of the present invention performs adaptive adjustment first. Based on the user's language background tag, the system adjusts the weights of the two-channel assessment before the assessment starts, and significantly increases the "tone pattern weight" that represents the correctness of the tone contour to 0.85, while correspondingly reducing the "tone range weight" that represents the absolute pitch range to 0.15, so as to ensure that the core of the assessment focuses on the correctness of the tone shape. The construction of the corresponding tone error corpus also focuses on annotating fundamental tone pattern errors of the "tone confusion" category, for example "mispronouncing the second tone (Yangping) as the fourth tone (Qusheng)".

[0074] When a learner records the target word "huí lái (huí lái)", if both of the originally rising second tones are pronounced as level tones, the assessment procedure is as follows: after completing preprocessing and feature extraction, the system enters the two-channel assessment path. In the first assessment channel, the system calculates the "tone pattern distance" between the shape feature vector of the learner's level tone and the shape feature vector of the standard second tone (rising tone) model. Since the shapes do not match at all, this distance will be very large, far exceeding the preset similarity threshold; therefore, when entering the second assessment channel, the system determines that the learner has made a fundamental tone pattern error, and directly assigns a penalty value to the "tone range distance"; in the final weighted fusion step, since the large "tone pattern distance" is multiplied by a weight as high as 0.85, the generated assessment score will be very low, thereby accurately reflecting the learner's fundamental problem in tone mastery.

[0075] Based on the core assessment result that the "tone pattern distance" is much larger than the threshold, the system will generate diagnostic text such as "Tone shape error. Your pronunciation is level, but the target tones of both 'huí' and 'lái' need to rise continuously". In the visual feedback, two flat red curves of the learner's pronunciation and two rising green curves of the standard pitch will be presented on the interface, forming an extremely sharp contrast; based on the judgment that "tone confusion" is the main error type, the system will automatically recommend the most basic "tone shape discrimination" exercises, for example, a gamified exercise of identifying shapes by listening (level "-", rising "´", falling-rising "ˇ", falling "`"), to help learners fundamentally establish the connection between tone shapes and auditory perception, and lay a solid foundation for subsequent pronunciation imitation and correction.

[0076] Although specific embodiments of the present invention have been shown, those skilled in the art should understand that various adjustments, modifications, substitutions and improvements to these embodiments are possible without departing from the core principle and spirit of the present invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for assessing and improving the tone of Chinese learners, characterized in that, Includes the following steps: The speech signal input by Chinese learners is preprocessed to extract the fundamental frequency F0 curve and vocal tract parameters of the speech signal, and a voiced-unvoiced adaptive fusion model is constructed. The voiced-unvoiced adaptive fusion model fuses the fundamental frequency-related features generated by the fundamental frequency F0 curve and the fundamental frequency-independent features generated by the vocal tract parameters. Specifically, this includes: extracting the fundamental frequency F0 curve using a fundamental frequency extraction algorithm, performing harmonic correction and spline interpolation to generate fundamental frequency-related features; extracting vocal tract parameters in parallel, including linear prediction model gain, to generate fundamental frequency-independent features; and switching and fusing fundamental frequency-related features and fundamental frequency-independent features through the voiced / unvoiced adaptive fusion model based on the voiced / unvoiced characteristics of the speech segment, wherein the voiced segment uses fundamental frequency-related features, the unvoiced segment uses fundamental frequency-independent features, and the transition segment between the voiced and unvoiced segments is weighted and fused. A corpus of tone errors is constructed, typical random error patterns are classified and labeled, and a multi-level tone feature set is generated based on the aforementioned voiced / voiced adaptive fusion model. Based on the multi-level tone feature set, the speech signal is aligned with the standard reference speech at the vowel level in the time domain, and the tone shape feature vector and tone domain feature vector are extracted and decoupled. Based on decoupled tone shape feature vectors and tone domain feature vectors, a dual-channel evaluation path is established, and the tone shape distance and tone domain distance are calculated hierarchically. Specifically, in the first evaluation channel, the Euclidean distance between the learner's tone shape feature vector and the shape feature vectors of all standard tone models is calculated, and the distance with the target tone is taken as the tone shape distance. In the second evaluation channel, based on the result of the first channel: if the tone shape distance is lower than a preset tone shape similarity threshold, the difference between the learner's tone domain feature vector and the standard reference tone is calculated as the tone domain distance; if the tone shape distance is higher than the threshold, a preset penalty value is assigned to the tone domain distance, and the tone shape distance and tone domain distance are weighted and fused according to preset tone shape weights and tone domain weights to generate the final evaluation score. Based on the calculated values ​​of tone distance and tone range distance, personalized diagnostic feedback information is generated for Chinese learners.

2. The method for tone assessment and improvement for Chinese learners according to claim 1, characterized in that, The preprocessing of the speech signal input by the Chinese learner includes: digital sampling, noise suppression, amplitude normalization, and pre-emphasis filtering of the speech signal to generate a standardized speech signal.

3. The method for assessing and improving tone in Chinese learners according to claim 1, characterized in that, The construction of the tone error corpus includes: Speech data from non-native Chinese learners is collected, including pronunciation samples of different tones. The collected data is labeled, including correct tones and typical random error patterns. The typical random error patterns include mispronouncing the second tone as the third tone, insufficient lowering of the third tone, and a flat lowering of the fourth tone. A tone error corpus is constructed. The speech samples in the tone error corpus are processed by the voiced / voiced adaptive fusion model to generate corresponding multi-level tone feature sets, which are then associated with the speech samples and labels and stored together in the tone error corpus. The tone error corpus is used to verify the accuracy and stability of the evaluation results.

4. The method for tone assessment and improvement for Chinese learners according to claim 1, characterized in that, The multi-level tone feature set includes: Frame-level features include Mel frequency cepstral coefficients, fundamental frequency F0 curve, and its first and second derivatives; Segment-level features include the fundamental frequency F0 curve fitting parameters of the syllable, syllable duration, mean fundamental frequency, and mean energy. Articulation features are extracted from Mel frequency cepstral coefficients using a pre-trained deep neural network to characterize vocal tract articulation actions; Prosodic context features are composed of acoustic information of adjacent syllables and characterize coarticulation and tone sandhi.

5. The method for assessing and improving tone patterns for Chinese learners according to claim 1, characterized in that, The speech signal is aligned with the standard reference speech at the vowel level in the time domain, and the tone shape feature vector and tone domain feature vector are extracted and decoupled, including: Using a dynamic time warping algorithm, the speech signal is time-domain aligned with the standard reference speech at the boundaries of initials and finals to determine the speech segments to be compared; tone shape feature vectors are extracted, including normalized start pitch, end pitch, lowest pitch, highest pitch and their relative time positions, average slope of rising and falling segments and polynomial fitting coefficients; tone domain feature vectors are extracted, including fundamental frequency mean, fundamental frequency dynamic range and energy mean.

6. The method for assessing and improving tone in Chinese learners according to claim 1, characterized in that, The above-mentioned dual-channel evaluation path calculates the tone shape distance and tone range distance of the tone layer by layer. Based on preset tone shape weights and tone range weights, the tone shape distance and tone range distance are weighted and fused to generate the final evaluation score, including: Based on the principle of perceived importance of tone type over tone domain in Chinese linguistics, the output distances of the two channels are weighted and summed using tone type weight and tone domain weight to generate the total distance of tone errors, which is then mapped to the final evaluation score. The tone type weight and tone domain weight are pre-set by statistical analysis of the tone error corpus and in accordance with the principle of perceived importance of tone type over tone domain in Chinese linguistics, with the tone type weight being higher than the tone domain weight.

7. The method for assessing and improving tone patterns for Chinese learners according to claim 1, characterized in that, The personalized diagnostic feedback information includes: Based on the values ​​of pitch shape distance and pitch range distance, feedback information is generated on the learning platform system. When the pitch shape distance is greater than a preset threshold, the system provides feedback on pitch shape errors and differences between the target pitch and the actual pitch. When the pitch shape distance is less than a preset threshold and the pitch range distance is greater than a preset threshold, the system provides suggestions that the pitch shape is correct but the pitch range fluctuation is insufficient. The system provides real-time pitch curve visualization, superimposing and comparing the learner's fundamental frequency F0 curve with the standard pitch fundamental frequency F0 curve. Based on the values ​​of pitch shape distance and pitch range distance, targeted practice content is recommended, including pitch shape identification exercises and exaggerated pitch fluctuation imitation exercises.

Images