Chinese learner-oriented tone evaluation and improvement method

Abstract

The invention relates to the technical field of speech recognition, in particular to a Chinese learner-oriented tone evaluation and improvement method, which comprises the following steps of: extracting a fundamental frequency F0 curve and sound channel parameters of a speech signal, constructing a turbid and clear adaptive fusion model, and fusing to generate fundamental frequency related characteristics and fundamental frequency unrelated characteristics; constructing a tone error corpus, carrying out tone classification labeling, generating a multi-level tone feature set through a turbid and clear adaptive fusion model, aligning a voice signal with a reference voice time domain, and extracting and decoupling a tone shape feature vector and a tone domain feature vector; and based on the decoupled feature vectors, establishing a dual-channel evaluation path, hierarchically calculating the tone type distance and the tone domain distance of the tones, carrying out weighted fusion, generating a final evaluation score, and finally generating feedback information for the Chinese learner. According to the method, a complete method from feature decoupling to two-channel evaluation is constructed, the pronunciation problem of the learner is quantitatively diagnosed, and the pertinence and efficiency of Chinese tone learning are improved.

Description

Technical Field

[0001] This invention relates to the field of speech recognition technology, specifically to a method for tone assessment and improvement for Chinese learners. Background Technology

[0002] In the field of speech recognition for Chinese learners, automatic tone evaluation is mainly used to identify learners' language. Existing technologies have evolved from early template matching algorithms to advanced methods that use deep learning models for comprehensive evaluation. For example, some methods combine deep neural networks with hidden Markov models, integrating multi-level features such as acoustics and prosody for tone recognition and evaluation; other methods combine phonetics knowledge to improve the accuracy of tone recognition by constructing models with specific topological structures.

[0003] However, despite the increasing complexity of existing speech recognition methods, they still have fundamental limitations in applications aimed at Chinese language learners. The core objective of these methods is to output a classification result or a global similarity score, failing to deconstruct learners' random and diverse pronunciation errors into different linguistic dimensions. Specifically, they cannot identify and quantify whether pronunciation deviations stem from structural errors in tone contours or from overall shifts in pitch range. This ambiguity in the assessment results prevents the system from providing accurate and diagnostic feedback, thus failing to meet the need for actionable guidance in language teaching.

[0004] Therefore, this invention proposes a method for tone assessment and improvement for Chinese learners. Summary of the Invention

[0005] The purpose of this invention is to provide a method for tone assessment and improvement for Chinese learners. This method uses a dual-channel assessment by constructing a voiced-voiced adaptive fusion model combined with a tone error corpus to achieve tone diagnosis and personalized feedback for Chinese learners.

[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for assessing and improving the tone of Chinese learners, including: The speech signal input by Chinese learners is preprocessed to extract the fundamental frequency F0 curve and vocal tract parameters of the speech signal, and a voiced-unvoiced adaptive fusion model is constructed. The voiced-unvoiced adaptive fusion model fuses the fundamental frequency-related features generated by the fundamental frequency F0 curve and the fundamental frequency-independent features generated by the vocal tract parameters. A corpus of tone errors is constructed, typical random error patterns are classified and labeled, and a multi-level tone feature set is generated based on the aforementioned voiced / voiced adaptive fusion model. Based on the multi-level tone feature set, the speech signal is aligned with the standard reference speech at the vowel level in the time domain, and the tone shape feature vector and tone domain feature vector are extracted and decoupled. Based on the decoupled tone shape feature vector and tone domain feature vector, a dual-channel evaluation path is established, the tone shape distance and tone domain distance of the tone are calculated in layers, and the tone shape distance and tone domain distance are weighted and fused according to the preset tone shape weight and tone domain weight to generate the final evaluation score. Based on the calculated values ​​of tone distance and tone range distance, personalized diagnostic feedback information is generated for Chinese learners.

[0007] Preferably, the preprocessing of the speech signal input by the Chinese learner includes: digitally sampling the speech signal, noise suppression, amplitude normalization, and pre-emphasis filtering to generate a standardized speech signal.

[0008] Preferably, the construction of the turbidity-clarity adaptive fusion model, which fuses the fundamental frequency F0 curve to generate fundamental frequency-related features and the duct parameters to generate fundamental frequency-independent features, includes: The fundamental frequency F0 curve is extracted using a fundamental frequency extraction algorithm, and harmonic correction and spline interpolation are performed to generate fundamental frequency-related features. In parallel, vocal tract parameters, including linear prediction model gain, are extracted to generate fundamental frequency-independent features. Based on the voiced characteristics of the speech segment, the voiced / unvoiced adaptive fusion model switches between and fuses fundamental frequency-related and fundamental frequency-independent features. Voiced segments use fundamental frequency-related features, unvoiced segments use fundamental frequency-independent features, and the transition segments between voiced and unvoiced segments undergo weighted fusion.

[0009] Preferably, the construction of the tone error corpus includes: Speech data from non-native Chinese learners is collected, including pronunciation samples of different tones. The collected data is labeled, including correct tones and typical random error patterns. The typical random error patterns include mispronouncing the second tone as the third tone, insufficient lowering of the third tone, and a flat lowering of the fourth tone. A corpus of tone errors is constructed. The speech samples in the corpus of tone errors are processed by the voiced-voiced adaptive fusion model to generate corresponding multi-level tone feature sets. The feature sets, along with the speech samples and labels, are stored in the corpus. The corpus of tone errors is used to verify the accuracy and stability of the evaluation results.

[0010] Preferably, the multi-level tone feature set includes: Frame-level features include Mel frequency cepstral coefficients, fundamental frequency F0 curve, and its first and second derivatives; Segment-level features include the fundamental frequency F0 curve fitting parameters of the syllable, syllable duration, mean fundamental frequency, and mean energy. Articulation features are extracted from Mel frequency cepstral coefficients using a pre-trained deep neural network to characterize vocal tract articulation actions; Prosodic context features are composed of acoustic information of adjacent syllables and characterize coarticulation and tone sandhi.

[0011] Preferably, the speech signal is aligned with the standard reference speech at the vowel level in the time domain, and the tone shape feature vector and tone domain feature vector are extracted and decoupled, including: Using a dynamic time warping algorithm, the speech signal is time-domain aligned with the standard reference speech at the boundaries of initials and finals to determine the speech segments to be compared; tone shape feature vectors are extracted, including normalized start pitch, end pitch, lowest pitch, highest pitch and their relative time positions, average slope of rising and falling segments and polynomial fitting coefficients; tone domain feature vectors are extracted, including fundamental frequency mean, fundamental frequency dynamic range and energy mean.

[0012] Preferably, the above-mentioned dual-channel evaluation path is established, and the tone shape distance and tone range distance of the tone are calculated hierarchically. The tone shape distance and tone range distance are then weighted and fused according to preset tone shape weights and tone range weights to generate a final evaluation score, including: In the first evaluation channel, the Euclidean distance between the learner's tone shape feature vector and the shape feature vectors of all standard tone models is calculated, and the distance to the target tone is taken as the tone shape distance. In the second evaluation channel, based on the result of the first channel: if the tone shape distance is lower than a preset tone shape similarity threshold, the difference between the learner's tone domain feature vector and the standard reference tone domain feature vector is calculated as the tone domain distance; if the tone shape distance is higher than the threshold, a preset penalty value is assigned to the tone domain distance. Based on the principle of perceived importance of tone shape over tone domain in Chinese linguistics, the output distances of the two channels are weighted and summed using tone shape weight and tone domain weight to generate the total tone error distance, which is mapped to the final evaluation score. The tone shape weight and tone domain weight are preset by statistical analysis of the tone error corpus and in combination with the principle of perceived importance of tone shape over tone domain in Chinese linguistics, and the tone shape weight is higher than the tone domain weight.

[0013] Preferably, the personalized diagnostic feedback information includes: Based on the values ​​of pitch shape distance and pitch range distance, feedback information is generated on the learning platform system. When the pitch shape distance is greater than a preset threshold, the system provides feedback on pitch shape errors and differences between the target pitch and the actual pitch. When the pitch shape distance is less than a preset threshold and the pitch range distance is greater than a preset threshold, the system provides suggestions that the pitch shape is correct but the pitch range fluctuation is insufficient. The system provides real-time pitch curve visualization, superimposing and comparing the learner's fundamental frequency F0 curve with the standard pitch fundamental frequency F0 curve. Based on the values ​​of pitch shape distance and pitch range distance, targeted practice content is recommended, including pitch shape identification exercises and exaggerated pitch fluctuation imitation exercises.

[0014] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention decomposes and decouples tone information into two evaluation dimensions: "tone shape" and "tone range," thereby achieving precise quantification and attribution of pronunciation deviations, improving the accuracy and depth of diagnosis, and overcoming the limitations of existing technologies that can only provide fuzzy, single similarity scores.

[0015] 2. This invention establishes a dual-channel assessment path and diagnosis-practice mapping mechanism based on decoupling dimensions, which enables the direct transformation of quantitative assessment results into actionable personalized learning guidance. This makes the feedback no longer a general "right" or "wrong", but a specific and actionable improvement suggestion, thereby improving the efficiency of Chinese language teaching and self-study.

[0016] 3. This invention constructs an adaptive fusion model for voiced and unvoiced sounds, dynamically integrating fundamental frequency-related features and fundamental frequency-independent features such as duct parameters. This achieves continuity and stability of feature extraction in unvoiced, weak, and fundamental frequency-free scenarios, significantly enhancing the robustness of the evaluation and its applicability to real-world environments. It also solves the problem of traditional methods over-relying on high-quality fundamental frequency signals.

[0017] 4. This invention constructs and applies a corpus of tone errors specifically labeled with typical error patterns of Chinese learners (such as "insufficient tone sinking in the third tone"), enabling the assessment model to identify and quantify these specific, non-standard pronunciation patterns, making the assessment results closer to real teaching scenarios, and improving the diagnostic accuracy of common difficulties for learners. Attached Figure Description

[0018] Figure 1 This is a flowchart of a method for assessing and improving the tone of Chinese learners according to the present invention; Figure 2 This is a flowchart illustrating the internal logic of the turbidity-clarity adaptive fusion model in an embodiment of the present invention. Figure 3 This is a flowchart of the dual-channel conditional evaluation logic according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the diagnosis-practice mapping mechanism in an embodiment of the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions and advantages of the present invention clearer, the following will further elaborate on the present invention by referring to the accompanying drawings and in combination with specific embodiments. It should be understood that the specific embodiments described herein are only preferred examples of the present invention, aiming to explain the technical principles of the present invention rather than limiting the protection scope of the present invention in any form. Based on the content disclosed herein, any non-substantive modification, equivalent replacement or improvement made by any person skilled in the art within the spirit and principles of the present invention shall be regarded as falling within the protection scope of the present invention.

[0020] Please refer to Figures 1 to 4 , the present invention provides a tone evaluation and improvement method for Chinese learners.

[0021] Embodiment 1: Referring to Figure 1 , a tone evaluation and improvement method for Chinese learners in this embodiment includes: Preprocessing the speech signal input by the Chinese learner, extracting the fundamental frequency F0 curve and vocal tract parameters of the speech signal, and constructing a voiceless-voiced adaptive fusion model, where the voiceless-voiced adaptive fusion model fuses the fundamental frequency-related features generated by the fundamental frequency F0 curve and the fundamental frequency-unrelated features generated by the vocal tract parameters; Constructing a tone error corpus, classifying and labeling typical random error patterns, and generating a multi-level tone feature set based on the voiceless-voiced adaptive fusion model; Based on the multi-level tone feature set, performing time-domain alignment of the speech signal and the standard reference speech at the initial consonant and final vowel levels, and extracting and decoupling the tone shape feature vector and tone domain feature vector; <00,00076>Based on the decoupled tone shape feature vector and tone domain feature vector, establishing a dual-channel evaluation path, calculating the tone pattern distance and tone domain distance of the tone layer by layer, and performing weighted fusion on the tone pattern distance and tone domain distance according to the preset tone pattern weight and tone domain weight to generate a final evaluation score; Generating personalized diagnostic feedback information for Chinese learners based on the calculated values of the tone pattern distance and tone domain distance respectively.

[0022] Further, the preprocessing of the speech signal input by the Chinese learner includes: When a Chinese learner inputs a piece of speech through a microphone, such as a target word "nǐ hǎo", this method first preprocesses the input speech.

[0023] Digital sampling: Sampling the analog speech signal at a sampling rate of 16 kHz and a quantization precision of  16 bits to ensure that sufficient speech details are captured; Noise suppression: Algorithms such as spectral subtraction or wavelet transform are used to filter out environmental background noise; Amplitude normalization: The amplitude of the sampled signal is linearly scaled to the range of [-1.0, 1.0] to eliminate the influence of different recording volumes; Pre-emphasis filtering: The signal is processed by a first-order high-pass filter to enhance the energy of the high-frequency part, compensate for the natural attenuation of high-frequency energy during the sound production process, and facilitate subsequent feature extraction.

[0024] This embodiment reduces interference from non-linguistic factors such as recording equipment, environment, and volume through standardized preprocessing, ensuring the stability and reliability of subsequent acoustic feature analysis.

[0025] Further, preferably, the construction of the turbidity-clarity adaptive fusion model, which fuses the fundamental frequency F0 curve to generate fundamental frequency-related features and the duct parameters to generate fundamental frequency-independent features, includes: This embodiment employs a fundamental frequency extraction algorithm, specifically the YIN algorithm, to extract the original fundamental frequency (F0) curve from the preprocessed speech signal. First, this method uses a harmonic correction module to correct common octave errors (misclassifying the actual fundamental frequency as a second harmonic) and half-frequency errors (misclassifying the second harmonic as the fundamental frequency) in the original F0 curve by detecting the energy of the harmonic structure. A cubic spline interpolation algorithm is then applied to the corrected F0 curve. This algorithm mathematically guarantees the continuity of the first and second derivatives at the interpolation points, generating a smooth and natural transition curve in the F0 interruption region caused by voiceless consonants (such as the initials / s / , / f / , / ch / ). Finally, a fundamental frequency-related feature that exists continuously throughout the entire syllable duration is generated as the first input.

[0026] While performing the above steps, the same speech signal is processed in parallel. The signal is divided into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. For each frame, the 12th-order linear prediction coefficients are calculated using the autocorrelation method. The energy of the prediction error signal of this linear prediction model is the linear prediction model gain, and its physical meaning is related to the energy of the excitation source of the speech. The linear prediction model gains of each frame are constructed into a continuous sequence. This sequence is independent of the presence or absence of the fundamental frequency and can effectively reflect the energy changes of the speech signal. This sequence serves as the fundamental frequency-independent feature of the second input.

[0027] For each speech analysis frame, by calculating underlying acoustic parameters such as its short-time energy, zero-crossing rate, and spectral flatness, the unvoiced probability of this frame is jointly estimated. This probability is a continuous value between 0 and 1, and this unvoiced probability value is used as the dynamic fusion weight. Among them, the method for jointly estimating the unvoiced probability of this frame is specifically as follows: Use a logistic regression classifier that has been pre-trained on speech data annotated with unvoiced / voiced information. This classifier takes the short-time energy, zero-crossing rate, and spectral flatness as input features, and its output is the unvoiced probability of this frame.

[0028] Finally, apply this weight to perform a frame-level weighted linear combination on the two feature streams. Specifically, the final fused feature of each frame is calculated by the following deterministic formula: Fused feature = (unvoiced probability × F0-related feature) + ((1 - unvoiced probability) × F0-unrelated feature) Taking the character "去 (qù)" as an example, its initial consonant / q / is voiceless. At the beginning stage of pronunciation, the calculated unvoiced probability is close to 0. At this time, the value of the fused feature is mainly determined by the F0-unrelated feature; when the pronunciation transitions to the final sound / ü / , which is voiced, the calculated unvoiced probability rises rapidly and approaches 1. At this time, the value of the fused feature is mainly determined by the F0-related feature (F0). The detailed internal logic process of this model is as Figure 2 shown.

[0029] To further improve the fusion quality, when calculating the dynamic fusion weight, this method also introduces the consideration of the F0 quality confidence. Specifically, first, calculate the F0 quality confidence of each frame in the F0-related feature. This confidence characterizes the periodic intensity of the F0 signal of this frame and the clarity of the harmonic structure; based on the unvoiced probability and the F0 quality confidence, generate the dynamic fusion weight; among them, when the F0 quality confidence is lower than the preset quality threshold, this frame is voiceless. This method reduces the proportion of the F0-related feature in the weighted fusion, reduces the interference of low-quality F0 signals on the fused feature, and significantly enhances the robustness of the unvoiced / voiced adaptive fusion model.

[0030] The unvoiced / voiced adaptive fusion model of this method fundamentally solves the problem of poor robustness caused by the over-dependence on F0 in existing methods by parallelly extracting and dynamically fusing F0-related and F0-unrelated features according to the physical characteristics of speech, ensuring the stability and reliability of this method when processing complex syllables including voiceless sounds and aspirated sounds. In addition, the model greatly improves the accuracy and smoothness of the F0-related feature through the correction and smoothing of the F0 curve, laying a high-quality data foundation for subsequent tone shape analysis.

[0031] Furthermore, the construction of the tone error corpus includes: Systematically collect speech data for non-native Chinese learners. To ensure the broad representativeness of the corpus, the mother tongue backgrounds of the collection objects cover a variety of typical languages, such as but not limited to English, Japanese, Korean, French, and Russian. The collected speech content comprehensively covers all combinations of initials, finals, and tones in Mandarin Chinese, specifically including: Single-syllable characters: single characters with all four tones and the light tone, such as "妈(mā)", "麻(má)", "马(mǎ)", "骂(mà)", "吗(ma)"; Two-syllable words: covering all theoretically possible tone combinations, such as "飞机(fēi jī)", "学习(xué xí)", "你好(nǐ hǎo)", "再见(zài jiàn)"; Specific short sentences: sentences containing easily confused tone combinations and typical tone sandhi phenomena (such as the tone sandhi of "一" and "不", and the tone sandhi of three consecutive third tones).

[0032] Technicians with a professional background in teaching Chinese as a foreign language or phonetics knowledge independently cross-annotate each piece of collected speech. The annotation content not only includes standard Chinese pinyin and correct tones, but also systematically classifies and annotates pronunciation errors. This classification system summarizes the random errors of learners into several typical patterns, for example: Tone pattern confusion error: such as mispronouncing the second tone as the third tone. The objective judgment criterion is that the fundamental frequency (F0) curve of the actual pronunciation shows a complete "descending first and then ascending" shape; Tone range execution error: such as insufficient sinking of the third tone. The objective judgment criterion is that after normalizing the F0 curve of the pronunciation for the speaker, the value of the actual valley point is higher than the 30% quantile position of the speaker's personal pitch range from low to high; such as "the fourth tone descends gently, constructing a tone error corpus", the objective judgment criterion is that the absolute value of the average descending slope from the inflection point to the end point of the F0 curve is less than 50% of the average slope of all correct fourth tone samples of the speaker; The annotation process adopts a consistency test to ensure the objectivity and reliability of the annotation results.

[0033] For each speech sample in the corpus, it is pre-processed through a浊清自适应融合模型 (turbid-clear adaptive fusion model) to extract its multi-level tone feature set; finally, each record in the corpus structurally contains at least four parts: the original speech signal, the manually annotated text and error labels, the corresponding standard reference pronunciation sample, and the pre-calculated acoustic feature set.

[0034] The corpus will verify the accuracy and stability of the evaluation method described in this invention. For example, all samples labeled as "insufficient tone drop" can be selected from the corpus and input into the method of this invention in batches for evaluation. The accuracy of the diagnosis on this specific error type can be quantified by statistically analyzing the proportion of diagnoses that give "correct tone pattern but incorrect tone range".

[0035] This embodiment constructs a tone error corpus, which systematically and structurally classifies and quantifies the fuzzy and random tone errors of non-native language learners, providing a high-quality validation dataset for the field of tone teaching and assessment. At the same time, this corpus provides a solid and quantifiable validation foundation for the effectiveness of the assessment method.

[0036] Furthermore, the multi-level tone feature set includes: First level: Frame-level features. This level of features can capture the instantaneous, low-level acoustic characteristics of the speech signal. After dividing the signal into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds, the following features are extracted from each frame: 13-Vimel frequency cepstral coefficients (MFCC): used to characterize the spectral envelope of the signal in this frame, i.e. the timbre characteristics of the sound. Fundamental frequency F0 value: The fused F0 value in this frame, output by the turbidity-clarity adaptive fusion model; First and second derivatives: Calculate the difference and the difference coefficient of the above MFCC and F0 characteristics to characterize their dynamic change trend; The final frame-level feature vector is (13+1)×3=42 dimensions.

[0037] The second level: segmental level features, which characterize the overall acoustic pattern of a complete syllable (such as "nǐ" or "hǎo"), specifically include: F0 curve fitting parameters: The F0 curve within the syllable is fitted using a third-order polynomial with the least squares method, and the four coefficients of the polynomial are extracted as features; Duration feature: The duration of the syllable (in milliseconds); Statistical characteristics: mean F0, standard deviation of F0, and mean energy of all frames within the syllable; These features together describe the macroscopic tonal profile and energy distribution of a syllable.

[0038] The third level: pronunciation features. In this embodiment, it is implemented by a deep neural network (DNN) with a bottleneck layer that has been pre-trained on a large amount of general speech data. The frame-level MFCC sequence is input into this DNN, and the output of its middle layer (i.e., the bottleneck layer) is extracted. The dimension of this bottleneck layer is usually low (e.g., 40 dimensions), and its output has been proven to be able to effectively represent more compact and robust acoustic information that is highly correlated with specific pronunciation actions.

[0039] The fourth level: prosodic context features. When analyzing the tone of the current target syllable, this method will simultaneously extract the segment-level features of the immediately preceding and following syllables and concatenate these features with the features of the current syllable. For example, when analyzing the syllable "nǐ" in the word "nǐ hǎo", the segment-level features of the following "hǎo" will be introduced as prosodic context features, which is crucial for accurately judging whether the tone of "nǐ" has changed from the third tone to the second tone.

[0040] In this embodiment, by constructing a multi-level tone feature set with rich information and comprehensive dimensions, and by integrating information from different levels, it simultaneously captures the micro-dynamics (frame level), macro-profile (segment level), internal pronunciation actions (pronunciation features), and context dependence (prosodic context) of tones, making the description of tones more complete and accurate. Secondly, the introduction of pronunciation features and prosodic context features enables the model to better handle complex tone-changing phenomena and has a stronger tolerance for local errors in F0 extraction.

[0041] Furthermore, the speech signal and the standard reference speech are aligned in the time domain at the initial consonant and final consonant levels, and the tone shape feature vector and the tone domain feature vector are extracted and decoupled, including: The speech signal input by the learner is processed through a pre-set speech recognition module or a forced alignment algorithm to accurately determine the start and end time points of the initial consonant part and the final consonant part of the target syllable therein, that is, the initial consonant-final consonant boundary, and accurately place the focus of analysis on the main part carrying tone information, that is, the final consonant segment. Subsequently, the dynamic time warping algorithm (DTW) is applied to non-linearly align the feature sequence of the final consonant segment pronounced by the learner with the feature sequence of the corresponding final consonant segment of the standard reference pronunciation stored in the corpus. The DTW algorithm finds a path with the minimum energy dissipation to establish the best correspondence between each feature frame of the two time series.

[0042] After obtaining the aligned and duration-normalized final consonant segments, two feature vectors designed to respectively represent different perceptual dimensions of tones are extracted in parallel, that is, the tone shape feature vector and the tone domain feature vector.

[0043] To extract tone shape feature vectors, this method first traverses the fundamental frequency (F0) curves within the aligned vowel segments to find the fundamental frequency values ​​at the start, end, highest, and lowest points. Next, the fundamental frequency values ​​at these four key points are normalized. A preferred normalization method is T-Norm, which involves subtracting the speaker's average fundamental frequency over a longer speech segment (e.g., a complete practice session) from the current fundamental frequency value, and then dividing the difference by the speaker's fundamental frequency standard deviation within the same segment. The normalized value obtained through this process... The shape feature vector reflects only the relative pitch of the key point with respect to the speaker's personal vocal range, and no longer includes its absolute pitch information; the shape feature vector also includes other parameters describing the shape, such as the time points of the highest and lowest points, expressed as the percentage of the total vowel duration; the average slope of the main rising and falling segments, used to describe the speed of tone change; and four polynomial coefficients obtained by least-squares fitting of the entire F0 curve with a third-order polynomial, which can characterize the curvature of the entire curve in a compact way.

[0044] Simultaneously, this method also directly extracts and calculates the absolute physical quantities within the aligned vowel segments, without any normalization. These quantities specifically include: the arithmetic mean of the fundamental frequency values ​​of all voiced frames within the syllable, i.e., the fundamental frequency mean, measured in Hertz (Hz); the difference between the highest and lowest fundamental frequency values ​​within the syllable, i.e., the fundamental frequency dynamic range, also measured in Hertz (Hz), used to characterize the actual amplitude of pitch fluctuations; and the arithmetic mean of the energy values ​​of all frames within the syllable, i.e., the energy mean, used to characterize the loudness of the sound.

[0045] Furthermore, to assess the smoothness of pronunciation, this method introduces a third dimension in addition to extracting tone shape feature vectors. Specifically, after fitting the F0 curve with a third-order polynomial, the method calculates the root mean square of the sum of squares of the differences between each point on the original F0 curve and the corresponding point on the fitted curve. This value is defined as the fitting residual and directly used as the "smoothness distance." A smooth, natural pronunciation will have an F0 curve that closely matches the smooth polynomial curve, resulting in a small fitting residual (smoothness distance). Conversely, a shaky, unstable pronunciation will have an F0 curve that deviates from the fitted curve, leading to a large fitting residual. By introducing a novel "smoothness" evaluation dimension related to pronunciation stability, this method can quantify subtle defects such as undetectable pronunciation shakiness, thereby greatly improving the accuracy of the assessment and the ability to perceive the true quality of pronunciation.

[0046] This embodiment extracts "shape" features by normalizing the fundamental frequency and extracts "tonality" features by retaining its absolute value. It successfully separates a complex, coupled acoustic signal into two dimensions that are significantly different in perception and can be independently evaluated: the shape dimension, which mainly reflects the tone contour, and the tone dimension, which mainly reflects the pitch range. This makes it possible to conduct independent and accurate quantitative evaluations of "whether the tone is correct" and "whether the pitch range is full," which is a fundamental prerequisite for achieving accurate diagnosis and feedback.

[0047] Furthermore, the aforementioned dual-channel evaluation path calculates the tone pattern distance and tone range distance in layers, and weights and fuses the tone pattern distance and tone range distance according to preset tone pattern weights and tone range weights to generate a final evaluation score, including: In the first evaluation channel, the correctness of the learner's pronunciation tonal contour morphology is assessed. This method calculates the Euclidean distance between the learner's pronunciation tonal shape feature vector and the shape feature vectors of four pre-established standard tone models (obtained by taking the arithmetic mean of shape vectors from a large number of standard pronunciations in the tone error corpus). The method uses the calculated distance to the target tone model (e.g., the standard fourth tone model if the target tone is the fourth tone) as the final tone distance.

[0048] Subsequently, in the second evaluation channel, the accuracy of the physical implementation of tone is evaluated, and its calculation process is based on the evaluation results of the first channel. If the tone distance calculated in the previous step is lower than a preset tone similarity threshold, the method for determining the preset tone similarity threshold includes: calculating the intra-class distance distribution of all positive samples (i.e., correct pronunciation) and the inter-class distance distribution of negative samples (i.e., incorrect pronunciation) on the tone error corpus, and selecting the distance value that minimizes the classification error rate of the two classes of samples as the threshold, for example, by calculating the equal error rate point. This method determines that the learner's tone is basically correct. Under this condition, the method further calculates the normalized difference between the learner's tone domain feature vector and the tone domain feature vector of the standard reference pronunciation, and uses this as the tone domain distance; conversely, if the tone distance is higher than the threshold, the method determines that the learner has fundamentally confused tone. At this time, the accuracy of the tone domain is no longer meaningful for evaluation, and the method directly assigns a preset penalty value, such as 1.0, to the tone domain distance, which is sufficient to dominate the evaluation result.

[0049] Finally, to generate the total distance of tone errors, the method applies a preset tone type weight to the tone type distance and a preset tone range weight to the tone range distance, and then sums the two weighted distances. The tone type weight and tone range weight are not fixed but adaptively adjustable. Specifically, a learner proficiency profile is created for each Chinese learner, which is dynamically updated based on the learner's historical assessment data, considering the average tone type distance and average tone range distance. Before weighted fusion, the tone type weight and tone range weight are adaptively adjusted according to the learner proficiency profile. Specifically, when the profile indicates the learner is at a beginner level, the method automatically increases the tone type weight; when the profile indicates the learner is at an advanced level, the method automatically increases the tone range weight. The specific rules for adaptive adjustment are as follows: if the historical average tuning distance is greater than 1.5 times the historical average tuning range distance, the tuning weight will be temporarily increased by 0.1 and the tuning range weight will be decreased by 0.1; conversely, if the historical average tuning range distance is greater than 1.5 times the historical average tuning distance, the tuning range weight will be temporarily increased by 0.1 and the tuning weight will be decreased by 0.1.

[0050] This method upgrades the static assessment system into a personalized coaching model that can dynamically adjust the assessment focus based on the learner's proficiency, realizing intelligent assessment that is tailored to individual needs and significantly improving the relevance and efficiency of learning.

[0051] The detailed logic flow of this dual-channel evaluation path is as follows: Figure 3 As shown, the weights are pre-set based on statistical analysis of the tone error corpus, which found that tone pattern errors affect speech intelligibility more severely than tone range errors, and in accordance with this linguistic principle. For example, a typical setting is a tone pattern weight of 0.7 and a tone range weight of 0.3. Finally, the method maps the total error distance to an intuitive score of 0 to 100 using a non-linear function, such as subtracting the total distance from 1 and then multiplying by 100.

[0052] Before generating the final evaluation score, this method also considers the "smoothness distance" calculated above. If the smoothness distance is greater than a preset threshold (e.g., 0.5), it means that the learner's pronunciation has obvious instability or trembling. In this case, regardless of the results of the tone pattern and tone range evaluation, the final score will be deducted an additional fixed penalty score, such as 10 points.

[0053] This embodiment establishes a dual-channel, conditional assessment path that simulates the diagnostic logic of a language expert, assessing tone pattern first and then tone range, thus achieving a high degree of interpretability in the assessment process and results. This method can clearly distinguish the nature and severity of errors, making the assessment score no longer a vague number, but a transparent result that comprehensively quantifies the learner's pronunciation problems across two orthogonal dimensions.

[0054] Furthermore, the personalized diagnostic feedback information includes: First, the system generates diagnostic feedback information in text form. This method accesses a pre-set feedback template library and selects and fills the corresponding template based on whether the tone distance and pitch range distance exceed their respective preset thresholds. If the calculated tone distance is greater than its threshold, the system will select a template indicating an incorrect tone shape. The content of this template will explain the specific differences between the learner's pronunciation tone and the target tone, such as pointing out that a rising tone was mistakenly pronounced as a flat tone. Conversely, if the tone distance is less than its threshold but the pitch range distance is greater than its threshold, the system will select a template indicating an incorrect pitch range execution. The content of this template will explain that the learner's pronunciation shape is correct, but the amplitude or range of its pitch fluctuations is insufficient.

[0055] Secondly, the system provides real-time pitch curve visualization and comparison. On the user interface, the system overlays and compares the learner's fundamental frequency (F0) curve with the fundamental frequency (F0) curve of the standard reference speech. This method renders two graphical fundamental frequency curves on the user interface. The first curve is drawn based on the fundamental frequency data of the standard reference speech and is assigned a first visual attribute, such as being displayed as a solid green line, representing the correct target. The second curve is drawn in real-time based on the fundamental frequency data extracted from the learner's current pronunciation and is assigned a second visual attribute, such as being displayed as a dashed red line, representing the actual pronunciation. The overlay display of the two curves in the same coordinate system allows learners to intuitively and instantly compare the differences between their pronunciation and the standard pronunciation in terms of pitch variation trajectory.

[0056] To recommend targeted practice content, this recommendation process is also performed based on the values ​​of tone shape distance and tone range distance. This embodiment implements this function through a diagnostic-practice mapping mechanism, which can be implemented as a lookup table or a set of conditional rules. It takes the contribution of tone shape distance and tone range distance to the total distance of tone errors as input. If the mechanism determines that tone shape distance is the primary contributor to errors, it automatically retrieves and recommends exercises related to "tone shape discrimination" from a preset practice content library; if it determines that tone range distance is the primary contributor to errors, it accordingly retrieves and recommends exercises related to "exaggerated pitch variation imitation." The detailed logical flow of this diagnostic-practice mapping mechanism is as follows: Figure 4 As shown, this mechanism precisely maps the quantified evaluation distance to specific practice recommendations through a cascaded decision-making process.

[0057] The specific logical steps of this mechanism are as follows: Execution of Rule 1: The system first determines whether the tone distance is greater than the preset threshold A, which represents the critical value at which the tone is fundamentally confused.

[0058] If so, i.e., the tone distance is greater than the threshold A, then the incorrect attribution is determined to be "tone confusion", and the system will directly output and recommend practice content related to "tone shape discrimination".

[0059] Execution of Rule 2: If the conditions of Rule 1 are not met (i.e. the pitch pattern is basically correct), the process will proceed to the next level of judgment. The system will then judge whether the pitch range distance is greater than the preset threshold B. Threshold B represents the critical value for whether the pitch fluctuation is sufficient.

[0060] If so, i.e., the pitch range distance is greater than the threshold B, then the error attribution is determined to be "insufficient pitch range execution", and the system will output and recommend practice content related to "exaggerated imitation of pitch fluctuations".

[0061] Execution of Rule 3: If the conditions of Rule 2 are not met (i.e., the tone domain is also basically qualified), the process continues to the final level of judgment. The system judges whether the smoothness distance is greater than the preset threshold C. The threshold C represents the critical value for whether the pronunciation is stable and smooth.

[0062] If so, i.e., the smoothness distance > threshold C, then the error attribution is determined to be "poor pronunciation stability", and the system will output and recommend practice content related to "stable maintenance of long vowels".

[0063] Final judgment: If none of the "yes" paths of the above rules are triggered, the system will ultimately judge the pronunciation as "basic standard" and recommend "comprehensive consolidation exercises".

[0064] This embodiment establishes a "diagnosis-practice" mapping mechanism, which attributes errors based on quantified tone distance and tone range distance, thereby transforming assessment results into personalized and actionable guidance. This method can not only clearly identify the learner's error dimensions (tone or tone range), but also provide a three-in-one solution including visual comparison and targeted practice recommendations, thus forming an efficient learning loop and significantly improving learners' self-correction ability and learning efficiency.

[0065] This embodiment constructs a complete technical process integrating voiced / voiced adaptive fusion, a tone error corpus, feature decoupling, dual-channel evaluation, and diagnostic feedback. This results in a robust, accurate, and highly interpretable method for evaluating and improving Chinese tone patterns. This method not only overcomes the poor robustness of existing methods when dealing with complex speech, but also solves the fundamental problem of traditional evaluation methods' inability to accurately attribute errors by quantitatively evaluating tone errors as two independent dimensions: "shape" and "tone range." Ultimately, it provides learners with actionable and personalized learning guidance, significantly improving the efficiency of teaching and learning Chinese tone patterns.

[0066] Example 2: One specific embodiment of the present invention is applied in the International Chinese Language Teaching Center of a university to learners whose native language is a tonal language (such as Thai or Vietnamese). This embodiment aims to solve the problem of specific pronunciation errors caused by the negative transfer of the native language's tonal system to these learners.

[0067] First, a targeted corpus of tone errors is constructed. Specifically, a large amount of Chinese pronunciation data of learners with this background is collected and recorded, covering all initials, finals and tone combinations. Then, professionals with backgrounds in teaching Chinese as a foreign language and phonetics cross-label the collected speech. The labeling content includes not only the correct pinyin and tone, but also the classification of typical error patterns, such as: "the first tone (55) is pronounced as a middle tone (33)", "the fourth tone (falling tone) has insufficient falling momentum", and "the third tone (rising tone) has tone sandhi error". Finally, the labeled speech samples, text, error labels and corresponding standard pronunciation samples are stored in the database to form a corpus specifically for model fine-tuning and evaluation verification.

[0068] When a learner records the target word "teacher (lǎo shī)" through a client, the method of this invention begins to execute. The system first preprocesses the input speech signal by digital sampling, noise suppression, and amplitude normalization. Based on this, the system extracts a stable and reliable fundamental frequency (F0) curve from the preprocessed signal using a built-in voiced / voiced adaptive fusion model. Next, the system uses a vowel boundary recognition algorithm to focus the analysis on the vowel segments of "lǎo" and "shī" that carry tone information, and extracts two sets of independent feature vectors in parallel: one is the tone shape feature vector, representing the relative contour of the tone, obtained after normalizing the fundamental frequency curve; the other is the unnormalized tone range feature vector, representing the absolute physical properties of pitch.

[0069] The system then initiates a dual-channel evaluation path. In the first evaluation channel, the system calculates the Euclidean distance between the tone shape feature vector of the learner's pronunciation of "lǎo" and the shape feature vector of the standard third tone model. This distance is defined as the "tone shape distance." The calculation in the second evaluation channel is based on the result of the first channel. The system will determine whether the calculated "tone shape distance" is lower than a preset tone shape similarity threshold. If it is lower than the threshold, it indicates that the tone shape of the learner's pronunciation is basically correct. The system will then continue to calculate the difference between the tone feature vector of the learner's pronunciation and the tone feature vector of the standard reference pronunciation, and define this difference as the "tone shape distance." Conversely, if the "tone shape distance" is higher than the threshold, it indicates that the learner has suffered from fundamental tone shape confusion, and the system will directly assign a preset penalty value to the "tone shape distance." Finally, the system uses a weighted sum of "tone distance" and "tone range distance" based on a pre-set weight according to linguistic principles (tone type weight is 0.7, tone range weight is 0.3) to generate a total distance of tone error, which is then mapped to an intuitive evaluation score from 0 to 100.

[0070] After the assessment is completed, the system will convert the results into personalized diagnostic and practice recommendations. If the assessment results show that the "tone shape distance" is small but the "tone range distance" is large, the system will select and fill the built-in feedback template library to generate text feedback such as "The tone shape of the third tone (falling then rising) is basically correct, but the pitch range fluctuation is insufficient, and the amplitude of the sound sinking and rising is not enough".

[0071] Meanwhile, on the user interface, the system will overlay the learner's fundamental frequency F0 curve (represented by a red dashed line) with the standard pronunciation's fundamental frequency F0 curve (represented by a green solid line), allowing learners to intuitively see that the turning point of their curve is significantly smaller than that of the standard curve. Based on the judgment that "tonal range distance" is the main contributor to errors, the system will automatically retrieve and recommend exercises related to "exaggerated imitation of pitch fluctuations" from the exercise content library, thereby guiding learners to make targeted improvements.

[0072] Example 3: Another specific embodiment of the present invention is applied to the teaching of learners whose native language is a non-tonal language (such as English or French) in the same teaching environment. This embodiment aims to solve the problem that such learners have difficulty distinguishing and pronouncing the four basic tone shapes due to a lack of tone concept.

[0073] In this embodiment, to address the fundamental challenges faced by learners whose native language is a non-tonal language, the method of the present invention first undergoes adaptive adjustment. Based on the user's language background label, before the assessment begins, the system adjusts the weights of the dual-channel assessment, significantly increasing the "tone pattern weight" representing the correctness of the tone contour to 0.85, while correspondingly reducing the "tone range weight" representing the absolute pitch range to 0.15, to ensure that the core of the assessment focuses on the correctness of the tone shape. The construction of the corresponding tone error corpus also focuses on annotating fundamental tone pattern errors of the "tone confusion" type, such as mispronouncing the second tone (rising tone) as the fourth tone (falling tone).

[0074] When a learner records the target word "huí lái", if they pronounce both second tones that should rise as flat tones, the assessment process is as follows: After the system completes preprocessing and feature extraction, it enters the dual-channel assessment path. In the first assessment channel, the system calculates the "tone pattern distance" between the shape feature vector of the learner's flat tone and the shape feature vector of the standard second tone (rising tone) model. Since the shapes do not match at all, this distance will be very large, far exceeding the preset similarity threshold. Therefore, when entering the second assessment channel, the system determines that the learner has made a fundamental tone pattern error and directly assigns a penalty value to the "tone range distance". In the final weighted fusion step, due to the huge "tone pattern distance" multiplied by a weight as high as 0.85, the generated assessment score will be very low, thus accurately reflecting the fundamental problem of the learner in tone mastery.

[0075] Based on the core assessment result that the "tone pattern distance" is much greater than the threshold, the system will generate diagnostic text such as "The tone shape is incorrect. Your pronunciation is flat, while the target tones 'huí' and 'lái' both need to rise continuously". In the visual feedback, two flat red curves of the learner will be presented on the interface in sharp contrast to the two rising green curves in the standard pitch; based on the judgment that "tone pattern confusion" is the main error type, the system will automatically recommend the most basic "tone shape discrimination" exercises, such as gamified exercises of identifying graphics (flat "-", rising "´", falling-rising "ˇ", falling "`") by listening, to help the learner fundamentally establish the connection between the tone shape and the auditory perception, laying a solid foundation for subsequent pronunciation imitation and correction.

[0076] Although specific examples of the present invention have been shown, those of ordinary skill in the art should understand that various adjustments, modifications, substitutions, and improvements to these examples are possible without departing from the core principles and spirit of the present invention. The protection scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for assessing and improving the tone of Chinese learners, characterized in that, Includes the following steps: The speech signal input by Chinese learners is preprocessed to extract the fundamental frequency F0 curve and vocal tract parameters of the speech signal, and a voiced-unvoiced adaptive fusion model is constructed. The voiced-unvoiced adaptive fusion model fuses the fundamental frequency-related features generated by the fundamental frequency F0 curve and the fundamental frequency-independent features generated by the vocal tract parameters. A corpus of tone errors is constructed, typical random error patterns are classified and labeled, and a multi-level tone feature set is generated based on the voiced-voiced adaptive fusion model. Based on the multi-level tone feature set, the speech signal is aligned with the standard reference speech at the vowel level in the time domain, and the tone shape feature vector and tone domain feature vector are extracted and decoupled. Based on the decoupled tone shape feature vector and tone domain feature vector, a dual-channel evaluation path is established, the tone shape distance and tone domain distance of the tone are calculated in layers, and the tone shape distance and tone domain distance are weighted and fused according to the preset tone shape weight and tone domain weight to generate the final evaluation score. Based on the calculated values ​​of tone distance and tone range distance, personalized diagnostic feedback information is generated for Chinese learners.

2. The method for tone assessment and improvement for Chinese learners according to claim 1, characterized in that, The preprocessing of the speech signal input by the Chinese learner includes: digitally sampling the speech signal, noise suppression, amplitude normalization, and pre-emphasis filtering to generate a standardized speech signal.

3. The method for assessing and improving tone in Chinese learners according to claim 1, characterized in that, The construction of the voicing-clarification adaptive fusion model, which fuses the fundamental frequency-related features generated from the fundamental frequency F0 curve and the fundamental frequency-independent features generated from the duct parameters, includes: The fundamental frequency F0 curve is extracted using a fundamental frequency extraction algorithm, and harmonic correction and spline interpolation are performed to generate fundamental frequency-related features. In parallel, vocal tract parameters, including linear prediction model gain, are extracted to generate fundamental frequency-independent features. Based on the voiced characteristics of the speech segment, the voiced / unvoiced adaptive fusion model switches between and fuses fundamental frequency-related and fundamental frequency-independent features. Voiced segments use fundamental frequency-related features, unvoiced segments use fundamental frequency-independent features, and the transition segments between voiced and unvoiced segments undergo weighted fusion.

4. The method for assessing and improving tone in Chinese learners according to claim 1, characterized in that, The construction of the tone error corpus includes: Speech data from non-native Chinese learners is collected, including pronunciation samples of different tones. The collected data is labeled, including correct tones and typical random error patterns. The typical random error patterns include mispronouncing the second tone as the third tone, insufficient lowering of the third tone, and a flat lowering of the fourth tone. A tone error corpus is constructed. The speech samples in the tone error corpus are processed by the voiced / voiced adaptive fusion model to generate corresponding multi-level tone feature sets, which are then associated with the speech samples and labels and stored together in the tone error corpus. The tone error corpus is used to verify the accuracy and stability of the evaluation results.

5. The method for tone assessment and improvement for Chinese learners according to claim 1, characterized in that, The multi-level tone feature set includes: Frame-level features include Mel frequency cepstral coefficients, fundamental frequency F0 curve, and its first and second derivatives; Segment-level features include the fundamental frequency F0 curve fitting parameters of the syllable, syllable duration, mean fundamental frequency, and mean energy. Articulation features are extracted from Mel frequency cepstral coefficients using a pre-trained deep neural network to characterize vocal tract articulation actions; Prosodic context features are composed of acoustic information of adjacent syllables and characterize coarticulation and tone sandhi.

6. The method for assessing and improving tone in Chinese learners according to claim 1, characterized in that, The speech signal is aligned with the standard reference speech at the vowel level in the time domain, and the tone shape feature vector and tone domain feature vector are extracted and decoupled, including: Using a dynamic time warping algorithm, the speech signal is time-domain aligned with the standard reference speech at the boundaries of initials and finals to determine the speech segments to be compared; tone shape feature vectors are extracted, including normalized start pitch, end pitch, lowest pitch, highest pitch and their relative time positions, average slope of rising and falling segments and polynomial fitting coefficients; tone domain feature vectors are extracted, including fundamental frequency mean, fundamental frequency dynamic range and energy mean.

7. The method for assessing and improving tone patterns for Chinese learners according to claim 1, characterized in that, The above-mentioned dual-channel evaluation path calculates the tone shape distance and tone range distance of the tone layer by layer. Based on preset tone shape weights and tone range weights, the tone shape distance and tone range distance are weighted and fused to generate the final evaluation score, including: In the first evaluation channel, the Euclidean distance between the learner's tone shape feature vector and the shape feature vectors of all standard tone models is calculated, and the distance to the target tone is taken as the tone shape distance. In the second evaluation channel, based on the result of the first channel: if the tone shape distance is lower than a preset tone shape similarity threshold, the difference between the learner's tone domain feature vector and the standard reference tone domain feature vector is calculated as the tone domain distance; if the tone shape distance is higher than the threshold, a preset penalty value is assigned to the tone domain distance. Based on the principle of perceived importance of tone shape over tone domain in Chinese linguistics, the output distances of the two channels are weighted and summed using tone shape weight and tone domain weight to generate the total tone error distance, which is mapped to the final evaluation score. The tone shape weight and tone domain weight are preset by statistical analysis of the tone error corpus and in combination with the principle of perceived importance of tone shape over tone domain in Chinese linguistics, and the tone shape weight is higher than the tone domain weight.

8. The method for assessing and improving tone patterns for Chinese learners according to claim 1, characterized in that, The personalized diagnostic feedback information includes: Based on the values ​​of pitch shape distance and pitch range distance, feedback information is generated on the learning platform system. When the pitch shape distance is greater than a preset threshold, the system provides feedback on pitch shape errors and differences between the target pitch and the actual pitch. When the pitch shape distance is less than a preset threshold and the pitch range distance is greater than a preset threshold, the system provides suggestions that the pitch shape is correct but the pitch range fluctuation is insufficient. The system provides real-time pitch curve visualization, superimposing and comparing the learner's fundamental frequency F0 curve with the standard pitch fundamental frequency F0 curve. Based on the values ​​of pitch shape distance and pitch range distance, targeted practice content is recommended, including pitch shape identification exercises and exaggerated pitch fluctuation imitation exercises.

Images