Chinese tone correction system and method based on cross-modal adversarial generation

Abstract

The present invention relates to the field of speech processing technology, and in particular to a Chinese tone error correction system and method for cross-modal adversarial generation. The system includes a multi-channel tone spectrum-specific decomposition module for processing speech signals in parallel and extracting multi-scale tone features; an adaptive spectrum attention mask module for generating a spectrum enhancement mask and dynamically enhancing frequency regions related to tone features; a tone representation module for generating a tone representation vector through contrastive learning; a tone error detection module for analyzing the tone representation vector to determine the type and pattern of tone errors; and a multi-modal feedback module for generating comprehensive guidance feedback on tone pronunciation and tongue position based on the type and pattern of tone errors, thereby improving the accuracy of tone error detection. The system uses multi-channel tone spectrum-specific decomposition and adaptive spectrum attention mask technology to increase the tone error detection rate to 95% and the accuracy to 92%, and in particular significantly improves the detection accuracy of the third tone.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a Chinese tone correction system and method based on cross-modal adversarial generation, which are used to assist Southeast Asian learners in learning and correcting Chinese tones. Background Art

[0002] As a tonal language, Chinese tones are crucial for distinguishing word meanings. For non-native learners, especially those from Southeast Asia, mastering Chinese tones is a major learning challenge. Traditional speech recognition and error correction technologies primarily focus on accurate speech-to-text conversion, paying insufficient attention to the specificity of tones. Furthermore, error correction methods are limited, typically providing only text-based feedback and lacking intuitive and effective guidance.

[0003] In the prior art, a Chinese invention patent with the authorization publication number CN113012705B discloses a method and device for correcting speech text, which includes: using a word error detector to determine whether there is an abnormal unit in the speech text extracted from the speech data; if there is an abnormal unit, selecting a candidate correction word from the correction word reference library whose edit distance with the abnormal unit is less than the edit distance threshold, and replacing the abnormal unit with the candidate correction word. If there is no abnormal unit, it is determined that the speech text is correct. The speech text correction method and correction device provided by this application, based on the created word error detector, can avoid the situation where the speech recognition product cannot recognize the user's speech due to the user's personal speech pronunciation habits, thereby improving the user experience.

[0004] However, the above invention has the following problems: First, the accuracy of tone feature extraction is insufficient, and a unified spectrum analysis is used to process all acoustic features, ignoring the distribution characteristics of tone features in specific frequency bands; second, the error correction feedback is single, and most of them only provide simple feedback on acoustic modalities, lacking guidance at the level of pronunciation organs; third, it does not take into account the differences in learners' language backgrounds, such as the interference pattern of tonal languages ​​such as Vietnamese and Thai on Chinese tone learning; fourth, the response delay is high, affecting the learning experience; and fifth, it lacks the ability to adapt to personalized learning.

[0005] Therefore, there is an urgent need for a tone correction system that can accurately analyze tone features, provide multimodal feedback, and adapt to learners from different language backgrounds, especially one that is optimized for the difficulties faced by Southeast Asian learners in learning Chinese tones. Summary of the Invention

[0006] The purpose of the present invention is to provide a Chinese tone correction system and method for cross-modal adversarial generation, which solves the problems existing in the existing Chinese tone correction system through technologies such as multi-channel tone spectrum specialized decomposition, adaptive spectral attention masking, tone contrast learning representation, precise error pattern recognition, and comprehensive guidance feedback of tone pronunciation and tongue position movement.

[0007] The present invention proposes a Chinese tone correction system based on cross-modal adversarial generation, comprising:

[0008] A multi-channel tone spectrum decomposition module is used to divide the speech signal into low-frequency, mid-frequency, and high-frequency bands for parallel processing and extract multi-scale tone features;

[0009] an adaptive spectral attention mask module, connected to the multi-channel tone spectrum specialized decomposition module, configured to receive the multi-scale tone features sent by the multi-channel tone spectrum specialized decomposition module, generate a spectral enhancement mask based on the multi-scale tone features, and dynamically enhance frequency regions related to the tone features;

[0010] a tone representation module, connected to the adaptive spectral attention mask module, configured to receive the enhanced spectral features sent by the adaptive spectral attention mask module and generate a tone representation vector through contrastive learning;

[0011] a tone error detection module, connected to the tone characterization module, configured to receive the tone characterization vector sent by the tone characterization module, and analyze the tone characterization vector to determine the tone error type and error mode;

[0012] And a multimodal feedback module is connected to the tone error detection module, used to receive the tone error type and error pattern sent by the tone error detection module, and generate comprehensive guidance feedback on tone pronunciation and tongue position movement based on the tone error type and error pattern.

[0013] Preferably, the multi-channel tone spectrum specialized decomposition module includes:

[0014] a frequency band hierarchical processing unit, configured to divide the speech signal into a low frequency band, a mid-frequency band, and a high frequency band, wherein the low frequency band is [80 Hz, 500 Hz), the mid-frequency band is [500 Hz, 2000 Hz), and the high frequency band is [2000 Hz, 8000 Hz];

[0015] a tone core band enhancement unit for applying a non-uniform Mel filter bank to the low frequency band, densely distributing filters in the range of 80 Hz to 400 Hz;

[0016] and a multi-scale feature extraction unit for applying different time-frequency transforms to the low-frequency band, the mid-frequency band and the high-frequency band respectively to generate multi-scale tone features.

[0017] Preferably, the adaptive spectrum attention mask module includes:

[0018] an attention calculation unit, configured to calculate a spectral attention score based on the multi-scale tone features;

[0019] a tone adaptation unit, configured to adjust the weight of the spectral attention score according to the tone type, wherein the attention weight of the third tone is higher than that of other tones;

[0020] Multi-head attention unit, used to implement multiple parallel attention mechanisms, each of which focuses on the tone features of different frequency ranges;

[0021] and a mask generating unit, configured to generate a spectrum enhancement mask based on the attention score, and enhance the frequency region related to the tone feature.

[0022] Preferably, the tone characterization module includes:

[0023] A comparison sample library unit is used to store standard samples, similar tone samples, hard negative samples, and soft negative samples;

[0024] a tone similarity calculation unit, used to calculate the similarity between the enhanced spectral features and the comparison sample;

[0025] a representation encoder unit, configured to encode the enhanced spectral features into a tone representation vector;

[0026] and a native language interference suppression unit for identifying and reducing the interference of native language tone patterns on Chinese tone pronunciation.

[0027] Preferably, the tone error detection module includes:

[0028] an error feature library unit for storing features of typical tone error patterns;

[0029] A hierarchical classification unit for determining tone type, error severity, and specific error patterns through three-level classification;

[0030] A correction strategy library unit, used to store correction strategies corresponding to different error modes;

[0031] and a strategy matching unit for matching an optimal correction strategy according to the detected error pattern.

[0032] Preferably, the multimodal feedback module includes:

[0033] Acoustic-tongue position mapping unit, used to predict ideal tongue position parameters based on the enhanced spectral features;

[0034] Tongue position dynamic tracking unit, used to generate the three-dimensional coordinates of 18 tongue position key points;

[0035] A collaborative optimization unit for establishing a two-way enhancement mechanism for tone improvement and tongue position adjustment;

[0036] and a feedback presentation unit for generating visual, auditory, and tactile multimodal feedback.

[0037] Preferably, the acoustic-tongue position mapping unit includes:

[0038] A forward mapping subunit, configured to predict ideal tongue position parameters based on the enhanced spectral features;

[0039] A reverse mapping subunit for generating acoustic adjustment suggestions based on the difference between the actual tongue position and the ideal tongue position;

[0040] and a mapping optimization subunit for adjusting mapping parameters for different tone types, wherein the tongue position accuracy requirement for the third tone is higher than that for other tones.

[0041] Preferably, the tongue position dynamic tracking unit includes:

[0042] Key point positioning subunit, used to locate the three-dimensional coordinates of the tongue tip, tongue body, tongue root and auxiliary points;

[0043] The timing modeling subunit is used to capture the timing characteristics of tongue position movement;

[0044] and the critical moment enhancement subunit, which is used to highlight the tongue position representation at the critical moment of tone articulation.

[0045] Preferably, the feedback presentation unit includes:

[0046] The visual channel subunit is used to present 3D tongue position model animation, tone contour curves, and pronunciation intensity heat maps;

[0047] Auditory channel subunit, used to play standard pronunciation examples and adjusted speech synthesis;

[0048] and graded feedback sub-units to provide beginner, intermediate, and advanced feedback content based on learner levels.

[0049] The Chinese tone correction method for cross-modal adversarial generation includes:

[0050] The speech signal is divided into low-frequency band, mid-frequency band and high-frequency band for parallel processing to extract multi-scale tone features;

[0051] generating a spectrum enhancement mask based on the multi-scale tone features to dynamically enhance frequency regions related to the tone features;

[0052] Based on the enhanced spectral features, a tone representation vector is generated through contrastive learning;

[0053] Analyzing the tone representation vector to determine the tone error type and error pattern;

[0054] And based on the tone error type and error pattern, generate comprehensive guidance feedback on tone pronunciation and tongue position movement.

[0055] The present invention has the following beneficial effects:

[0056] 1. Improved accuracy of tone error detection: Through multi-channel tone spectrum-specific decomposition and adaptive spectral attention masking technology, the tone error detection rate has increased to 95%, with an accuracy rate of 92%. In particular, the accuracy of detecting the more difficult third tone has been significantly improved.

[0057] 2. Enhanced learning outcomes: Customized for the language background characteristics of Southeast Asian learners, the app improves learners' third tone accuracy by 58% and accelerates overall learning progress by approximately 40%.

[0058] 3. Improve user experience: System response delay is reduced to less than 300 milliseconds, and multimodal feedback provides intuitive guidance, significantly improving learning experience and efficiency.

[0059] 4. Achieve personalized adaptation: Dynamically adjust model parameters and error correction strategies based on the user's language background and learning process, and continuously optimize as the user uses it.

[0060] 5. Expanded application value: The core technology can be used in other fields such as tonal language learning and speech disorder treatment, and has broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 Schematic diagram of the overall system architecture of the present invention;

[0062] Figure 2 Schematic diagram of the structure of the multi-channel tone spectrum specialized decomposition module of the present invention;

[0063] Figure 3 Schematic diagram of the structure of the adaptive spectral attention mask module of the present invention;

[0064] Figure 4 Schematic diagram of the structure of the tone characterization module of the present invention;

[0065] Figure 5 Schematic diagram of the structure of the tone error detection module of the present invention;

[0066] Figure 6 Schematic diagram of the structure of the multimodal feedback module of the present invention;

[0067] Figure 7 Schematic diagram of the structure of the acoustic-tongue position mapping unit of the present invention;

[0068] Figure 8 Schematic diagram of the structure of the tongue position dynamic tracking unit of the present invention;

[0069] Figure 9 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0070] Please refer to Figures 1-9 , the embodiments of the present invention are described in detail below with reference to the accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0071] like Figure 1 As shown in FIG, the Chinese tone error correction system for cross-modal adversarial generation provided by the present invention includes a multi-channel tone spectrum specialized decomposition module 1, an adaptive spectral attention mask module 2, a tone representation module 3, a tone error detection module 4, and a multimodal feedback module 5. The connections and data flow between these modules are as follows:

[0072] The multi-channel tone spectrum specialized decomposition module 1 receives the speech signal input, divides it into low-frequency band, medium-frequency band and high-frequency band for parallel processing, and extracts multi-scale tone features; the adaptive spectrum attention mask module 2 is connected to the multi-channel tone spectrum specialized decomposition module 1, receives the multi-scale tone features, generates a spectrum enhancement mask, and dynamically enhances the frequency area related to the tone features; the tone representation module 3 is connected to the adaptive spectrum attention mask module 2, receives the enhanced spectrum features, and generates a tone representation vector through comparative learning; the tone error detection module 4 is connected to the tone representation module 3, receives the tone representation vector, analyzes and determines the tone error type and error pattern; the multimodal feedback module 5 is connected to the tone error detection module 4, receives the tone error type and error pattern, and generates comprehensive guidance feedback on tone pronunciation and tongue position movement.

[0073] like Figure 2 As shown, the multi-channel tone spectrum specialized decomposition module 1 includes a frequency band layering processing unit 11 , a tone core band enhancement unit 12 and a multi-scale feature extraction unit 13 .

[0074] The frequency band hierarchical processing unit 11 is used to divide the speech signal into a low frequency band, a medium frequency band and a high frequency band, wherein the low frequency band is [80Hz, 500Hz), the medium frequency band is [500Hz, 2000Hz), and the high frequency band is [2000Hz, 8000Hz]. In one embodiment of the present invention, the low frequency band is the main area where the fundamental frequency of the tone changes, and a higher sampling density is adopted. The extraction parameters are set to a frame length of 32ms and a frame shift of 10ms; the medium frequency band contains consonant and vowel formant information, and the parameters are set to a frame length of 25ms and a frame shift of 15ms; the high frequency band contains voiced and unvoiced sound and fricative sound features, and the parameters are set to a frame length of 20ms and a frame shift of 15ms.

[0075] The tone core band enhancement unit 12 applies a non-uniform Mel filter bank to the low-frequency band, densely distributing filters in the 80Hz-400Hz range. Specifically, 16 filters are distributed in the 80Hz-400Hz range, 6 filters in the 400Hz-1000Hz range, and 4 filters in the 1000Hz-8000Hz range. Furthermore, a Hanning window is used for pre-emphasis processing, with a pre-emphasis coefficient set to 0.97, effectively enhancing high-frequency components. In practical applications, this unit provides targeted enhancement of tone-related frequency bands, improving the accuracy of subsequent tone feature extraction. For example, when a Vietnamese learner pronounced the tone-contrasting words "ma" (first tone), "ma" (second tone), "ma" (third tone), and "ma" (fourth tone), the low-point features of the third tone were significantly enhanced after processing by this unit, making frequency variations that would otherwise be easily overlooked more prominent. Experiments show that this design improves the signal-to-noise ratio of tone features by approximately 9.7dB.

[0076] The multi-scale feature extraction unit 13 applies different time-frequency transforms to different frequency bands to generate multi-scale tone features. In a preferred embodiment, a dual-path parallel processing approach is employed: a short-time Fourier transform (STFT) path and a wavelet packet transform (WPT) path. The STFT employs a 512-point FFT with a 50% overlap ratio and a triangular window function. The WPT employs a six-level decomposition using a biorthogonal wavelet basis to finely decompose the low-frequency band, reconstructing the wavelet packet coefficients into frequency band energy features, with a focus on retaining the frequency band coefficients associated with tone variation.

[0077] The representation of multi-scale tone features is a three-dimensional tensor ,in: The number of time frames indicates the number of sampling points in the time dimension of the audio signal, usually ranging from 300 to 500 frames; is the number of frequency points, indicating the number of sampling points in the frequency dimension; is the number of feature channels, which indicates the number of features generated by different feature extraction methods. For the low frequency band, Usually 300-500, is 256, is 16; for the mid-frequency band, 200-400, is 128, is 8; for high frequency band, 200-400, is 64, This three-dimensional tensor structure enables the system to simultaneously analyze changes in tone in three dimensions: time, frequency, and features, providing a rich information basis for subsequent processing.

[0078] like Figure 3As shown, the adaptive spectral attention mask module 2 includes an attention calculation unit 21, a tone adaptation unit 22, a multi-head attention unit 23 and a mask generation unit 24.

[0079] The attention calculation unit 21 is used to calculate the spectrum attention score based on the multi-scale tone features. In one embodiment of the present invention, the attention calculation adopts the self-attention mechanism to decompose the spectrum graph into a two-dimensional time-frequency feature graph, and generate three representation vectors of query, key, and value for each time-frequency point. The calculation formula of the attention score matrix A is:

[0080] ,

[0081] in: is the attention score matrix, with dimension , represents the attention weight of each time-frequency point; is the query matrix, with dimension ; is the key matrix with dimension ; express The transpose of is the dimension of the key vector, typically set to 64; softmax is the softmax function used to normalize the attention scores into a probability distribution. This unit effectively captures time-frequency correlations within the spectrum through a self-attention mechanism, providing a foundation for subsequent spectrum enhancement. For example, when a Thai learner pronounces "I want to buy two books" (with the fourth tone), this mechanism automatically focuses on the key frequency region of the fourth tone's descending phase, helping the system accurately capture tonal characteristics.

[0082] How the attention calculation unit 21 identifies the target tone type from incorrect tones and assigns a score involves the system's fundamental operating mechanism. This system does not directly infer the correct tone from the incorrect tone, but instead employs a method that combines goal guidance and feature matching. Specifically, at runtime, the system knows the target Chinese character and its standard tone (i.e., the practice content) that the learner is currently practicing. This provides prior information for determining the tone type. The system first performs a coarse classification, identifying the tone pattern to which the acoustic features most closely resemble, while also referencing the target tone information in the practice content.

[0083] Even incorrect tone pronunciations retain some of the basic characteristics of the original target tone. For example, when a learner attempts to pronounce the third tone but the low point is insufficient, the tone curve will still show a downward trend, but the amplitude will not be enough. The system uses the multi-scale features extracted by the multi-channel tone spectrum specialized decomposition module, combined with standard samples, similar tone samples, and hard negative samples in the comparison sample library for pattern matching to confirm the target tone. This hierarchical recognition strategy of coarse classification followed by fine classification enables the system to determine the target tone type even in the presence of tone deviations, thus providing a basis for subsequent attention weight adjustments.

[0084] In actual applications, when Vietnamese learners try to pronounce "ma" (the third tone), the system will know that the target is the third tone based on the practice content, and then analyze whether its acoustic characteristics conform to the basic pattern of the third tone. Even if the pronunciation is not standard, its tone type can be determined and the attention weight can be adjusted accordingly, with special attention paid to the low point and rising segment characteristics of the third tone.

[0085] The tone adaptation unit 22 is used to adjust the weight of the spectral attention score based on the tone type; preferably, the third tone has a higher attention weight than other tones. In specific implementation, the attention calculation parameters are adjusted for different tone types: the first tone uses a wide time window (±150ms); the second tone focuses on the rising frequency segment, using a medium window (±100ms); the third tone focuses on the turning point, using a narrow window (±80ms) and increasing the attention weight; the fourth tone focuses on the falling slope, using a decreasing weight window (120ms→30ms). In actual application scenarios, when the system detects that the learner is practicing the third tone, it automatically increases attention to the low point and rising segment of the tone. This is particularly important for Southeast Asian learners, who often have difficulty accurately pronouncing the complete curve of the third tone. Experiments show that this adaptive design improves the separability of the second and third tones in the feature space by 42.3%.

[0086] The multi-head attention unit 23 is used to implement multiple parallel attention mechanisms, each of which focuses on the tone features of different frequency intervals. In one embodiment of the present invention, an 8-head parallel attention mechanism is implemented: heads 1-2 focus on the 80Hz-200Hz fundamental frequency interval; heads 3-5 focus on the 200Hz-350Hz tone change sensitive area; heads 6-8 focus on the 350Hz-500Hz sound quality and tone auxiliary information area. Each attention head independently calculates the attention score and mask, and finally generates a comprehensive mask through weighted merging. The weight distribution is dynamically adjusted according to the currently detected tone type. For example, when the third tone is detected, the weights of heads 3-5 are automatically increased. The design of the multi-head attention mechanism enables the system to pay attention to different aspects of the tone features at the same time, improving the comprehensiveness and robustness of feature extraction.

[0087] The mask generation unit 24 is used to generate a spectrum enhancement mask based on the attention score to enhance the frequency region related to the tone feature. The calculation formula is:

[0088] ,

[0089] in: Frequency and time The enhancement mask value at ; Frequency and time The attention score at is the original spectrum value; is the first layer weight matrix, with dimension ,in is the dimension of the attention score, is the dimension of the original spectrum value, is the hidden layer dimension; is the second layer weight matrix, with dimension ; ReLU is the ReLU activation function, which is used to introduce nonlinear transformation; The sigmoid activation function compresses the output to the range of 0-1; Represents the feature connection operation, which connects the attention score and the original spectrum value on the feature dimension. The calculation formula is:

[0090] ,

[0091] in: is the enhanced spectrum value; is the original spectrum value; is the tone enhancement coefficient, and its values ​​corresponding to the four tones are 1.0, 1.0, 1.25, and 1.0 respectively; The mask value is set to enhance the tone. The mask value range is controlled between [0.5, 2.5] to ensure a natural and smooth enhancement effect. In practical applications, this adaptive enhancement mechanism can provide targeted processing based on the characteristics of different tones. For example, to address the flattening of the third tone, a common problem for Laotian learners, the system automatically enhances the low and rising points to help learners perceive the complete tone curve.

[0092] like Figure 4 As shown, the tone characterization module 3 includes a comparison sample library unit 31 , a tone similarity calculation unit 32 , a characterization encoder unit 33 and a native language interference suppression unit 34 .

[0093] The comparison sample library unit 31 is used to store standard samples, similar tone samples, hard negative samples, and soft negative samples. In one embodiment of the present invention, a hierarchical comparison sample library is constructed, comprising four levels of reference samples: standard samples are derived from the standard pronunciation of professional Mandarin teachers (10-15 variants per tone); similar tone samples are variants of the same tone produced by different speakers (20-30 per tone); hard negative samples are samples of easily confused tone pairs (particularly the initial parts of the second vs. third tone, and the first vs. fourth tone); and soft negative samples are slightly deviated versions of the same tone, graded by the degree of deviation. For Southeast Asian learners, additional specialized sample sets are constructed, including tone pronunciation comparisons for Vietnamese, Thai, and Lao. For example, for Vietnamese learners, comparison samples of Vietnamese and Chinese tones are specifically collected to help the system understand the phenomenon that Vietnamese learners tend to pronounce the Chinese third tone as the Vietnamese accent tone (nặng).

[0094] The tone similarity calculation unit 32 is used to calculate the similarity between the enhanced spectral features and the comparison sample. Preferably, the tone contour similarity calculation method is designed to consider three key dimensions: overall shape similarity, key point similarity and duration similarity. The calculation formula is:

[0095] ,

[0096] in: is the comprehensive tone similarity, ranging from [0,1], where larger values ​​indicate more similarity; is the fundamental frequency profile similarity calculated by dynamic time warping (DTW), ranging from [0, 1]; is the key point similarity, ranging from [0, 1], obtained by calculating the frequency deviation of key moments (such as starting point, end point, turning point); is the duration similarity, ranging from [0,1], which is calculated by comparing the duration of the tone with the matching degree of the standard template; 、 、 is the weight coefficient, satisfying , usually set to , , The similarity thresholds for different tones are: 0.82 for the first tone, 0.85 for the second tone, 0.88 for the third tone, and 0.80 for the fourth tone. Tones above these thresholds are considered close to the standard. These thresholds were determined through extensive experimentation. The third tone threshold is set highest because it is the most challenging for Southeast Asian learners to pronounce correctly and requires a more stringent standard.

[0097] The representation encoder unit 33 is used to encode the enhanced spectral features into a tone representation vector. In one embodiment of the present invention, a contrastive learning framework is used to train the tone feature encoder, so that the representations of the same tone are aggregated and the representations of different tones are separated. The encoder adopts a four-layer structure: a bidirectional GRU layer captures temporal features, a temporal attention layer highlights key frames, a feature integration layer fuses multi-scale features, and a mapping layer generates the final representation. Contrastive learning loss function Defined as:

[0098] ,

[0099] in: To compare the learning loss function value, the smaller it is, the better the learning effect; is the representation vector of the query sample, with a dimension of 128; is the representation vector of the positive sample, with a dimension of 128; For the The representation vector of negative samples has a dimension of 128; is the cosine similarity function, which calculates the similarity between two vectors in the range of [-1,1]; is the temperature parameter, with a value of 0.08, which controls the smoothness of the similarity distribution; For the The weight of each negative sample is related to the difficulty of confusing the current tone. For example, for the third tone, the weight of the negative sample confused with the second tone is set to 1.5, and the weight of the negative sample confused with the fourth tone is set to 1.2; represents the sum of all negative samples. This loss function learns discriminative tone representations by reducing the distance between the query sample and the positive sample while increasing the distance between the query sample and the negative sample. The resulting tone representation vector has a dimension of 128 and is normalized to the interval [-1, 1].

[0100] The native language interference suppression unit 34 is used to identify and reduce the interference of native language tone patterns on the pronunciation of Chinese tones. Tone interference models are established for different language backgrounds, with special attention paid to: the influence of Vietnamese on the third tone of Chinese (the problem of insufficient descent); the influence of Thai on the second tone of Chinese (the problem of excessive rise); and the influence of Lao on the overall tone pattern of the tone. For example, for Vietnamese learners, the system will automatically identify whether there is a tendency to weaken or omit the rising segment of the third tone of Chinese in their pronunciation, which is a typical problem affected by the Vietnamese tone system. Through the tone transfer suppression network, the system can identify such patterns and specifically emphasize the importance of the rising segment of the third tone in the complete pronunciation in the feedback. This unit significantly improves the tone learning effect of learners from different language backgrounds and makes the error correction process more targeted.

[0101] like Figure 5As shown, the tone error detection module 4 includes an error feature library unit 41 , a hierarchical classification unit 42 , a correction strategy library unit 43 and a strategy matching unit 44 .

[0102] The error feature library unit 41 is used to store features of typical tone error patterns. In this embodiment of the present invention, 54 typical tone error patterns were extracted and classified based on 10,000 hours of Chinese pronunciation data from Southeast Asian learners. These error patterns were multi-dimensionally annotated based on language background, difficulty level, and frequency distribution. The main error types and their proportions are: insufficient or reversed third tone (28.5%), including subtypes such as insufficient low pitch, missing rise, and flattened curve; confusion between the second and third tones (24.3%), including incorrect starting point, improper rise angle, and deviation from the end position; insufficiently steep fourth tone descent (19.2%), including variants such as too low a starting point, slow descent, and abnormal length; flattened tone (15.5%); and other error types (12.5%). The error feature vector has a dimension of 256 and describes the acoustic features of the error pattern. In practical applications, for example, when a Thai learner practices the phrase "I am very good," the system can accurately identify the pronunciation problem with the third tone "very" as insufficient low pitch, a common error pattern among Thai learners.

[0103] Hierarchical classification unit 42 is used to determine tone type, error severity, and specific error patterns through a three-level classification process. This three-level classification architecture allows for coarse-to-fine error identification: the first level is tone type classification, identifying basic tone categories (four tones + light tone); the second level is error severity assessment, categorized as mild (deviation <15%), moderate (deviation 15%-30%), and severe (deviation >30%); and the third level is specific error pattern identification, pinpointing specific types within 54 error patterns. The classifier uses a variant residual network architecture and takes a tone representation vector as input. The classification threshold is set to 0.75 and dynamically adjusted based on user feedback. For example, if the system determines that a user's tone deviation is between 15% and 30%, the error severity is determined to be moderate and the subsequent feedback is adjusted accordingly, ensuring that the feedback is both targeted and not frustrating to the learner. Accurate classification by this unit provides the foundation for the subsequent generation of personalized correction strategies.

[0104] The correction strategy library unit 43 is used to store correction strategies corresponding to different error patterns. The strategy library includes: strategies such as visual comparison of tone contours, reinforcement of key point pronunciation, guidance on tone transitions, and correction of the positions of speech organs. Each strategy is parametrically designed for different error types. For example, for the problem of insufficient low point of the third tone, specific guidance on the downward movement of the middle part of the tongue is provided: Please try to let the middle part of the tongue move downward in the middle of pronouncing "ma", feel the position of the lowest point of the tongue, and then lift it upward; for the problem of improper rising angle of the second tone, tone contour comparison and key point marking are provided: Note that your tone rising amplitude is not enough. Please observe the difference between the standard curve (showing a rising angle of about 45 degrees) and your pronunciation (showing a rising angle of about 20 degrees). The design of the strategy library is based on phonetic knowledge and teaching experience, and is verified through a large number of tests to ensure the practicality and effectiveness of the correction suggestions.

[0105] The strategy matching unit 44 is used to match the best correction strategy according to the detected error pattern. A reinforcement learning algorithm based on Thompson sampling is used for strategy selection, modeling the strategy selection problem as a multi-armed bandit problem, with each strategy as an arm and the strategy effect as the reward. The probability distribution of strategy selection is a Beta distribution, and the parameters are dynamically updated according to the historical success and failure times. The probability of strategy selection is calculated as follows:

[0106] ,

[0107] where is the probability of selecting strategy ; is the Beta distribution value of strategy ; is the number of successes of strategy plus 1; is the number of failures of strategy plus 1; is the number of successes of strategy plus 1; is the number of failures of strategy plus 1; represents the sum over all available strategies. When the learner makes adjustments according to the system's suggestions, the system will evaluate the correction effect. If the tone accuracy improves, the value of this strategy will be increased; if the effect is not obvious, the value will be increased. With the accumulation of user feedback, the system automatically optimizes strategy selection to improve the correction effect. For example, the system may find that for the third tone problem of Vietnamese learners, the effect of tongue position guidance is better than simple tone curve visualization, so it will gradually increase the probability of selecting the tongue position guidance strategy. Experiments show that compared with fixed strategies, this method increases the correction efficiency by about 15%.

[0108] like Figure 6 As shown, the multimodal feedback module 5 includes an acoustic-tongue position mapping unit 51 , a tongue position dynamic tracking unit 52 , a collaborative optimization unit 53 and a feedback presentation unit 54 .

[0109] The acoustic-tongue position mapping unit 51 is used to predict the ideal tongue position parameters based on the enhanced spectral features. Figure 7 As shown, the unit includes a forward mapping subunit 511, a reverse mapping subunit 512 and a mapping optimization subunit 513. The forward mapping subunit 511 predicts the ideal tongue position parameters based on the enhanced spectrum features, and the mapping function is:

[0110] ,

[0111] in: is the tongue position parameter vector with a dimension of 18×3, representing the three-dimensional coordinates of 18 tongue position key points; is the enhanced spectrum feature; The mapping network is implemented by a multi-layer neural network. The reverse mapping subunit 512 generates acoustic adjustment suggestions based on the difference between the actual tongue position and the ideal tongue position. The mapping function is:

[0112] ,

[0113] in: is the acoustic correction quantity, indicating the acoustic characteristics that need to be adjusted; is the actual tongue position parameter; is the ideal tongue position parameter; It is a reverse mapping network. The mapping optimization subunit 513 adjusts the mapping parameters for different tone types, among which the tongue position accuracy requirement for the third tone is higher than that for other tones. Through the transfer learning method, pre-training is performed based on X-ray pronunciation data and MRI speech data sets to establish an accurate acoustic-tongue position mapping relationship. In actual application, when the learner practices pronouncing "I" (the third tone) well, the system will predict the ideal tongue position based on its acoustic characteristics and generate specific adjustment suggestions. For example, when pronouncing the third tone, the tip of the tongue needs to be moved down by about 5mm, and the middle of the tongue needs to be moved down by about 8mm to achieve the correct low point position.

[0114] How the acoustic-tongue mapping unit 51 solves the many-to-one mapping problem where the same spectrum can be generated by different oral movements is indeed a classic challenge in the field of speech-tongue mapping. The present invention adopts multiple technical strategies to constrain the solution space and improve the mapping accuracy:

[0115] First, the system leverages the physical constraints of tone pronunciation to narrow the mapping space. Chinese tones have specific tongue position patterns, such as the low point of the third tone corresponding to the lowest position of the middle of the tongue. This phonetic knowledge is encoded as a priori constraints, significantly reducing the number of possible tongue position configurations.

[0116] Second, the system not only relies on single-frame static mapping but also leverages temporal information. The temporal modeling subunit 522 captures the dynamic characteristics of tongue position movement and processes continuous frame sequences using a three-layer bidirectional network. This enables the system to distinguish between situations where the tongue position trajectories differ despite similar spectra over a short period of time.

[0117] Third, the mapping function is optimized through large-scale data pre-training. Using transfer learning, the system is pre-trained on X-ray pronunciation data and MRI speech datasets. This real-world human pronunciation data establishes a statistical correlation model between acoustic features and tongue position parameters, significantly improving the accuracy and robustness of the mapping.

[0118] Fourth, the mapping optimization subunit 513 dynamically adjusts the mapping parameters for different tone types, especially requiring higher tongue position accuracy for the third tone. This specialized design enables the system to accurately map different tones.

[0119] Finally, the system uses a two-way mapping mechanism (forward and reverse mapping) to form a closed-loop verification. When it detects that the mapping result may be ambiguous, the system will combine multimodal information (such as tone type, language background, etc.) to make a comprehensive judgment and select the most likely tongue position configuration.

[0120] Practice has proven that, despite theoretically presenting a one-to-many mapping problem, the comprehensive application of the aforementioned strategies enables the system to generate reasonably accurate tongue position parameters in the vast majority of cases, providing effective pronunciation guidance for learners. Extensive test data demonstrates that the tongue position parameters generated by this system match real MRI observations with over 85% accuracy, fully meeting the requirements of teaching applications.

[0121] The tongue position dynamic tracking unit 52 is used to generate the three-dimensional coordinates of the tongue position key points. Figure 8As shown, this unit includes a key point location subunit 521, a timing modeling subunit 522, and a key moment enhancement subunit 523. The key point location subunit 521 locates the three-dimensional coordinates of the tongue tip, tongue body, tongue root, and auxiliary points, totaling 18 key points: the tongue tip (3 points), tongue body (9 points), tongue root (3 points), and auxiliary points (3 points). The timing modeling subunit 522 captures the temporal characteristics of tongue position movement using a three-layer bidirectional network with 256 units per layer and a sampling rate of 30Hz. The key moment enhancement subunit 523 highlights the tongue position representation at key moments in tone articulation, such as the descent and rise of the third tone. The tongue position representation uses a relative coordinate system, referenced by oral anatomy, and is expressed in millimeters. For example, at the key moment of the third tone, the system pays special attention to the position and shape of the tongue at the descent point, as well as the trajectory of the tongue as it moves upward from the descent point. This information is crucial for guiding learners in correctly pronouncing the third tone. This precise tongue position modeling enables learners to intuitively understand the correct position and movement of the articulators.

[0122] The collaborative optimization unit 53 is used to establish a two-way enhancement mechanism for tone improvement and tongue position adjustment. Tone deviation drives tongue position adjustment suggestions. For example, when the low point of the third tone is insufficient, the middle of the tongue is prompted to drop; tongue position adjustment feedback affects the tone pronunciation strategy and provides more accurate pronunciation organ control suggestions. A collaborative enhancement network is designed to connect the tone and tongue position feedback channels to form a closed-loop feedback system. Preferably, the feedback priority is dynamically adjusted, and tongue position guidance is provided first for beginners, while advanced learners focus on tone pattern guidance. For example, for a Thai student who has just started learning Chinese, the system will first provide intuitive tongue position guidance: pay attention to the movement of the tongue in the 3D model and try to move your tongue along with the model; for learners who already have a certain foundation, the system will pay more attention to the tone pattern: the overall shape of your tone is close to the standard, but the rate of the rising segment needs to be accelerated, so pay attention to the speed of change in the demonstration audio. Through the collaborative optimization of tone and tongue position, the limitations of a single feedback mode are effectively solved.

[0123] The feedback presentation unit 54 is used to generate multimodal feedback through visual, auditory, and tactile means. This unit includes a visual channel subunit 541, an auditory channel subunit 542, and a graded feedback subunit 543. The visual channel subunit 541 presents a 3D tongue position model animation, tone contour curves, and pronunciation intensity heatmaps; the auditory channel subunit 542 plays standard pronunciation examples and adjusted speech synthesis; and the graded feedback subunit 543 provides elementary, intermediate, and advanced feedback content based on the learner's level. Feedback presentation adopts a graded strategy to avoid information overload: elementary feedback includes simplified tone curves and basic tongue position guidance; intermediate feedback includes detailed tone deviation analysis and key tongue position adjustment suggestions; and advanced feedback includes complete tongue position movement trajectories and detailed tone adjustment guidance. For example, when a Lao learner practices the word "thank you," they might receive basic feedback: "The tone of the first "xie" (the fourth tone) doesn't fall quickly enough. Note the rapid descent from high to low." Intermediate feedback might include: "Your fourth tone starts correctly, but the descent rate is only 70% of the standard. Please compare the curves and note the rapid change in tongue position from raised to relaxed." Advanced feedback might include more detailed data and more precise guidance: "Your fourth tone's descent duration is 240ms, 33% longer than the standard 180ms, and the frequency variation is 80Hz, 33% less than the standard 120Hz. Please adjust tongue position and airflow control according to the 3D model." Feedback is intelligently timed to ensure it doesn't interrupt the user's pronunciation flow, primarily providing syllable breaks or at the end of sentences. This multimodal, graded feedback design significantly improves learning outcomes and user experience.

[0124] The present invention also provides a Chinese tone error correction method for cross-modal adversarial generation, comprising the following steps:

[0125] Step 1: Divide the speech signal into low-frequency, mid-frequency, and high-frequency bands for parallel processing to extract multi-scale tone features;

[0126] Step 2: generating a spectrum enhancement mask based on the multi-scale tone feature, and dynamically enhancing the frequency region related to the tone feature;

[0127] Step 3: Generate a tone representation vector through contrastive learning based on the enhanced spectral features;

[0128] Step 4: Analyze the tone representation vector to determine the tone error type and error pattern;

[0129] Step five: generating comprehensive guidance feedback on tone pronunciation and tongue position based on the tone error type and error pattern.

[0130] Specifically, in step 1, the speech signal is divided into a low-frequency band [80Hz, 500Hz], a mid-frequency band [500Hz, 2000Hz], and a high-frequency band [2000Hz, 8000Hz]. A non-uniform Mel filter bank is applied to the low-frequency band, and filters are densely distributed in the 80-400Hz range. Two parallel processing paths are used: a short-time Fourier transform path and a wavelet packet transform path to generate multi-scale tone features.

[0131] In step 2, the spectral attention score is calculated based on the multi-scale tone features, the attention weight is adjusted according to the tone type, and the multi-head attention mechanism is used to focus on the tone features in different frequency ranges. A spectral enhancement mask is generated to enhance the frequency regions related to the tone features.

[0132] In step three, the similarity between the enhanced spectral features and the comparison samples is calculated using a comparison sample library (including standard samples, similar tone samples, hard negative samples, and soft negative samples). A tone representation vector is generated through a contrastive learning encoder, while simultaneously identifying and reducing the interference of native language tone patterns.

[0133] In step 4, the tone representation vector is analyzed to determine the tone error type and error pattern. This specifically involves determining the tone error type and error pattern through a three-level classification (tone type, error severity, and specific error pattern) based on the error feature library, and then querying the correction strategy library to match the optimal correction strategy.

[0134] In step five, comprehensive guidance feedback on tone pronunciation and tongue position is generated based on the tone error types and error patterns. Specifically, this includes predicting ideal tongue position parameters based on acoustic-tongue position mapping, generating three-dimensional coordinates for 18 key tongue position points, establishing a two-way reinforcement mechanism for tone improvement and tongue position adjustment, and providing multimodal feedback through visual channels (3D tongue position model animation, tone contour curves), auditory channels (standard pronunciation examples), and a graded feedback strategy.

[0135] The method of the present invention constructs a complete Chinese tone correction solution by organically combining technologies such as multi-channel tone spectrum specialized decomposition, adaptive spectral attention masking, tone contrast learning representation, precise error pattern recognition, and comprehensive guidance and feedback on tone pronunciation and tongue position movements. It is particularly suitable for helping Southeast Asian learners improve the accuracy of Chinese tone pronunciation.

[0136] Through the above-mentioned technical solution, the present invention achieves high-precision detection and precise correction of Chinese tone errors. This technology specifically addresses the third tone learning challenges faced by Southeast Asian learners. Through a comprehensive guidance and feedback mechanism for tone pronunciation and tongue position, it provides intuitive and effective learning guidance, significantly improving learning outcomes and user experience. Compared to existing technologies, this invention significantly improves tone error detection accuracy, error correction effectiveness, response speed, and user adaptability, promising broad application prospects.

[0137] The preferred embodiments of the present invention are described in detail above. However, the present invention is not limited to the specific details in the above embodiments. Within the technical concept of the present invention, various simple modifications can be made to the technical solution of the present invention, and these simple modifications all fall within the scope of protection of the present invention.

Claims

1. A Chinese tone correction system based on cross-modal adversarial generation, characterized by: include: A multi-channel tone spectrum decomposition module is used to divide the speech signal into low-frequency, mid-frequency, and high-frequency bands for parallel processing and extract multi-scale tone features; an adaptive spectral attention mask module, connected to the multi-channel tone spectrum specialized decomposition module, configured to receive the multi-scale tone features sent by the multi-channel tone spectrum specialized decomposition module, generate a spectral enhancement mask based on the multi-scale tone features, and dynamically enhance frequency regions related to the tone features; a tone representation module, connected to the adaptive spectral attention mask module, configured to receive the enhanced spectral features sent by the adaptive spectral attention mask module and generate a tone representation vector through contrastive learning; a tone error detection module, connected to the tone characterization module, configured to receive the tone characterization vector sent by the tone characterization module, and analyze the tone characterization vector to determine the tone error type and error mode; And a multimodal feedback module is connected to the tone error detection module, used to receive the tone error type and error pattern sent by the tone error detection module, and generate comprehensive guidance feedback on tone pronunciation and tongue position movement based on the tone error type and error pattern.

2. The system according to claim 1, wherein: The multi-channel tone spectrum specialized decomposition module includes: a frequency band hierarchical processing unit, configured to divide the speech signal into a low frequency band, a mid-frequency band, and a high frequency band, wherein the low frequency band is [80 Hz, 500 Hz), the mid-frequency band is [500 Hz, 2000 Hz), and the high frequency band is [2000 Hz, 8000 Hz]; a tone core band enhancement unit for applying a non-uniform Mel filter bank to the low frequency band, densely distributing filters in the range of 80 Hz to 400 Hz; and a multi-scale feature extraction unit for applying different time-frequency transforms to the low-frequency band, the mid-frequency band and the high-frequency band respectively to generate multi-scale tone features.

3. The system according to claim 1, wherein: The adaptive spectral attention mask module includes: an attention calculation unit, configured to calculate a spectral attention score based on the multi-scale tone features; a tone adaptation unit, configured to adjust the weight of the spectral attention score according to the tone type, wherein the attention weight of the third tone is higher than that of other tones; Multi-head attention unit, used to implement multiple parallel attention mechanisms, each of which focuses on the tone features of different frequency ranges; and a mask generating unit, configured to generate a spectrum enhancement mask based on the attention score, and enhance the frequency region related to the tone feature.

4. The system according to claim 1, wherein: The tone characterization module includes: A comparison sample library unit is used to store standard samples, similar tone samples, hard negative samples, and soft negative samples; a tone similarity calculation unit, used to calculate the similarity between the enhanced spectral features and the comparison sample; a representation encoder unit, configured to encode the enhanced spectral features into a tone representation vector; and a native language interference suppression unit for identifying and reducing the interference of native language tone patterns on Chinese tone pronunciation.

5. The system according to claim 1, wherein: The tone error detection module includes: an error feature library unit for storing features of typical tone error patterns; A hierarchical classification unit for determining tone type, error severity, and specific error patterns through three-level classification; A correction strategy library unit, used to store correction strategies corresponding to different error modes; and a strategy matching unit for matching an optimal correction strategy according to the detected error pattern.

6. The system according to claim 1, wherein: The multimodal feedback module includes: Acoustic-tongue position mapping unit, used to predict ideal tongue position parameters based on the enhanced spectral features; Tongue position dynamic tracking unit, used to generate the three-dimensional coordinates of 18 tongue position key points; A collaborative optimization unit for establishing a two-way enhancement mechanism for tone improvement and tongue position adjustment; and a feedback presentation unit for generating visual, auditory, and tactile multimodal feedback.

7. The system according to claim 6, characterized in that The acoustic-tongue position mapping unit includes: A forward mapping subunit, configured to predict ideal tongue position parameters based on the enhanced spectral features; A reverse mapping subunit for generating acoustic adjustment suggestions based on the difference between the actual tongue position and the ideal tongue position; and a mapping optimization subunit for adjusting mapping parameters for different tone types, wherein the tongue position accuracy requirement for the third tone is higher than that for other tones.

8. The system according to claim 6, wherein: The tongue position dynamic tracking unit includes: Key point positioning subunit, used to locate the three-dimensional coordinates of the tongue tip, tongue body, tongue root and auxiliary points; The timing modeling subunit is used to capture the timing characteristics of tongue position movement; and the critical moment enhancement subunit, which is used to highlight the tongue position representation at the critical moment of tone articulation.

9. The system according to claim 6, wherein: The feedback presentation unit includes: The visual channel subunit is used to present 3D tongue position model animation, tone contour curves, and pronunciation intensity heat maps; Auditory channel subunit, used to play standard pronunciation examples and adjusted speech synthesis; and graded feedback sub-units to provide beginner, intermediate, and advanced feedback content based on learner levels.

10. A Chinese tone correction method based on cross-modal adversarial generation, characterized in that: include: The speech signal is divided into low-frequency band, mid-frequency band and high-frequency band for parallel processing to extract multi-scale tone features; generating a spectrum enhancement mask based on the multi-scale tone features to dynamically enhance frequency regions related to the tone features; Based on the enhanced spectral features, a tone representation vector is generated through contrastive learning; Analyzing the tone representation vector to determine the tone error type and error pattern; And based on the tone error type and error pattern, generate comprehensive guidance feedback on tone pronunciation and tongue position movement.

Images