Mandarin pronunciation real-time correction method and system based on multi-modal streaming learning

Abstract

The application discloses a mandarin pronunciation real-time correction method and system based on multi-modal streaming learning, and belongs to the technical field of speech processing. The method collects user voice signals and face image information, pre-processes and splits the voice signals to obtain a voice frame sequence, extracts lip shape and tongue position features by using face image recognition technology, innovatively constructs a multi-modal topological feature space, establishes a time sequence corresponding relationship between voice and visual features through manifold consistency mapping, fuses multi-modal features by using a topological guided attention mechanism, performs frame-level phoneme prediction based on the fused features through a joint discriminant model, marks pronunciation error paragraphs and generates targeted correction suggestions. The application also includes a user feedback and system optimization mechanism, realizes personalized adaptation and continuous optimization, and significantly improves the pronunciation error detection accuracy and correction effect through the deep fusion of multi-modal information collaborative analysis and topological data analysis compared with the prior art.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, specifically to a method and system for real-time correction of Mandarin pronunciation based on multimodal streaming learning, which is particularly suitable for applications such as Mandarin teaching, speech rehabilitation training, and language learning. Background Technology

[0002] Mastering standard Mandarin pronunciation is crucial for personal communication, education, and career development. However, due to the influence of dialects or speech disorders, many people experience varying degrees of problems with Mandarin pronunciation, requiring effective corrective measures.

[0003] Traditional Mandarin pronunciation correction mainly relies on face-to-face instruction from professional teachers. While effective, this method is costly and has limited reach, making it difficult to meet the needs of large-scale implementation. With the development of computer technology, automatic pronunciation assessment systems based on speech recognition have emerged. These systems analyze users' speech signals, identify pronunciation errors, and provide feedback. However, existing technologies have the following main shortcomings:

[0004] On the one hand, most systems analyze only a single speech signal, ignoring visual information such as facial expressions, lip shape, and tongue position during pronunciation, resulting in limited corrective effects. Speech is a complex physiological process involving the coordinated action of multiple aspects such as the vocal cords, oral cavity, lip shape, and tongue position; analyzing only speech signals is insufficient to fully grasp the essence of pronunciation problems.

[0005] On the other hand, existing systems mostly adopt offline batch processing mode, lacking real-time interactive feedback capabilities. Users often need to complete the entire pronunciation before receiving the evaluation result. This delayed feedback mode reduces learning efficiency and is not conducive to users correcting errors in a timely manner.

[0006] Furthermore, existing technologies have significant shortcomings in multimodal data fusion. Simple methods such as feature concatenation or weighted averaging are insufficient to fully uncover the intrinsic relationships between different modalities, and cannot achieve deep information fusion and complementary enhancement.

[0007] Therefore, there is an urgent need for a Mandarin pronunciation correction method and system that can simultaneously analyze speech and visual information, achieve real-time interactive feedback, and deeply integrate multimodal data. Summary of the Invention

[0008] To address the shortcomings of existing technologies, the present invention aims to provide a method and system for real-time correction of Mandarin pronunciation based on multimodal streaming learning. By simultaneously analyzing the user's speech signals and facial visual information, a topological data analysis method is used to achieve deep fusion of multimodal data, and real-time interactive feedback is achieved based on a streaming learning framework, thereby improving the accuracy and efficiency of Mandarin pronunciation correction.

[0009] This invention proposes a real-time Mandarin pronunciation correction method based on multimodal streaming learning, including:

[0010] Collect user voice signals and facial image information;

[0011] The speech signal is preprocessed and segmented to obtain a speech frame sequence;

[0012] The facial image recognition technology is used to extract the user's lip shape and tongue position features;

[0013] A multimodal topological feature space is constructed, which includes speech feature topology, lip shape feature topology, and tongue position feature topology.

[0014] Based on the multimodal topological feature space, manifold consistency mapping is performed to establish a temporal correspondence between speech features and visual features;

[0015] Based on the manifold consistency mapping results, speech features and visual features are fused through a topology-guided attention mechanism to generate multimodal fusion features;

[0016] Based on the multimodal fusion features, the end-to-end trained joint discrimination model is invoked to perform frame-level prediction, thereby obtaining the predicted label of each phoneme during the user's pronunciation process.

[0017] Based on the predicted tags and the standard pronunciation library, mark the paragraphs in which the user pronounces them incorrectly;

[0018] The system generates correction suggestions for the segments with pronunciation errors and plays the corrected pronunciation from the user on the audio device.

[0019] Preferably, the steps of preprocessing and segmenting the speech signal specifically include:

[0020] The speech signal is denoised to remove noise.

[0021] Endpoint detection is performed on the denoised speech signal to identify the start and end points of the speech.

[0022] Pre-emphasis processing is performed on the speech signal that has passed endpoint detection;

[0023] The pre-emphasized speech signal is divided into speech frames with a frame length of 20ms and a frame shift of 10ms;

[0024] Windowing is applied to the segmented speech frames to improve the quality of the speech signal.

[0025] As a preferred embodiment, the steps for extracting the user's lip shape and tongue position features using facial image recognition technology specifically include:

[0026] Face detection is performed using the MTCNN algorithm based on OpenCV, and the face is automatically segmented.

[0027] The key feature points of the detected face are located by using a convolutional neural network to obtain the positional information of the eyebrows, left eye, right eye, nose tip, and mouth;

[0028] Based on the location information of the key feature points, calculate the three-dimensional coordinate position of the lips in space, the width of the mouth, the height of the mouth, and the position of the center point;

[0029] Calculate the offset between the lower lip and the upper lip;

[0030] Measure the distance between the tip of the tongue and the base of the tongue in a vertical plane to determine the tongue position depth.

[0031] Preferably, the steps for constructing the multimodal topological feature space specifically include:

[0032] A speech feature neighborhood graph is constructed based on the acoustic features of speech frames, and its topological structure features are analyzed.

[0033] Based on lip movement features, a lip feature neighborhood map is constructed, and its topological structure information is extracted.

[0034] Based on the tongue position change characteristics, a tongue position feature neighborhood graph is established to obtain its topological structure representation;

[0035] Analyze the correlations between different modal topologies and construct cross-modal topological relationships;

[0036] The topological features are represented in a multi-level manner, including local structural features, global structural features, and interactive structural features.

[0037] Preferably, the steps for performing manifold consistency mapping specifically include:

[0038] Construct a mapping function that preserves topological relationships to project features of different modalities into a common feature space;

[0039] Set a time window to ensure that modal data with different sampling rates can be effectively aligned;

[0040] Dynamic programming is used to determine the optimal timing alignment path and solve the intermodal delay problem;

[0041] Implement adaptive window adjustment, dynamically changing the window size according to the user's speaking speed;

[0042] Develop a feature buffer management strategy to balance processing latency and synchronization accuracy.

[0043] Preferably, the steps of fusing speech features and visual features through a topology-guided attention mechanism specifically include:

[0044] Calculate the importance index of feature points based on topological structure features;

[0045] Design a topological distance decay function to adapt attention allocation to the feature structure;

[0046] Construct a multi-layered attention mechanism that focuses on local, global, and interactive structural features respectively;

[0047] Implement an incremental attention update mechanism to adapt to the continuous arrival of streaming data;

[0048] The fusion weights of different modalities are dynamically adjusted based on the topological importance index.

[0049] The attention-weighted multimodal features are integrated into a unified fusion feature representation.

[0050] Preferably, the implementation of the joint discrimination model includes:

[0051] The input data consists of multimodal fusion features, and preliminary feature extraction is performed through convolutional layers.

[0052] The extracted features are input into the attention mechanism layer to obtain attention weights;

[0053] Features with attention weights are input into the pooling layer so that feature maps at the same time have the same dimension.

[0054] The feature maps of the same dimension are input into the Bi-LSTM layer and the CNN layer, and further fused into phoneme features;

[0055] The phoneme features are passed through a fully connected layer to output the predicted label;

[0056] By comparing the labels in the standard pronunciation library with the predicted labels, the incorrect segments in the user's pronunciation process can be identified.

[0057] Preferably, paragraphs marked with user pronunciation errors include the following error types:

[0058] Phoneme error: When the predicted phoneme label does not match the corresponding phoneme in the standard pronunciation library;

[0059] Lip shape error: When the lip shape features during a user's pronunciation do not match the lip shape features in a standard pronunciation;

[0060] Tongue position error: When the user's tongue position characteristics during pronunciation do not conform to the tongue position requirements in standard pronunciation;

[0061] Audiovisual mismatch error: When the phoneme corresponding to the visual feature is inconsistent with the actual phoneme emitted.

[0062] As a preferred approach, user feedback and system optimization steps are also included:

[0063] Collect user feedback on treatment recommendations;

[0064] Analyze user behavior patterns to evaluate system effectiveness;

[0065] Based on user feedback, dynamically adjust model parameters and discrimination thresholds;

[0066] High-quality samples are selected and added to the knowledge base to expand the standard pronunciation database;

[0067] For high-quality samples with similarity exceeding a preset threshold, update the model parameters;

[0068] Personalized parameter configurations can be built for different users to improve system adaptability.

[0069] A real-time Mandarin pronunciation correction system based on multimodal streaming learning includes:

[0070] A multimodal data acquisition module is used to acquire user voice signals and facial image information;

[0071] The speech preprocessing module is used to perform noise reduction, endpoint detection, pre-emphasis, framing, and windowing processing on the speech signal.

[0072] The visual feature extraction module is used to extract the user's lip shape and tongue position features based on the OpenCV-based MTCNN algorithm;

[0073] The topological feature representation module is used to construct the topological structure of speech features, lip shape features, and tongue position features, forming a multimodal topological feature space;

[0074] The manifold mapping module is used to perform manifold consistency mapping and establish temporal correspondences between speech features and visual features.

[0075] The topology attention module is used to calculate attention weights based on topological features to achieve the fusion of multimodal features;

[0076] The joint discrimination module is used to perform frame-level phoneme prediction based on fused features and output predicted labels;

[0077] The error detection module compares the predicted labels with the standard pronunciation library and marks the segments with pronunciation errors.

[0078] The correction generation module is used to generate correction suggestions for incorrect paragraphs and play the corrected pronunciation on the audio device;

[0079] The feedback optimization module is used to collect user feedback, dynamically adjust system parameters, and improve treatment effectiveness.

[0080] This invention constructs a multimodal feature space by introducing topological data analysis methods and innovatively designs a manifold consistency mapping and topology-guided attention mechanism, achieving deep fusion and complementary enhancement of speech and visual information. Simultaneously, it employs a streaming learning framework to realize real-time analysis and feedback of user pronunciation, significantly improving learning efficiency and user experience.

[0081] The present invention has the following beneficial effects:

[0082] 1) Multimodal information collaborative analysis: By simultaneously collecting and analyzing the user's voice signals and facial visual information, the acoustic features, lip shape changes and tongue position movements during the pronunciation process are fully grasped, enabling accurate localization and analysis of pronunciation problems.

[0083] 2) Deep integration empowered by topological data analysis: The innovative introduction of topological data analysis methods enables deep integration of multimodal information from the perspective of the inherent structure of data. Compared with traditional feature splicing or weighted averaging methods, it can better uncover the essential connections between different modal data and improve the integration effect.

[0084] 3) Real-time feedback supported by the streaming learning framework: The streaming processing architecture performs frame-level analysis and prediction of user pronunciation, achieving millisecond-level feedback latency, which greatly improves the interactive experience and learning efficiency.

[0085] 4) Personalized adaptation and continuous optimization: By collecting user feedback, the system parameters and model weights are dynamically adjusted to achieve personalized adaptation to the characteristics of different users, and the system performance is continuously improved through continuous learning.

[0086] 5) High-precision error detection and correction: Based on the joint discrimination model of multimodal fusion features, it can accurately identify various pronunciation errors, including phoneme errors, mouth shape errors, tongue position errors and audiovisual mismatch errors, and provide targeted correction suggestions. Attached Figure Description

[0087] Figure 1 This is a diagram illustrating the overall architecture of the real-time Mandarin pronunciation correction system based on multimodal streaming learning, as described in this invention.

[0088] Figure 2 This is a flowchart of the multimodal data acquisition and preprocessing process of the present invention;

[0089] Figure 3 This is a structural diagram of the topological feature representation and manifold mapping module of the present invention;

[0090] Figure 4 This is a schematic diagram of the topology-guided attention mechanism of the present invention;

[0091] Figure 5 This is a network structure diagram of the joint discrimination model of the present invention;

[0092] Figure 6 This is a flowchart of the error detection and correction feedback process of the present invention. Detailed Implementation

[0093] Please refer to Figure 1 - Figure 6 The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. Those skilled in the art should understand that the embodiments described herein are for illustrative purposes only and should not be construed as limiting the invention.

[0094] Reference Figure 1 The real-time Mandarin pronunciation correction system based on multimodal streaming learning provided by the present invention includes a multimodal data acquisition module 10, a speech preprocessing module 20, a visual feature extraction module 30, a topological feature representation module 40, a manifold mapping module 50, a topological attention module 60, a joint discrimination module 70, an error detection module 80, a correction generation module 90, and a feedback optimization module 100.

[0095] The multimodal data acquisition module 10 is used to simultaneously acquire the user's speech signal and facial image information; the speech preprocessing module 20 performs denoising, endpoint detection, pre-emphasis, framing, and windowing processing on the speech signal; the visual feature extraction module 30 extracts the user's lip shape and tongue position features based on the OpenCV MTCNN algorithm; the topological feature representation module 40 constructs the topological structure of speech features, lip shape features, and tongue position features to form a multimodal topological feature space; the manifold mapping module 50 performs manifold consistency mapping to establish the temporal correspondence between speech features and visual features; the topological attention module 60 calculates attention weights based on topological structure features to achieve the fusion of multimodal features; the joint discrimination module 70 performs frame-level phoneme prediction based on the fused features and outputs prediction labels; the error detection module 80 compares the prediction labels with the standard pronunciation library and marks the pronunciation error segments; the correction generation module 90 generates correction suggestions for the error segments and plays the corrected pronunciation on the speech terminal; the feedback optimization module 100 collects user feedback, dynamically adjusts system parameters, and improves the correction effect.

[0096] The real-time Mandarin pronunciation correction method based on multimodal streaming learning provided by this invention includes the following steps:

[0097] In a preferred embodiment of the invention, the system acquires the user's voice signal via a microphone and simultaneously captures the user's facial image via a camera. The voice signal uses a sampling rate of 16kHz and a quantization precision of 16bit to ensure sufficient acoustic detail is captured. The facial image uses a frame rate of 30fps and a resolution of at least 720p to ensure clear capture of changes in lip shape and tongue position.

[0098] Furthermore, to ensure the synchronization of voice and visual data, the system adds a unified timestamp to both data streams, laying the foundation for subsequent multimodal fusion. Preferably, the system also evaluates the acquisition environment, including lighting conditions and background noise, and prompts the user to make adjustments when conditions are not ideal to improve data quality.

[0099] The preprocessing and segmentation steps for speech signals specifically include:

[0100] First, the system performs denoising processing on the acquired speech signal, filtering out background noise and interference signals. This invention preferably employs an adaptive noise suppression algorithm based on spectral subtraction. This algorithm can dynamically adjust filtering parameters according to the characteristics of environmental noise, effectively removing noise while preserving the integrity of the speech signal.

[0101] Secondly, the system performs endpoint detection on the denoised speech signal to accurately identify the start and end points of the speech. This invention employs a dual-threshold endpoint detection algorithm based on short-time energy and zero-crossing rate. An energy threshold Eth and a zero-crossing rate threshold Zth are set. When both the short-time energy and zero-crossing rate of the signal exceed their respective thresholds, it is determined to be a speech segment. Preferably, Eth is set to 2.5 times the average energy of a silent segment, and Zth is set to 1.8 times the average zero-crossing rate of a silent segment. These parameter values ​​have been experimentally verified to achieve good endpoint detection results in most environments.

[0102] Then, the system pre-emphasizes the speech signal after endpoint detection to enhance the high-frequency components and improve the accuracy of subsequent analysis. The transfer function of the pre-emphasis filter is:

[0103] ,

[0104] in, The pre-weighting factor ranges from 0.9 to 0.95, with a preferred value of 0.97. For complex variables in the z-transform; This represents the unit delay operator. Pre-emphasis processing can compensate for the natural attenuation of the high-frequency components of the speech signal during pronunciation, improving the recognizability of high-frequency formants.

[0105] Next, the system divides the pre-emphasized speech signal into speech frames with a frame length of 20ms and a frame shift of 10ms. This parameter setting ensures a 10ms overlap between adjacent frames, guaranteeing both the continuity of the analysis and the ability to capture instantaneous changes in the speech signal.

[0106] Finally, the system performs windowing processing on the segmented speech frames to reduce spectral leakage and improve the accuracy of spectral analysis. This invention preferably uses a Hamming window, the expression of which is:

[0107] ,

[0108] in, This is the sample index, and its value ranges from 0 to N-1. The frame length (in terms of sample points) is used for a sampling rate of 16kHz and a frame length of 20ms. The value is 320; Pi is approximately 3.14159. It is a cosine function.

[0109] The specific steps for extracting a user's lip shape and tongue position features using facial image recognition technology include:

[0110] First, the system performs face detection based on OpenCV's MTCNN (Multi-task Convolutional Neural Network) algorithm, automatically segmenting the face. MTCNN is a three-stage cascaded face detection algorithm, consisting of P-Net (Proposal Network), R-Net (Refinement Network), and O-Net (Output Network), enabling robust face detection under different poses, expressions, and lighting conditions.

[0111] Next, the system uses a convolutional neural network to locate key feature points on the detected face, obtaining the positional information of the eyebrows, left eye, right eye, nose tip, and mouth. This invention preferably uses a 68-point facial feature point model, with 20 feature points in the lip region, which can accurately describe changes in lip shape. For tongue detection, the system additionally focuses on the visible part of the tongue when the mouth is open, extracting the tongue contour through a combination of color segmentation and edge detection.

[0112] Based on the acquired key feature point location information, the system calculates the three-dimensional coordinates of the lips in space, the width, height, and center point of the mouth. The specific calculation method is as follows:

[0113] The three-dimensional coordinates of the lips in space can be represented as:

[0114] ,

[0115] in, The coordinates of the lips are in pixels. The coordinates of the center point of the mouth are in two dimensions, in pixels. This represents the width of the mouth, in pixels. This represents the height of the mouth, in pixels. This is a facial depth estimate, in pixels. This refers to the rotation angle of the lips, expressed in radians. It is a cosine function; It is a sine function; The offset between the lower and upper lips is dimensionless and represents a relative proportion.

[0116] Preferably, the system determines the offset between the lower and upper lips by calculating the ratio of the vertical distances between the upper and lower lip feature points. When the upper and lower lips are symmetrical, The value is close to 0; when the lower lip is stretched downwards, The value is positive; when the upper lip is upturned, The value is negative.

[0117] Simultaneously, the system measures the distance between the tip of the tongue and the base of the tongue in a vertical plane to determine the tongue position depth. It can be calculated in the following ways:

[0118] ,

[0119] in, The depth of the tongue position is expressed in pixels. This is the distance from the tip of the tongue to the reference plane, in pixels. This is the distance from the base of the tongue to a reference plane, in pixels. The reference plane is typically chosen as the plane connecting the upper and lower incisors.

[0120] These geometric parameters enable the system to comprehensively describe the changes in lip shape and tongue position during the user's pronunciation, providing important visual information for subsequent multimodal fusion.

[0121] The specific steps for constructing a multimodal topological feature space include:

[0122] This invention innovatively introduces a topological data analysis method to construct a multimodal feature space from the perspective of the intrinsic structure of data, thereby capturing the essential characteristics and interrelationships of different modal data.

[0123] First, the system constructs a speech feature neighborhood map based on the acoustic features of speech frames and analyzes its topological structure features. Specifically, for each speech frame, an acoustic feature vector is extracted. The system calculates the similarity measure between feature vectors and constructs... Nearest neighbor graph. Cosine similarity is the preferred similarity metric.

[0124] ,

[0125] in, For feature vectors and The cosine similarity between them ranges from [-1, 1]. and They represent the first Frame and the The acoustic feature vector of the frame, with dimension . ; This represents the vector dot product operation; and Representing vectors respectively and The Euclidean norm (i.e., vector length). When the similarity exceeds a threshold... (The preferred value is 0.85) or yes of Neighbors (preferred) When the value is 5, a connection is established in the neighborhood graph.

[0126] Next, the system constructs a lip shape feature neighborhood map based on lip shape motion features and extracts its topological structure information. For the lip shape feature vector... The system uses Euclidean distance as a similarity metric:

[0127] ,

[0128] in, For feature vectors and The Euclidean distance between them is in the same units as the eigenvectors; and They represent the first Frame and the The lip shape feature vector of the frame, with dimension . ; This represents the Euclidean norm (L2 norm). When the distance is less than a threshold... (The preferred value is 0.7 times the average distance between frames) when establishing connections in the neighborhood graph.

[0129] Similarly, the system establishes a tongue position feature neighborhood map based on tongue position change features to obtain its topological structure representation. (Tongue position feature vector) The similarity between them is also measured using Euclidean distance, with a threshold. The preferred setting is 0.75 times the average inter-frame distance.

[0130] After constructing neighborhood graphs for each modality, the system further analyzes the correlations between the topologies of different modalities and constructs cross-modal topological relationships. Specifically, the system calculates the correspondence between features of different modalities in the time dimension. When feature points of two different modalities are close in time (time difference less than a threshold), the system considers these relationships. When the preferred value is 15ms and the pronunciation states are similar, a connection is established in the cross-modal topology.

[0131] Finally, the system represents the topological features at multiple levels, including local structural features, global structural features, and interactive structural features. Local structural features mainly describe the local topological relationships between each feature point and its nearest neighbors, global structural features capture the macroscopic structural characteristics of the entire feature space, and interactive structural features characterize the structural associations between different modal features.

[0132] This topology-based multimodal feature representation method can effectively capture the intrinsic structural information of data, providing a solid mathematical foundation for subsequent feature fusion.

[0133] The specific steps for performing manifold consistency mapping include:

[0134] To address the heterogeneity and temporal inconsistency issues of multimodal data, this invention designs a feature mapping mechanism based on manifold consistency.

[0135] First, the system constructs a mapping function that preserves topological relationships, projecting different modal features onto a common feature space. The design goal of the mapping function is to achieve a unified representation of different modal features while maintaining topological relationships. This invention employs the Laplace eigenmap method, with the objective function being:

[0136] ,

[0137] in, The feature matrix after mapping, The total number of feature points. For mapping spatial dimensions; and The first The and the first The representation of each feature point in the mapping space, with dimension . ; For feature points and The topological relationship weights between feature points and The value is 1 if there are connections in the topology, and 0 otherwise. Represents the square of the Euclidean distance; This indicates that for all feature point pairs Perform summation.

[0138] To ensure the uniqueness of the mapping, add the following constraints:

[0139] ,

[0140] in, Representation matrix Transpose of; It is a diagonal matrix, and its diagonal elements Representing feature points degree; for 3D identity matrix; This indicates the equality relationship between matrices.

[0141] By solving the eigenvalue decomposition of the above optimization problem, the mapping function can be obtained. Preferably, the system selects the eigenvector corresponding to the smallest non-zero eigenvalue as the mapping result. The mapping dimension is determined according to the amount of information to be retained, and usually a dimension that can retain 90% of the information is selected (an empirical value is 1 / 3 to 1 / 2 of the original feature dimension).

[0142] Next, the system sets a time window to ensure that modal data with different sampling rates can be effectively aligned. Considering that the audio signal frame length is 20ms and the frame shift is 10ms, while the visual signal sampling rate is 30fps (approximately 33.3ms per frame), the system sets a time window width of 50ms, which can cover the smallest common time unit of audio and visual signals.

[0143] Then, the system employs dynamic programming to determine the optimal timing alignment path, addressing the intermodal latency issue. Specifically, the system constructs a cumulative distance matrix. ,in This indicates that the first part of the speech feature sequence Frame and visual feature sequence before The cumulative distance for optimal frame alignment. The recursive formula is:

[0144] ,

[0145] in, The elements of the cumulative distance matrix represent the first few elements of the speech feature sequence. Frame and visual feature sequence before The cumulative distance for optimal frame alignment; For speech feature frames With visual feature frames The preferred distance metric between them is the Euclidean distance in the mapping space. This indicates taking the minimum value among the three values ​​in parentheses; , and 1) These represent the element values ​​at the corresponding positions in the cumulative distance matrix, used for recursive calculation. The initial conditions are... , , .

[0146] By backtracking the cumulative distance matrix, the system obtains the optimal alignment path, achieving temporal consistency of data from different modalities.

[0147] Furthermore, the system implements adaptive window adjustment, dynamically changing the window size according to the user's pronunciation speed. When the system detects that the user's pronunciation speed is too fast (the time interval between adjacent phonemes is less than a threshold), it will adjust the window size accordingly. (Preferred value: 60ms) The system reduces the window size to 35ms; when the pronunciation speed is slow (the time interval between adjacent phonemes is greater than the threshold), (The preferred value is 120ms), and the window size is increased to 65ms.

[0148] Finally, the system implements a feature buffer management strategy to balance processing latency and synchronization accuracy. The buffer size is set to 200ms, storing the most recent 20 frames of speech features and 6 frames of visual features. The system employs a sliding window mechanism, updating the buffer contents and recalculating the alignment relationship each time new data arrives, ensuring a balance between real-time performance and accuracy.

[0149] The steps for fusing speech and visual features using a topology-guided attention mechanism specifically include:

[0150] The most innovative part of this invention is the design of an attention mechanism based on topology structure, which enables intelligent fusion of multimodal features.

[0151] First, the system calculates the importance index of feature points based on topological features. For each feature point... Its topological importance index Defined as:

[0152] ,

[0153] in, For feature points The topological importance index is dimensionless; For feature points and The topological relationship weights between feature points and The value is 1 if there are connections in the topology, and 0 otherwise. For feature points The topological persistence of a feature point is dimensionless and represents its stability within the topological structure. This indicates that for all feature points Connected feature points Summation is performed. Topological persistence is usually obtained through persistent cohomology calculation, reflecting the structural importance of features at different scales.

[0154] Next, the system designs a topological distance decay function to adapt attention allocation to the feature structure. The decay function is defined as follows:

[0155] ,

[0156] in, For feature points and The attenuation coefficient between them has a value range of (0,1]; Representing feature points and The topological distance between them, that is, from the topological graph... arrive The shortest path length, expressed in terms of the number of edges; The kernel width parameter controls the decay rate; the unit is . Consistent; It is a natural exponential function. Preferably, The value is dynamically adjusted based on topological persistence; when the topological persistence of a feature point is high, A larger value leads to a more concentrated attention distribution; when topological persistence is low, A smaller value indicates a more even distribution of attention. Empirically, The value range is from 0.5 to 2.0, and the default value is 1.0.

[0157] Then, the system constructs a multi-layered attention mechanism, focusing on local, global, and interaction structural features respectively. For each structural feature, the system calculates the corresponding attention weights:

[0158] ,

[0159] ,

[0160] ,

[0161] in, , and These are the attention weights at the local, global, and interaction levels, respectively. All are dimensionless, taking values ​​in the range [0,1], and for a fixed... , (type can be local, global, or inter); , and These are the attention energy functions at the local, global, and interaction levels, respectively, and are typically designed as a combination of feature vector similarity metrics and topological distance decay functions; exp is the natural exponential function. This represents summing over all feature points k, used to normalize the attention weights.

[0162] The system implements an incremental attention update mechanism to adapt to the continuous arrival of streaming data. For newly arrived feature points... Its attention weights are updated as follows:

[0163] ,

[0164] in, The updated attention weights; Historical attention weights; These are the attention weights calculated so far; This is a smoothing factor, with a value range of [0,1], which controls the proportion of historical information retained. Preferably, The value is set between 0.2 and 0.4, with a default value of 0.3.

[0165] The fusion weights for different modalities are dynamically adjusted based on topological importance metrics. (Fusing weights) Defined as:

[0166] ,

[0167] ,

[0168] in, and are the fusion weights for speech features and visual features, respectively, both dimensionless, with values ​​ranging from [0,1]. ; and These represent the speech feature set and the visual feature set, respectively. For feature points The topological importance index; This represents the summation of all feature points in the speech feature set; This represents the summation of all feature points in the visual feature set.

[0169] Finally, the system integrates the attention-weighted multimodal features into a unified fusion feature representation:

[0170] ,

[0171] in, The fused feature representation has a dimension of ; and These are speech features and visual features, respectively, and after mapping, all have the same dimension. ; and These are the corresponding attention weights; and These represent weighted summations of all feature points in the speech feature set and the visual feature set, respectively.

[0172] This topology-based attention mechanism can intelligently allocate attention resources according to the inherent structure of the data, significantly improving the fusion effect. In practical applications, the system dynamically adjusts the attention allocation strategy according to different pronunciation scenarios. For example, it increases the weight of visual features in noisy environments and increases the weight of speech features in low light conditions, achieving robust multimodal fusion.

[0173] The implementation of the joint discriminant model specifically includes:

[0174] This invention designs an end-to-end joint discrimination model that performs frame-level phoneme prediction based on multimodal fusion features.

[0175] First, the input data consists of multimodal fusion features. Preliminary feature extraction is performed through convolutional layers. The convolutional layers employ a 1D convolutional structure with a kernel size of 3, a stride of 1, and 128 channels. The convolution operation is represented as follows:

[0176] ,

[0177] in, The feature map output by the convolutional layer has a dimension of . , For time steps; For multimodal fusion features, the dimension is ; This represents a one-dimensional convolution operation; and These are the weights and bias parameters of the convolutional layer, respectively; To modify the activation function of the linear unit, it is defined as follows: .

[0178] Next, the extracted features are input into the attention mechanism layer to obtain attention weights. The attention mechanism uses a self-attention structure to calculate the correlation between a feature and itself, as shown in the following formula:

[0179] ,

[0180] ,

[0181] ,

[0182] in, Attention energy value, representing location Position Attention intensity; , and These are the weight matrices for the query, key, and value, respectively, with each matrix having a dimension of 1. ; Indicates position The query vector, with dimensions of ; Indicates position The key vector, with dimension ; Let be the dimension of the key vector, and let be the value. ; Represents the dot product of vectors; This is a scaling factor used to stabilize the gradient; Attention weights, representing positions Position Attention allocation, with values ​​ranging from And for fixed , ; It is a natural exponential function; Indicates all positions Summation is performed to normalize the attention weights; Represents a value matrix, Indicates position A value vector with dimension . ; The output of the attention mechanism layer has a dimension of ; Indicates all positions Perform a weighted summation.

[0183] Then, the features with attention weights are input into the pooling layer, ensuring that the feature maps at the same time have the same dimension. The pooling layer uses adaptive average pooling to normalize feature sequences of different lengths to a fixed length (preferably 64), ensuring consistency in subsequent processing.

[0184] Next, the feature maps of uniform dimension are input into the Bi-LSTM layer and the CNN layer for further fusion into phoneme features. The Bi-LSTM layer has 256 hidden units, and the CNN layer uses a multi-scale convolutional structure with kernel sizes of 3, 5, and 7, and 64 kernels of each size. The outputs of the Bi-LSTM and CNN are then concatenated and fused.

[0185] ,

[0186] ,

[0187] ,

[0188] in, The output of the Bi-LSTM layer has a dimension of (Concatenation of the outputs of forward and backward LSTM); Bi-LSTM represents bidirectional long short-term memory network operation; and These are the weights and bias parameters of the Bi-LSTM layer, respectively; The output of the CNN layer has a dimension of (The outputs of three types of convolutional kernels are concatenated); CNN represents the operation of a convolutional neural network; and These are the weights and bias parameters of the CNN layer, respectively; The fused phoneme features have the following dimensions: Concat represents a tensor concatenation operation, which concatenates tensors along the feature dimension.

[0189] Finally, the phoneme features are passed through a fully connected layer to output predicted labels. The output dimension of the fully connected layer is equal to the number of phoneme categories (for Mandarin, a preferred value is 65, including combinations of initials, finals, and tones). The Softmax function is applied to obtain the predicted probability for each phoneme.

[0190] ,

[0191] in, The predicted phoneme probability distribution has a dimension of . Each row represents the phoneme probability distribution at a time step, and the sum of the elements in each row is 1; and These are the weights and bias parameters of the fully connected layer, with dimensions of [dimensions to be filled in]. And 65; Softmax is the normalized exponential function, defined as , is used to convert a real vector into a probability distribution.

[0192] The predicted label is determined by taking the category with the highest probability:

[0193] ,

[0194] in, The predicted phoneme label, with values ​​ranging from 0 to 64; Indicates that returning makes Index of the maximum value ; This represents the value of the i-th element in the predicted probability distribution.

[0195] The system compares the predicted labels with the labels in the standard pronunciation database to identify the incorrect segments in the user's pronunciation process. The standard pronunciation database contains the standard pronunciation features of each phoneme in standard Mandarin, recorded and annotated by professional Mandarin teachers. When the predicted label does not match the corresponding label in the standard pronunciation database, the system marks that segment as a pronunciation error.

[0196] This invention can identify and mark various types of pronunciation errors, including:

[0197] 1) Phoneme error: When the predicted phoneme label does not match the corresponding phoneme in the standard pronunciation library, the system marks this segment as a phoneme error. Phoneme errors may manifest as errors in initial consonants, final vowels, or tones. For example, mispronouncing zh as z, in as ing, or the fourth tone as the second tone, etc. The system will specifically mark the type and position of the incorrect phoneme, providing precise guidance for subsequent correction.

[0198] 2) Lip shape error: When the lip shape characteristics during the user's pronunciation cannot match the lip shape characteristics in the standard pronunciation, the system marks this segment as a lip shape error. Lip shape errors usually manifest as incomplete or incorrect lip shapes. For example, when pronouncing the u sound, the lips are not fully rounded, or when pronouncing the i sound, the lips are not fully spread, etc. The system calculates the similarity score by comparing the user's lip shape characteristics with the lip shape templates in the standard pronunciation library. When the score is lower than the threshold (preferred value is 0.75), it is determined as a lip shape error.

[0199] 3) Tongue position error: When the tongue position characteristics during the user's pronunciation do not meet the tongue position requirements in the standard pronunciation, the system marks this segment as a tongue position error. Tongue position errors are common in the pronunciation of apical consonants, palatal consonants, and velar consonants. For example, when pronouncing the s sound, the position of the tip of the tongue is inaccurate, or when pronouncing the g sound, the back of the tongue is not lifted sufficiently, etc. The system determines a tongue position error when the difference exceeds the threshold (preferred value is 1.5 times the standard deviation of the tongue position depth) by comparing the user's tongue position characteristics with the standard template.

[0200] 4) Audio-visual mismatch error: When the phoneme corresponding to the visual characteristics is inconsistent with the actually pronounced phoneme, the system marks this segment as an audio-visual mismatch error. Such errors usually indicate that the user's lip shape and actual pronunciation do not match, which may be caused by incorrect pronunciation habits or inaccurate understanding. The system determines an audio-visual mismatch error when the phoneme categories predicted by the two are different and the difference is significant (confidence difference is greater than 0.3) by analyzing the consistency between the speech recognition result and the visual feature prediction result.

[0201] For each type of error, the system will record detailed error information, including the time period when the error occurred, the type of error, the severity of the error, and the corresponding standard pronunciation reference. This information provides a basis for generating subsequent correction suggestions.

[0202] After determining the paragraphs with the user's pronunciation errors, the system will generate corresponding correction suggestions for different types of errors and play the corrected user pronunciation at the speech end.

[0203] For phoneme errors, the system will provide a demonstration pronunciation of the correct phoneme and combine it with text explanations to guide the user on how to pronounce correctly. For example, for the case of mispronouncing zh as z, the system will explain the difference between the two (the former has the tip of the tongue against the upper palate, and the latter has the tip of the tongue against the lower gum), and provide typical word examples (such as "zhi" and "zi") for contrastive practice.

[0204] For incorrect lip movements, the system will provide a standard lip-probe image or video, accompanied by animation to highlight key changes in lip shape. The system will also offer specific suggestions for adjusting the lip shape, such as slightly widening the mouth or spreading the lips to the sides.

[0205] For incorrect tongue positioning, the system will display the correct tongue position using a side profile diagram and provide perceptual training methods to help users establish correct tongue position awareness. For example, for training apical consonants, the system will suggest that users first feel the position of the tip of the tongue touching the hard palate, and then gradually transition to the pronunciation of the target phoneme.

[0206] For audio-visual mismatch errors, the system will simultaneously display the correct form of the speech and lip movements, and guide the user to coordinate lip movements and pronunciation to establish the correct pronunciation-lip movement mapping relationship.

[0207] When playing the corrected user pronunciation on the voice terminal, the system seamlessly splices the user's original pronunciation with the standard pronunciation to form a complete speech stream. To maintain timbre consistency, the system uses sound conversion technology to adjust the timbre of the standard pronunciation to be close to that of the user, reducing auditory inconsistencies.

[0208] Preferably, the system will also design progressive practice plans, from simple monophone exercises to complex word and sentence exercises, to help users gradually master the correct pronunciation methods. The system will record the user's practice history and progress, dynamically adjusting the difficulty and focus of the practice content to achieve a personalized learning experience.

[0209] This invention also includes user feedback and system optimization steps, specifically including:

[0210] First, the system collects user feedback on the treatment suggestions. This feedback can be explicit, such as users evaluating the effectiveness of the suggestions via interface buttons (e.g., helpful, partially helpful, not helpful); or implicit, such as the system monitoring whether the user has practiced according to the suggestions and whether their pronunciation has improved after practice.

[0211] Secondly, the system analyzes user behavior patterns to evaluate its effectiveness. The system records user behavior data such as usage frequency, duration of each session, and number of repetitions, combining this data with pronunciation improvement to comprehensively assess the system's actual effectiveness. The system also identifies user learning habits and preferences, such as whether they prefer visual or auditory guidance, or short or long practice sessions, providing a basis for subsequent personalized optimization.

[0212] Then, based on user feedback, the system dynamically adjusts model parameters and discrimination thresholds. For example, when the detection of a certain type of error frequently triggers negative user feedback, the system will appropriately adjust the discrimination threshold for that type of error to reduce false alarms; when a certain corrective suggestion receives positive feedback, the system will increase the weight of that type of suggestion, increasing its frequency of occurrence. Specifically, the system adopts an adaptive threshold adjustment strategy:

[0213] ,

[0214] in, The adjusted threshold, and Units are consistent; The threshold before adjustment; The learning rate has a range of values: 1 represents completely negative feedback, 1 represents completely positive feedback, and 0.5 represents neutral feedback. The feedback scores are mapped to the [-1,1] interval, so that positive feedback increases the threshold and negative feedback decreases the threshold.

[0215] Next, the system selects high-quality samples to add to the knowledge base, expanding the standard pronunciation database. When a user's pronunciation is determined by the system to be correct and representative, the system adds that sample to the candidate sample pool. For candidate samples, the system calculates their similarity to existing samples to ensure sample diversity and representativeness.

[0216] For high-quality samples with similarity exceeding a preset threshold, the system updates the model parameters. Specifically, the system employs an incremental learning strategy, fine-tuning the model using mini-batch gradient descent.

[0217] ,

[0218] in, These are the updated model parameters; These are the model parameters before the update; The learning rate, with a value ranging from (0,1], controls the magnitude of the update, with an optimal value of 0.001. Indicates about parameters The gradient operator; The loss function is typically cross-entropy loss; For the features and labels of the new samples; Indicates the use of parameters For new samples The calculated loss value; This represents the gradient of the loss function with respect to the parameters.

[0219] Finally, the system builds personalized parameter configurations for different users to improve system adaptability. The system constructs user profiles based on information such as the user's learning progress, pronunciation characteristics, and error patterns, and adjusts system parameters accordingly, such as the sensitivity of error detection, the level of detail in correction suggestions, and the speed of progression in practice difficulty. Personalized parameters are determined through collaborative filtering and transfer learning methods, that is, referencing effective configurations from similar users and fine-tuning them based on the characteristics of the current user.

[0220] Through this closed-loop feedback and optimization mechanism, the system can continuously improve the treatment effect, adapt to the needs of different users, and achieve continuous self-improvement.

[0221] The real-time Mandarin pronunciation correction system based on multimodal streaming learning provided by this invention includes a multimodal data acquisition module, a speech preprocessing module, a visual feature extraction module, a topological feature representation module, a manifold mapping module, a topological attention module, a joint discrimination module, an error detection module, a correction generation module, and a feedback optimization module.

[0222] The multimodal data acquisition module is used to acquire user voice signals and facial image information. This module includes a high-quality microphone and a camera, capable of simultaneously acquiring the user's voice and facial images. The microphone uses a 16kHz sampling rate and 16-bit quantization accuracy, and features noise reduction, enabling it to acquire clear voice signals in general environments. The camera operates at a 30fps frame rate, with a resolution of at least 720p, and is equipped with autofocus, clearly capturing subtle changes in the user's face.

[0223] The speech preprocessing module performs denoising, endpoint detection, pre-emphasis, framing, and windowing on the speech signal. This module implements the preprocessing steps described above, effectively improving the quality of the speech signal and laying the foundation for subsequent analysis. In particular, this module employs an adaptive noise suppression algorithm, which dynamically adjusts filtering parameters according to environmental noise characteristics, effectively removing noise while preserving the integrity of the speech signal.

[0224] The visual feature extraction module is used to extract user lip shape and tongue position features based on the OpenCV-based MTCNN algorithm. This module implements the feature extraction steps described in claim 3, enabling precise localization of facial key points and calculation of the geometric features of the lips and tongue. The built-in MTCNN algorithm employs a three-stage cascaded structure, achieving robust face detection and feature point localization under different poses, expressions, and lighting conditions.

[0225] The topological feature representation module is used to construct the topological structure of speech features, lip shape features, and tongue position features, forming a multimodal topological feature space. This module implements the topological feature construction steps described in claim 4, and can capture the essential characteristics and interrelationships of different modal features from the perspective of the intrinsic structure of the data. The module adopts an efficient topological calculation algorithm, supports real-time processing, and controls the computational complexity to the O(nlogn) level.

[0226] The manifold mapping module performs manifold consistency mapping, establishing temporal correspondences between speech and visual features. This module implements the aforementioned manifold mapping steps, addressing the heterogeneity and temporal inconsistencies of multimodal data, thus laying the foundation for subsequent fusion. The module employs the Laplacian eigenmap method, solving the mapping function through eigenvalue decomposition and combining it with dynamic programming to determine the optimal temporal alignment path.

[0227] The topology attention module is used to calculate attention weights based on topological features, enabling the fusion of multimodal features. This module implements the attention mechanism described above, intelligently allocating attention resources according to the inherent structure of the data to achieve efficient multimodal fusion. The module designs a multi-level attention mechanism and an incremental update strategy, adapting to the continuous arrival of streaming data and maintaining real-time performance.

[0228] The joint discrimination module is used for frame-level phoneme prediction based on fused features, outputting predicted labels. This module implements the discrimination model described above, employing an end-to-end deep learning architecture, including convolutional layers, attention mechanism layers, pooling layers, Bi-LSTM layers, CNN layers, and fully connected layers, capable of accurately predicting the phoneme category for each frame. The module uses a hybrid precision training strategy to improve computational efficiency while maintaining accuracy, supporting real-time inference.

[0229] The error detection module compares predicted labels with the standard pronunciation library, marking segments with pronunciation errors. This module implements the error detection functions described above, capable of identifying various types of pronunciation errors, including phoneme errors, mouth shape errors, tongue position errors, and sound-sight mismatch errors, and recording detailed error information. The module's built-in standard pronunciation library contains all common phoneme combinations in Mandarin, recorded and annotated by professional Mandarin teachers to ensure the authority of the reference standard.

[0230] The correction generation module generates correction suggestions for incorrect passages and plays the corrected pronunciation on the audio input device. This module can generate corresponding correction suggestions based on different types of errors, including text descriptions, demonstration pronunciations, mouth shape demonstrations, and tongue position guidance. The module uses voice conversion technology to adjust the timbre of the standard pronunciation to closely resemble the user's voice, ensuring a natural and consistent playback of the corrected pronunciation.

[0231] The feedback optimization module is used to collect user feedback, dynamically adjust system parameters, and improve treatment effectiveness. This module implements the feedback and optimization functions described above. By collecting explicit and implicit user feedback, analyzing user behavior patterns, dynamically adjusting model parameters and discrimination thresholds, selecting high-quality samples to expand the knowledge base, and constructing personalized parameter configurations, it achieves continuous optimization and personalized adaptation of the system.

[0232] The aforementioned modules are connected via a pipelined architecture to form an end-to-end real-time processing system. The system employs a multi-threaded parallel processing strategy: the acquisition, preprocessing, and feature extraction of speech and visual data are performed in independent threads, while multimodal fusion and discrimination are performed in the main thread, ensuring real-time performance. The system's end-to-end latency is controlled within 150ms, meeting the requirements of real-time interaction.

[0233] The real-time Mandarin pronunciation correction method and system based on multimodal streaming learning provided by this invention have broad application prospects. In the field of language education, it can be used in Mandarin training institutions, primary and secondary school Chinese language teaching, foreign language learning centers, and other scenarios to provide accurate pronunciation guidance. In the field of speech rehabilitation, it can assist in the treatment process of speech disorder rehabilitation, stuttering correction, and speech training for hearing-impaired individuals. In distance education platforms, it can provide automated pronunciation assessment and guidance, support teachers to remotely guide students' pronunciation, and provide professional-level pronunciation guidance for self-learners.

[0234] Experimental results demonstrate that this system significantly outperforms existing technologies in error detection accuracy, real-time performance, and learning efficiency. The error detection accuracy exceeds 92%, end-to-end latency is controlled within 150ms, user learning efficiency is improved by 30% compared to traditional methods, 80% of pronunciation errors are corrected within 5 practice sessions, and the learning effect remains stable after 3 months. These data indicate that this invention not only possesses technological innovation but also significant practical value.

[0235] The embodiments described above are merely illustrative of specific implementations of the present invention, and while the descriptions are detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A method for real-time correction of mandarin pronunciation based on multi-modal streaming learning, characterized in that, The steps include the following: Collect user voice signals and facial image information; The speech signal is denoised to remove noise; endpoint detection is performed on the denoised speech signal to identify the start and end points of the speech; the speech signal after endpoint detection is pre-emphasized; the pre-emphasized speech signal is divided into speech frames with a frame length of 20ms and a frame shift of 10ms; the segmented speech frames are windowed to obtain a speech frame sequence. The facial image recognition technology is used to extract the user's lip shape and tongue position features; A multimodal topological feature space is constructed, comprising speech feature topology, lip shape feature topology, and tongue position feature topology. The steps for constructing the multimodal topological feature space include: establishing a speech feature neighborhood map based on the acoustic features of the speech frame and analyzing its topological features; establishing a lip shape feature neighborhood map based on lip movement features and extracting its topological information; establishing a tongue position feature neighborhood map based on tongue position change features and obtaining its topological representation; analyzing the correlation between different modal topological structures and constructing cross-modal topological relationships; and representing the topological features at multiple levels, including local structural features, global structural features, and interactive structural features. Based on the multimodal topological feature space, manifold consistency mapping is performed to establish a temporal correspondence between speech features and visual features; Based on the manifold consistency mapping result, speech features and visual features are fused through a topology-guided attention mechanism to generate multimodal fusion features. The steps of the topology-guided attention mechanism include: calculating the importance index of feature points based on topological structure features; designing a topological distance decay function to adapt the attention allocation to the feature structure; dynamically adjusting the fusion weights of different modalities based on the topological importance index; and integrating the attention-weighted multimodal features into a unified fusion feature representation. Based on the multimodal fusion features, the end-to-end trained joint discrimination model is invoked to perform frame-level prediction, obtaining the predicted label of each phoneme in the user's pronunciation process; the joint discrimination model includes, in sequence, a convolutional layer, an attention mechanism layer, a pooling layer, a Bi-LSTM layer, a CNN layer, and a fully connected layer; Based on the predicted tags and the standard pronunciation library, mark the paragraphs in which the user pronounces them incorrectly; The system generates correction suggestions for the segments with pronunciation errors and plays the corrected pronunciation from the user on the audio device.

2. The real-time Mandarin pronunciation correction method based on multimodal streaming learning according to claim 1, characterized in that: The denoising process for the speech signal specifically employs an adaptive noise suppression algorithm based on spectral subtraction. The endpoint detection adopts a dual-threshold endpoint detection algorithm based on short-time energy and zero-crossing rate. The energy threshold is set to 2.5 times the average energy of the silent segment, and the zero-crossing rate threshold is set to 1.8 times the average zero-crossing rate of the silent segment. The transfer function of the pre-emphasis filter used in the pre-emphasis processing is: ， in: Let be the transfer function of the pre-emphasis filter; This is the pre-emphasis coefficient; for Complex variables in transformation; Represents the unit delay operator; The windowing process uses a Hamming window, whose expression is: ， in: The window function value for the Hamming window; For sample index; The frame length.

3. The method for real-time correction of Mandarin pronunciation based on multimodal streaming learning according to claim 1, characterized in that, The specific steps for extracting a user's lip shape and tongue position features using facial image recognition technology include: Face detection is performed using the MTCNN algorithm based on OpenCV, and the face is automatically segmented. The key feature points of the detected face are located by using a convolutional neural network to obtain the positional information of the eyebrows, left eye, right eye, nose tip, and mouth; Based on the location information of the key feature points, calculate the three-dimensional coordinate position of the lips in space, the width of the mouth, the height of the mouth, and the position of the center point; Calculate the offset between the lower lip and the upper lip; The tongue position depth is determined by measuring the distance between the tip of the tongue and the base of the tongue in a vertical plane. The tongue position depth is calculated as follows: ， in: This refers to the depth of the tongue position. This is the distance from the tip of the tongue to the reference plane; This is the distance from the root of the tongue to the same reference plane.

4. The method for real-time correction of Mandarin pronunciation based on multimodal streaming learning according to claim 1, characterized in that, The specific steps for performing manifold consistency mapping include: Construct a mapping function that preserves topological relationships to project features of different modalities into a common feature space; Set a time window to ensure that modal data with different sampling rates can be effectively aligned; Dynamic programming is used to determine the optimal timing alignment path and solve the intermodal delay problem; Implement adaptive window adjustment, dynamically changing the window size according to the user's speaking speed; Develop a feature buffer management strategy to balance processing latency and synchronization accuracy.

5. The method for real-time correction of Mandarin pronunciation based on multimodal streaming learning according to claim 1, characterized in that, The steps of fusing speech and visual features through topology-guided attention mechanisms also include: Construct a multi-layered attention mechanism that focuses on local, global, and interactive structural features respectively; Implement an incremental attention update mechanism to adapt to the continuous arrival of streaming data; The feature points Topological importance index Defined as: ， in: For feature points The topological importance index; For feature points and Topological relationship weights between them; For feature points Topological persistence; The topological distance decay function is defined as follows: ， in: For feature points and The attenuation coefficient between them; Representing feature points and Topological distance between them; This is the kernel width parameter.

6. The method for real-time correction of Mandarin pronunciation based on multimodal streaming learning according to claim 1, characterized in that, The implementation of the joint discriminant model specifically includes: The input data consists of multimodal fusion features, and preliminary feature extraction is performed through convolutional layers. The extracted features are input into the attention mechanism layer to obtain attention weights; Features with attention weights are input into the pooling layer so that feature maps at the same time have the same dimension. The feature maps of the same dimension are input into the Bi-LSTM layer and the CNN layer, and further fused into phoneme features; The phoneme features are passed through a fully connected layer to output the predicted label; By comparing the labels in the standard pronunciation library with the predicted labels, the incorrect segments in the user's pronunciation process can be identified.

7. The method for real-time correction of Mandarin pronunciation based on multimodal streaming learning according to claim 1, characterized in that, The paragraphs marked with user pronunciation errors include the following error types: Phoneme error: When the predicted phoneme label does not match the corresponding phoneme in the standard pronunciation library; Lip shape error: When the lip shape features during a user's pronunciation do not match the lip shape features in a standard pronunciation; Tongue position error: When the user's tongue position characteristics during pronunciation do not conform to the tongue position requirements in standard pronunciation; Audiovisual mismatch error: When the phoneme corresponding to the visual feature is inconsistent with the actual phoneme emitted.

8. The method for real-time correction of Mandarin pronunciation based on multimodal streaming learning according to claim 1, characterized in that, It also includes user feedback and system optimization steps: Collect user feedback on treatment recommendations; Analyze user behavior patterns to evaluate system effectiveness; Based on user feedback, dynamically adjust model parameters and discrimination thresholds; High-quality samples are selected and added to the knowledge base to expand the standard pronunciation database; For high-quality samples with similarity exceeding a preset threshold, update the model parameters; Personalized parameter configurations can be built for different users to improve system adaptability.

9. A real-time Mandarin pronunciation correction system based on multimodal streaming learning, used to implement the real-time Mandarin pronunciation correction method based on multimodal streaming learning as described in any one of claims 1 to 8, characterized in that, include: A multimodal data acquisition module is used to acquire user voice signals and facial image information; The speech preprocessing module is used to perform noise reduction, endpoint detection, pre-emphasis, framing, and windowing processing on the speech signal. The visual feature extraction module is used to extract the user's lip shape and tongue position features based on the OpenCV-based MTCNN algorithm; The topological feature representation module is used to construct speech feature topology, lip shape feature topology, and tongue position feature topology to form a multimodal topological feature space. The manifold mapping module is used to perform manifold consistency mapping and establish temporal correspondences between speech features and visual features. The topology attention module is used to calculate attention weights based on topological features to achieve the fusion of multimodal features; The joint discrimination module is used to perform frame-level phoneme prediction based on fused features and output predicted labels; The error detection module compares the predicted labels with the standard pronunciation library and marks the segments with pronunciation errors. The correction generation module is used to generate correction suggestions for incorrect paragraphs and play the corrected pronunciation on the audio device; The feedback optimization module is used to collect user feedback, dynamically adjust system parameters, and improve treatment effectiveness.

Images