A Mandarin pronunciation evaluation system based on deep learning

Abstract

This application discloses a Mandarin pronunciation assessment system based on deep learning, which relates to the technical field of speech scoring. The system includes a data input port for obtaining a transcript, a standard pitch curve, and a standard lip shape video, capturing the user's facial lip shape movements and sounds during reading as a video, and extracting audio from the video; a data processor for processing the video and audio; a pitch evaluator for evaluating the audio to obtain a pitch score; a lip shape evaluator for obtaining a lip shape score; a pronunciation evaluator for obtaining a pronunciation score; and a score output port for combining the pitch score, lip shape score, and pronunciation score to generate a final score. This application has the effect of improving the accuracy and adaptability of Mandarin pronunciation scoring.

Description

Technical Field

[0001] The present application relates to the technical field of speech scoring, and in particular to a Mandarin pronunciation evaluation system based on deep learning. Background Art

[0002] Mandarin pronunciation and lip shape assessment has important application value in language teaching, speech recognition, oral examinations and other fields.

[0003] In the existing technology, there are many difficulties in the process of pronunciation scoring, such as insufficient pronunciation scoring methods, defects in lip shape evaluation technology, and challenges in multimodal fusion. In the pronunciation scoring method, the traditional approach relies on manual feature extraction, forced alignment errors will accumulate, and the correlation between lip shape and pronunciation is not taken seriously, which in turn affects the accuracy of pronunciation scoring. In the process of lip shape evaluation, traditional lip shape scoring judges accuracy by manually defining the distance threshold of lip key points. It cannot adapt to the facial structure and pronunciation habits of different people when speaking. At the same time, the asynchrony of audio and video will cause errors in matching lip shape and phonemes. In the process of multimodal fusion, the feature dimensions between different data are quite different. Direct fusion is prone to introduce noise. At the same time, during the data processing process, error transmission and poor real-time performance will occur, which ultimately leads to inaccurate scoring. Summary of the Invention

[0004] The purpose of the present invention is to provide a Mandarin pronunciation evaluation system based on deep learning to solve the problems raised in the above background technology.

[0005] This application provides a Mandarin pronunciation evaluation system based on deep learning, the system comprising:

[0006] Data input port: used to obtain a transcript, a standard pitch curve, and a standard lip shape video, capture the user's facial lip shape movements and voice when reading as a video, and extract audio from the video;

[0007] A data processor is used to process the video and the audio to obtain a feature vector of the audio and a test single-word video and a standard single-word video of the video;

[0008] A tone evaluator is configured to evaluate the audio according to the feature vector to obtain a tone score;

[0009] Lip shape evaluator: used to perform similarity matching on the test single-word video and the standard single-word video to obtain a lip shape score;

[0010] A pronunciation evaluator is configured to generate a standard pronunciation feature vector based on the transcript, extract a test pronunciation feature vector based on the audio, perform similarity matching on the standard pronunciation feature vector and the test pronunciation feature vector to obtain a pronunciation score;

[0011] Scoring output terminal: used to combine the pitch score, the lip shape score and the pronunciation score to generate a final score.

[0012] Preferably, the standard pitch curve is:

[0013] Load pre-prepared pitch data and corresponding scoring data, including pitch contour data of different speakers;

[0014] Extracting the gender and specific tone of the speaker, screening according to the gender and the specific tone to obtain a pronunciation sample, and extracting a pronunciation score of the pronunciation sample;

[0015] A target pronunciation sample having a pronunciation score higher than a preset score threshold is selected, a polynomial regression fitting is performed on the target pronunciation sample to obtain an average regression parameter, and a standard pitch curve is obtained according to the average regression parameter.

[0016] Preferably, the data processor includes:

[0017] Forced aligner, pitch detector, and pitch feature extractor;

[0018] A forced aligner is configured to input the transcript, the audio, the standard pitch curve, and a preset tone sequence into the forced aligner, wherein the forced aligner extracts the audio timestamp data corresponding to each word in the transcript in the audio, and segments the audio according to the audio timestamp data to obtain audio segments;

[0019] Pitch detector: used to perform pitch detection on the audio segment of each word to obtain a test pitch curve corresponding to the pronunciation of each word;

[0020] Pitch feature extractor: used to perform multi-dimensional comparison between the test pitch curve and the standard pitch curve, extract the maximum value, minimum value, mean value, standard deviation, curve length, dynamic time warping score, pitch boundary score and polynomial regression features in the test pitch curve, and organize them to obtain a feature vector;

[0021] The dynamic time warping score is specifically:

[0022] Extract the first-order difference of the test pitch curve, use DTW to calculate the DTW distance between the test pitch curve and the standard pitch curve, and normalize the distance The distance is converted to a score (range 0~100) through exponential mapping, formula:

[0023]

[0024] in , is the adjustment parameter, e is the base of natural logarithm;

[0025] The pitch boundary score is specifically:

[0026] Extract the starting value and ending value of the test pitch curve, the standard starting value and standard ending value of the standard test pitch, calculate the standard deviation between the starting value, the ending value and the standard starting value, the standard ending value, and calculate the score in combination with the Gaussian distribution probability, formula:

[0027]

[0028] Where, e is the base of natural logarithm, pitch value is the test pitch value, μ is the mean of the standard pitch, and σ is the standard deviation of the standard pitch;

[0029] The polynomial regression features are specifically:

[0030] The standard pitch curve and the test pitch curve are respectively fitted with quadratic polynomials to obtain the linear coefficient difference and the quadratic coefficient difference of the standard pitch curve and the test pitch curve, and the polynomial regression feature is obtained according to the linear coefficient difference and the quadratic coefficient difference.

[0031] Preferably, the data processor includes:

[0032] Keypoint detector, forced aligner, and alignment refiner;

[0033] Key point detector: used to extract facial key points of the speaker in the video, wherein the facial key points include left-right mouth corner key points and upper-lower lip key points;

[0034] Obtaining a mouth corner motion change and a lip center motion change according to the left-right mouth corner key points, the upper-lower lip center key points, and the video, and obtaining a test horizontal-vertical lip distance curve according to the mouth corner motion change and the lip center motion change;

[0035] Obtaining a standard horizontal-vertical lip distance curve according to the standard lip shape video;

[0036] A forced aligner is used to extract the video timestamp data of each word in the video, and to force the pronunciation audio and lip-syncing image of each word to align according to the video timestamp data and the audio timestamp data to obtain a rough alignment result;

[0037] An alignment refiner is configured to extract an alignment error from the coarse alignment result, generate an error compensation according to the alignment error, perform a refined alignment according to the error compensation, the test horizontal-vertical lip distance curve, and the coarse alignment result, and generate a test single-word video;

[0038] A standard single-word video is generated based on the standard horizontal-vertical lip distance curve and preset standard alignment data.

[0039] Preferably, the step of aligning the interior of the refiner is specifically as follows:

[0040] Identifying the input test horizontal-vertical lip distance curve, the standard horizontal-vertical lip distance curve, the standard alignment data, and the coarse alignment result to obtain the start and end time of each character;

[0041] Based on the start and end times, the start and end times of each word are extended toward adjacent words to form overlapping areas, and silence segments are extracted;

[0042] Identifying the silence segment to obtain short silence and long silence, merging the short silence into the previous phoneme, and dividing the long silence into separate intervals;

[0043] Extracting test lip motion features and standard lip motion features from the test horizontal-vertical lip distance curve and the standard horizontal-vertical lip distance curve, and performing minimum-maximum normalization processing on the test lip motion features and the standard lip motion features;

[0044] Extracting a golden interval of the standard alignment data, and applying a triangular window weight to the golden interval;

[0045] Extract the horizontal motion features and vertical motion features from the test lip motion features, and jointly construct a distance matrix based on the horizontal motion features and the vertical motion features. The formula is:

[0046]

[0047] Among them, D vertical is the distance matrix of the lip vertical motion features, D horizontal is the distance matrix of the lip horizontal motion features, W decay is the decreasing triangular window weight, W increase is the incremental triangular window weight;

[0048] Performing dynamic programming on the rough alignment result to find an optimal alignment path, and obtaining the precise start and end frames of the corresponding text in the video according to the optimal alignment path;

[0049] The pronunciation duration of each word in the video is constrained according to the precise start and end frames, and the pronunciation duration of each word is determined to be greater than 5 frames. The start and end times of each word are continuously connected to obtain a fine alignment result.

[0050] Preferably, the steps of generating a test word video and a standard word video are specifically:

[0051] Segment the video according to the fine alignment result to obtain a test single-word video;

[0052] The standard lip-sync video is segmented according to the fine alignment result to obtain a standard single-word video.

[0053] Preferably, the lip shape evaluator comprises:

[0054] Two first twin networks and a cosine similarity matcher, wherein the two first twin networks include a 3D convolutional layer + ResNet encoder, a feature enhancement module, and a GRU feature decoder;

[0055] Input the test single-word video and the standard single-word video into the two first twin networks respectively, and perform the same operation;

[0056] 3D convolution layer + ResNet encoder: used to extract image features from the test single-word video or the standard single-word video to obtain image coding features;

[0057] Feature enhancement module: used for extracting and enhancing features of the test single-word video or the standard single-word video to obtain optical flow field coding features, and merging the image coding features and the optical flow field coding features to obtain test video features or standard video features;

[0058] GRU feature decoder: used to extract feature information of the test video feature or the standard video feature in the time dimension, and decode the feature information to obtain a test video feature vector or a standard video feature vector;

[0059] Cosine similarity matcher: used to score the test video feature vectors and the standard video feature vectors output by the two twin networks to obtain a lip-sync score. Formula:

[0060]

[0061] Among them, F test is the test video feature vector output by the Siamese network, F std is the standard video feature vector output by the Siamese network.

[0062] Preferably, the pronunciation evaluator comprises:

[0063] A self-supervised learning model, two second twin networks, and the cosine similarity matcher;

[0064] Self-supervised learning model: used to input the transcript and the audio into the self-supervised learning model to obtain a test pronunciation feature vector and a standard pronunciation feature vector;

[0065] Second twin network: inputting the test pronunciation feature vector and the standard pronunciation feature vector into two second twin networks respectively, and the two second twin networks activate and standardize the test pronunciation feature vector or the standard pronunciation feature vector;

[0066] Cosine similarity matcher: scores the activated and normalized test pronunciation feature vector and the standard pronunciation feature vector to obtain a pronunciation score.

[0067] In summary, this application includes at least one of the following beneficial technical effects:

[0068] The transcript, standard pitch curve, and standard lip shape video are input into the data input port. Based on the transcript, the user is asked to read aloud, and the video is filmed to obtain a video. The video is then subjected to sound extraction to obtain audio. The transcript is then analyzed, and the tone of each character in the transcript is marked as the corresponding type. The audio is divided into single-word audio that matches the transcript text. For each character in the transcript, a detector is used to detect the tone. The tone change of the character is extracted and compared with the standard tone change features of the character, and the tone is scored using a tone evaluator. For the corresponding phonemes of the pinyin of the text in the transcript, the pronunciation of each phoneme in the recording is scored. For the single-word video corresponding to the text in the transcript, the single-word video features are compared with the standard lip shape features, and the lip shape score is obtained using a lip shape evaluator. Based on the tone score, pronunciation score, and lip shape score of the text, the user's overall score for each character can be obtained. This improves the accuracy and adaptability of Mandarin pronunciation scoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a module block diagram of a Mandarin pronunciation evaluation system based on deep learning provided in an embodiment of the present application.

[0070] Figure 2 This is a module block diagram of an audio processor in a Mandarin pronunciation evaluation system based on deep learning provided in an embodiment of the present application;

[0071] Figure 3 This is a module block diagram of a video processor in a Mandarin pronunciation evaluation system based on deep learning provided by an embodiment of the present application;

[0072] Figure 4This is a module block diagram of an alignment refiner in a deep learning-based Mandarin pronunciation evaluation system provided by an embodiment of the present application;

[0073] Figure 5 This is a module block diagram of a lip shape evaluator in a Mandarin pronunciation evaluation system based on deep learning provided by an embodiment of the present application;

[0074] Figure 6 This is a module block diagram of a pronunciation evaluator in a Mandarin pronunciation evaluation system based on deep learning provided in an embodiment of the present application.

[0075] Explanation of the reference numerals: 1. Data input port; 2. Data processor; 3. Tone evaluator; 4. Mouth shape evaluator; 5. Pronunciation evaluator; 6. Score output port. DETAILED DESCRIPTION

[0076] The following combination Figures 1-6 This application is further described in detail, but the embodiments of the present invention are not limited thereto.

[0077] The embodiments of the present application disclose a Mandarin pronunciation evaluation system based on deep learning.

[0078] In this embodiment, a Mandarin pronunciation evaluation system based on deep learning is provided, the system comprising:

[0079] Data input port 1: used to obtain the transcript, standard pitch curve, and standard lip shape video, capture the user's facial lip shape movements and voice when reading aloud as a video, and extract audio from the video;

[0080] Data processor 2: for processing video and audio, obtaining feature vectors of audio and test word video and standard word video of video;

[0081] Tone Evaluator 3: used to evaluate the audio according to the feature vector to obtain a pitch score;

[0082] Lip shape evaluator 4: used to perform similarity matching on the test word video and the standard word video to obtain a lip shape score;

[0083] Pronunciation Evaluator 5: Generates a standard pronunciation feature vector based on the transcript, extracts a test pronunciation feature vector based on the audio, performs similarity matching on the standard pronunciation feature vector and the test pronunciation feature vector to obtain a pronunciation score;

[0084] Scoring output 6: used to combine the pitch score, lip shape score and pronunciation score to generate the final score.

[0085] The standard pitch curve is as follows:

[0086] Load the pre-prepared pitch data and corresponding scoring data, including the pitch contour data of different speakers;

[0087] Extract the gender and specific pitch of the speaker, filter according to the gender and specific pitch to obtain pronunciation samples, and extract the pronunciation scores of the pronunciation samples;

[0088] Select target pronunciation samples with pronunciation scores higher than the preset score threshold, perform polynomial regression fitting on the target pronunciation samples to obtain average regression parameters, and obtain a standard pitch curve according to the average regression parameters.

[0089] In operation, taking the Mandarin pronunciation evaluation of user A as an example, load the pre-prepared pitch data and scoring data, which include the pitch contour data of different speakers. Suppose there are 100 recorded files of speakers stored in the pitch data, and each speaker reads 10 Chinese characters, and each Chinese character corresponds to a pitch curve data. The system first extracts the gender and specific pitch information of the speaker. For example, the gender of user A is male, and the Chinese character to be evaluated is "天" with the standard tone of the first tone. Filter the pronunciation samples according to the gender and tone, and select the data of all male speakers reading the character "天" from 100 speakers, and a total of 30 samples are obtained. Then extract the pronunciation scores of these samples, with the score range from 0 to 100 points, and the preset score threshold is 80 points. The system selects 15 samples with scores higher than 80 points as target pronunciation samples. Perform polynomial regression fitting on the pitch curves of the target pronunciation samples. Suppose the fitting result is: the average regression parameters of the 15 curves are the first-order coefficient of 0.5 and the second-order coefficient of -0.2. Generate a standard pitch curve according to these parameters. The starting pitch of this curve is 200Hz, the highest point in the middle is 220Hz, and the ending pitch is 210Hz, forming a smooth parabola shape. This standard pitch curve will be used as the benchmark for subsequent evaluation of user A's pronunciation.

[0090] The data processor 2 includes:

[0091] A forced aligner, a pitch detector, and a pitch feature extractor;

[0092] The forced aligner: used to input the transcript, audio, standard pitch curve, and preset tone sequence into the forced aligner. The forced aligner extracts the audio timestamp data corresponding to each character in the transcript in the audio, segments the audio according to the audio timestamp data to obtain audio segments; [[ID=2​​​Pitch feature extractor: used to compare the test pitch curve with the standard pitch curve in multiple dimensions, extract the maximum value, minimum value, mean value, standard deviation, curve length, dynamic time warping score, pitch boundary score and polynomial regression features of the test pitch curve, and organize them to obtain a feature vector;

[0095] Dynamic Time Warping Score, specifically:

[0096] Extract the first-order difference of the test pitch curve, use DTW to calculate the DTW distance between the test pitch curve and the standard pitch curve, and normalize the distance The distance is converted to a score (range 0~100) through exponential mapping, formula:

[0097]

[0098] in , is the adjustment parameter, e is the base of natural logarithm;

[0099] Pitch boundary scoring, specifically:

[0100] Extract the starting and ending values ​​of the test pitch curve, the standard starting and ending values ​​of the standard test pitch, calculate the standard deviation between the starting and ending values ​​and the standard starting and ending values, and calculate the score based on the Gaussian distribution probability. Formula:

[0101]

[0102] Where, e is the base of natural logarithm, pitch value is the test pitch value, μ is the mean of the standard pitch, and σ is the standard deviation of the standard pitch;

[0103] Polynomial regression features, specifically:

[0104] The standard pitch curve and the test pitch curve are fitted with quadratic polynomials respectively to obtain the linear coefficient difference and the quadratic coefficient difference of the standard pitch curve and the test pitch curve, and the polynomial regression characteristics are obtained according to the linear coefficient difference and the quadratic coefficient difference.

[0105] During operation, taking the example of user A reading the character "天", the forced aligner in the data processor receives the transcript, audio, standard pitch curve, and a preset tone sequence. The content of the transcript is "天", and the tone sequence is marked as the first tone. The forced aligner first analyzes the audio file to identify the pronunciation time period of the character "天". Assuming the total audio duration is 2 seconds, the start time of the pronunciation of the character "天" is determined to be 0.5 seconds and the end time is 1.2 seconds through the voice activity detection (VAD) algorithm. The audio is segmented into an audio segment from 0.5 seconds to 1.2 seconds. The pitch detector performs pitch tracking on this audio segment, extracts pitch data at 100 sampling points per second, and generates a test pitch curve for user A, with a starting pitch of 205 Hz, a highest point in the middle of 225 Hz, and an ending pitch of 215 Hz. The pitch feature extractor compares the test pitch curve with the standard pitch curve and extracts features such as the maximum value (225 Hz vs 220 Hz), minimum value (205 Hz vs 200 Hz), mean value (215 Hz vs 210 Hz), and standard deviation (8 Hz vs 5 Hz). The dynamic time warping (DTW) calculates the distance between the two as 12, and the normalized score is 85 points. In the pitch boundary scoring, the standard deviation between the test starting value and the standard starting value is 5 Hz, and the standard deviation of the ending value is 5 Hz. The score is calculated to be 90 points by combining the Gaussian distribution probability. In the polynomial regression feature, the first-order coefficient of the test curve is 0.6, the second-order coefficient is -0.25, and the differences from the standard parameters are 0.1 and -0.05. The finally generated feature vector contains all the above values and is used for subsequent pitch scoring.

[0106] The data processor 2 includes:

[0107] A key point detector, a forced aligner, and an alignment refinement unit;

[0108] The key point detector: used to extract the facial key points of the speaker in the video. The facial key points include the left - right mouth corner key points and the upper - lower lip middle key points;

[0109] Based on the left - right mouth corner key points, the upper - lower lip middle key points, and the video, the mouth corner movement change and the lip middle movement change are obtained. Based on the mouth corner movement change and the lip middle movement change, a test horizontal - vertical lip distance curve is obtained;

[0110] Based on the standard mouth shape video, a standard horizontal - vertical lip distance curve is obtained;

[0111] The forced aligner: used to extract the video timestamp data of the pronunciation of each character in the video. Based on the video timestamp data and the audio timestamp data, the pronunciation audio and the mouth shape picture of each character are forced to be aligned to obtain a rough alignment result;

[0112] Alignment Refiner: It is used to extract the alignment errors in the rough alignment results, generate error compensation based on the alignment errors, and perform refined alignment according to the error compensation, the test horizontal-vertical lip distance curve, and the rough alignment results to generate a test single-character video;

[0113] Generate a standard single-character video according to the standard horizontal-vertical lip distance curve and the preset standard alignment data.

[0114] During operation, taking the video of user A pronouncing the character "天" as an example, the key point detector extracts facial key points from the video. Assuming the video frame rate is 30fps, the detected coordinates of the left corner of the mouth are (100, 200), the right corner of the mouth is (300, 200), the midpoint of the upper lip is (200, 180), and the midpoint of the lower lip is (200, 220). Based on these key points, the horizontal lip distance is calculated as the X coordinate of the right corner of the mouth minus the X coordinate of the left corner of the mouth (300 - 100 = 200 pixels), and the vertical lip distance is calculated as the Y coordinate of the midpoint of the lower lip minus the Y coordinate of the midpoint of the upper lip (220 - 180 = 40 pixels). Tracking 10 frames of data continuously, the test horizontal-vertical lip distance curve is obtained: the horizontal lip distance gradually shrinks from 200 pixels to 180 pixels, and the vertical lip distance expands from 40 pixels to 50 pixels. At the same time, the standard horizontal-vertical lip distance curve of the character "天" is extracted from the standard mouth shape video: the horizontal lip distance is stable at 190 pixels, and the vertical lip distance is stable at 45 pixels. The forced aligner roughly aligns the pronunciation mouth shape picture with the audio according to the audio timestamp (0.5 seconds to 1.2 seconds) and the video timestamp (frame 15 to frame 36). The alignment refiner detects that there is an error of 2 frames in the rough alignment result and generates an error compensation parameter: adjust the starting frame of the video from 15 to 13 frames, and the ending frame from 36 to 38 frames. The corrected test single-character video covers frames 13 to 38, and the standard single-character video is segmented into the same duration according to the preset data.

[0115] The steps inside the alignment refiner are specifically as follows:

[0116] Identify the input test horizontal-vertical lip distance curve, standard horizontal-vertical lip distance curve, standard alignment data, and rough alignment results to obtain the start and end times of each character;

[0117] Based on the start and end times, expand the start and end times of each character to the adjacent characters to form an overlapping area, and extract the silent segments;

[0118] Identify the silent segments to obtain short silent segments and long silent segments, merge the short silent segments into the previous phoneme, and divide the long silent segments into separate intervals; [[ID=​​

[0120] Extract the golden interval of the standard alignment data and apply a triangular window weight to the golden interval;

[0121] Extract the horizontal motion feature and the vertical motion feature in the test lip movement feature, and jointly construct a distance matrix according to the horizontal motion feature and the vertical motion feature. The formula:

[0122]

[0123] where D vertical is the distance matrix of the vertical direction motion feature of the lip, D horizontal is the distance matrix of the horizontal direction motion feature of the lip, W decay is the decreasing triangular window weight, W increase is the increasing triangular window weight;

[0124] Perform dynamic programming on the rough alignment result to find the optimal alignment path, and obtain the accurate start and end frames of the corresponding text in the video according to the optimal alignment path;

[0125] Constrain the pronunciation duration of each word in the video according to the accurate start and end frames, determine that the pronunciation duration of each word is greater than 5 frames, and connect the start and end times of each word continuously to obtain the fine alignment result.

[0126] In the application, taking the pronunciation of the word "tian" of user A as an example, the alignment refiner identifies the start and end times of each word. Assume that the audio start and end times of the word "tian" are from 0.5 seconds to 1.2 seconds, and the video start and end frames are from the 13th frame to the 38th frame. The system expands 0.1 second to the adjacent words to form an overlapping area and extracts the silent segment. It is found that the short silent segment from 0.4 seconds to 0.5 seconds (0.1 second) is merged into the previous phoneme; the long silent segment from 1.2 seconds to 1.5 seconds (0.3 second) is separately divided into a silent interval. In the test lip movement feature, the average horizontal lip distance is 190 pixels, and the standard deviation is 10 pixels; the average vertical lip distance is 45 pixels, and the standard deviation is 5 pixels. The average horizontal lip distance of the standard lip movement feature is 190 pixels, and the standard deviation is 5 pixels; the average vertical lip distance is 45 pixels, and the standard deviation is 3 pixels. Perform min-max normalization on the two. The test horizontal feature is scaled to [0.8, 1.2], and the standard horizontal feature is [0.9, 1.1]. After applying the triangular window weight to the golden interval, the horizontal motion weight is 0.7, and the vertical motion weight is 0.3. When constructing the distance matrix, the horizontal difference is 0.2, the vertical difference is 0.1, and the total distance is 0.17. Find the optimal alignment path through dynamic programming, adjust the video start and end frames to the 12th frame to the 37th frame, ensure that the pronunciation duration of each word is greater than 5 frames (25 frames), and finally generate the fine alignment result.

[0127] The steps to generate the test single-word video and the standard single-word video are specifically as follows:

[0128] The video is segmented according to the fine alignment result to obtain the test single-character video;

[0129] The standard mouth shape video is segmented according to the fine alignment result to obtain the standard single-character video.

[0130] In operation, taking the fine alignment result of user A as an example, the system segments the original video (total duration 2 seconds, 60 frames) according to the fine alignment result. Suppose the start and end frames of the video of the character "天" are from the 12th frame to the 37th frame, a total of 25 frames, which are segmented into test single-character video segments. The total duration of the standard mouth shape video is 1.8 seconds, and 25-frame standard single-character video segments are segmented according to the same start and end ratio. The test single-character video contains the mouth shape changes when user A reads the character "天": in the first 5 frames, the lips are slightly open, in the middle 15 frames, the lips gradually expand to the maximum opening, and in the last 5 frames, the lips slowly close. The standard single-character video shows the standard pronunciation mouth shape: in the first 5 frames, the lips are steadily open, in the middle 15 frames, the maximum opening is maintained, and in the last 5 frames, they close evenly. The video segments of both are stored in units of 25 frames for further analysis by the mouth shape evaluator.

[0131] The mouth shape evaluator 4 includes:

[0132] [[ID=1,3]]Two first twin networks and a cosine similarity matcher, where the two first twin networks include a 3D convolutional layer + ResNet encoder, a feature enhancement module, and a GRU feature decoder;

[0133] The test single-character video and the standard single-character video are respectively input into the two first twin networks to perform the same operations;

[0134] 3D convolutional layer + ResNet encoder: used to extract image features from the test single-character video or the standard single-character video to obtain image encoded features;

[0135] Feature enhancement module: used to extract and enhance features from the test single-character video or the standard single-character video to obtain optical flow field encoded features, and merge the image encoded features and the optical flow field encoded features to obtain test video features or standard video features;

[0136] GRU feature decoder: used to extract the feature information of the test video features or the standard video features in the time dimension and decode the feature information to obtain test video feature vectors or standard video feature vectors;

[0137] Cosine similarity matcher: used to score the test video feature vectors and the standard video feature vectors output by the two twin networks to obtain a mouth shape score, formula:

[0138]

[0139] where, Ftest is the test video feature vector output by the Siamese network, F std is the standard video feature vector output by the Siamese network.

[0140] In operation, taking the test single-word video and the standard single-word video of user A as examples, the first Siamese network of the lip shape evaluator processes video data. The test video is input into the 3D convolutional layer and the ResNet encoder to extract the image features of each frame (for example, the convolutional layer outputs a 512-dimensional feature vector). The feature enhancement module extracts the optical flow field features to capture the lip movement trajectory (such as a horizontal movement speed of 10 pixels / frame and a vertical movement speed of 5 pixels / frame). After combining the image and optical flow features, the GRU decoder analyzes the time series and outputs the test video feature vector (for example, 256-dimensional). The standard video undergoes the same processing to obtain the standard video feature vector. The cosine similarity matcher calculates the similarity between the two. Assuming the test vector is [0.2, 0.5, 0.3] and the standard vector is [0.3, 0.6, 0.2], the cosine similarity is 0.92 (full score 1.0), corresponding to a lip shape score of 92 points.

[0141] The pronunciation evaluator 5 includes:

[0142] a self-supervised learning model, two second Siamese networks, and a cosine similarity matcher;

[0143] Self-supervised learning model: used to input the transcript and audio into the self-supervised learning model to obtain the test pronunciation feature vector and the standard pronunciation feature vector;

[0144] Second Siamese network: input the test pronunciation feature vector and the standard pronunciation feature vector into the two second Siamese networks respectively, and the two second Siamese networks activate and standardize the test pronunciation feature vector or the standard pronunciation feature vector;

[0145] Cosine similarity matcher: score the activated and standardized test pronunciation feature vector and the standard pronunciation feature vector to obtain the pronunciation score.

[0146] In operation, taking the pronunciation evaluation of user A as an example, the self-supervised learning model inputs the transcript "tian" and the audio, and extracts the test pronunciation feature vector (for example, the MFCC feature mean is 0.5 and the variance is 0.1). The standard pronunciation feature vector comes from the pre-trained model, with a mean of 0.6 and a variance of 0.05. The second Siamese network activates and standardizes both. The test vector is adjusted to [0.48, 0.12, 0.35], and the standard vector is [0.55, 0.10, 0.40]. The cosine similarity matcher calculates a score of 0.85, corresponding to a pronunciation score of 85 points. The final score output terminal synthesizes the tone score of 85 points, the lip shape score of 92 points, and the pronunciation score of 85 points, and generates a final score of 87 points after weighted averaging.

[0147] The above are all preferred embodiments of the present application, and are not intended to limit the scope of protection of the present application. Therefore, any equivalent changes made based on the structure, shape, and principle of the present application should be included in the scope of protection of the present application.

Claims

1. A Mandarin pronunciation evaluation system based on deep learning, characterized in that: include: Data input port: used to obtain a transcript, a standard pitch curve, and a standard lip shape video, capture the user's facial lip shape movements and voice when reading as a video, and extract audio from the video; A data processor is used to process the video and the audio to obtain a feature vector of the audio and a test single-word video and a standard single-word video of the video; A tone evaluator is configured to evaluate the audio according to the feature vector to obtain a tone score; Lip shape evaluator: used to perform similarity matching on the test single-word video and the standard single-word video to obtain a lip shape score; A pronunciation evaluator is configured to generate a standard pronunciation feature vector based on the transcript, extract a test pronunciation feature vector based on the audio, perform similarity matching on the standard pronunciation feature vector and the test pronunciation feature vector to obtain a pronunciation score; Scoring output terminal: used to combine the pitch score, the lip shape score and the pronunciation score to generate a final score; The data processor comprises: Forced aligner, pitch detector, and pitch feature extractor; A forced aligner is configured to input the transcript, the audio, the standard pitch curve, and a preset tone sequence into the forced aligner, wherein the forced aligner extracts the audio timestamp data corresponding to each word in the transcript in the audio, and segments the audio according to the audio timestamp data to obtain audio segments; Pitch detector: used to perform pitch detection on the audio segment of each word to obtain a test pitch curve corresponding to the pronunciation of each word; Pitch feature extractor: used to perform multi-dimensional comparison between the test pitch curve and the standard pitch curve, extract the maximum value, minimum value, mean value, standard deviation, curve length, dynamic time warping score, pitch boundary score and polynomial regression features in the test pitch curve, and organize them to obtain a feature vector; The dynamic time warping score is specifically: Extract the first-order difference of the test pitch curve, use DTW to calculate the DTW distance between the test pitch curve and the standard pitch curve, and normalize the distance The distance is converted to a score using the exponential mapping: in , is the adjustment parameter, e is the base of natural logarithm; The pitch boundary score is specifically: Extract the starting value and ending value of the test pitch curve, the standard starting value and standard ending value of the standard test pitch, calculate the standard deviation between the starting value, the ending value and the standard starting value, the standard ending value, and calculate the score in combination with the Gaussian distribution probability. Formula: Where, e is the base of natural logarithm, pitch value is the test pitch value, μ is the mean of the standard pitch, and σ is the standard deviation of the standard pitch; The polynomial regression features are specifically: The standard pitch curve and the test pitch curve are respectively fitted with quadratic polynomials to obtain the linear coefficient difference and the quadratic coefficient difference of the standard pitch curve and the test pitch curve, and the polynomial regression feature is obtained according to the linear coefficient difference and the quadratic coefficient difference.

2. A Mandarin pronunciation evaluation system based on deep learning according to claim 1, characterized in that: The standard pitch curve is as follows: Load pre-prepared pitch data and corresponding scoring data, including pitch contour data of different speakers; Extracting the gender and specific tone of the speaker, screening according to the gender and the specific tone to obtain a pronunciation sample, and extracting a pronunciation score of the pronunciation sample; A target pronunciation sample having a pronunciation score higher than a preset score threshold is selected, a polynomial regression fitting is performed on the target pronunciation sample to obtain an average regression parameter, and a standard pitch curve is obtained according to the average regression parameter.

3. A Mandarin pronunciation evaluation system based on deep learning according to claim 2, characterized in that: The data processor comprises: Keypoint detector, forced aligner, and alignment refiner; Key point detector: used to extract facial key points of the speaker in the video, wherein the facial key points include left-right mouth corner key points and upper-lower lip key points; Obtaining a mouth corner motion change and a lip center motion change according to the left-right mouth corner key points, the upper-lower lip center key points, and the video, and obtaining a test horizontal-vertical lip distance curve according to the mouth corner motion change and the lip center motion change; Obtaining a standard horizontal-vertical lip distance curve according to the standard lip shape video; A forced aligner is used to extract the video timestamp data of each word in the video, and to force the pronunciation audio and lip-syncing image of each word to align according to the video timestamp data and the audio timestamp data to obtain a rough alignment result; An alignment refiner is configured to extract an alignment error from the coarse alignment result, generate an error compensation according to the alignment error, perform a refined alignment according to the error compensation, the test horizontal-vertical lip distance curve, and the coarse alignment result, and generate a test single-word video; A standard single-word video is generated based on the standard horizontal-vertical lip distance curve and preset standard alignment data.

4. A Mandarin pronunciation evaluation system based on deep learning according to claim 3, characterized in that: The steps to align the inside of the tessellator are: Identifying the input test horizontal-vertical lip distance curve, the standard horizontal-vertical lip distance curve, the standard alignment data, and the coarse alignment result to obtain the start and end time of each character; Based on the start and end times, the start and end times of each word are extended toward adjacent words to form overlapping areas, and silence segments are extracted; Identifying the silence segment to obtain short silence and long silence, merging the short silence into the previous phoneme, and dividing the long silence into separate intervals; Extracting test lip motion features and standard lip motion features from the test horizontal-vertical lip distance curve and the standard horizontal-vertical lip distance curve, and performing minimum-maximum normalization processing on the test lip motion features and the standard lip motion features; Extracting a golden interval of the standard alignment data, and applying a triangular window weight to the golden interval; Extract the horizontal motion features and vertical motion features from the test lip motion features, and jointly construct a distance matrix based on the horizontal motion features and the vertical motion features. The formula is: Where Dvertical is the distance matrix of the lip vertical motion features, Dhorizontal is the distance matrix of the lip horizontal motion features, Wdecay is the decreasing triangular window weight, and Wincrease is the increasing triangular window weight; Performing dynamic programming on the rough alignment result to find an optimal alignment path, and obtaining the precise start and end frames of the corresponding text in the video according to the optimal alignment path; The pronunciation duration of each word in the video is constrained according to the precise start and end frames, and the pronunciation duration of each word is determined to be greater than 5 frames. The start and end times of each word are continuously connected to obtain a fine alignment result.

5. A Mandarin pronunciation evaluation system based on deep learning according to claim 4, characterized in that: The steps for generating test word videos and standard word videos are as follows: Segment the video according to the fine alignment result to obtain a test single-word video; The standard lip-sync video is segmented according to the fine alignment result to obtain a standard single-word video.

6. A Mandarin pronunciation evaluation system based on deep learning according to claim 5, characterized in that: The lip shape evaluator comprises: Two first twin networks and a cosine similarity matcher, wherein the two first twin networks include a 3D convolutional layer + ResNet encoder, a feature enhancement module, and a GRU feature decoder; Input the test single-word video and the standard single-word video into the two first twin networks respectively, and perform the same operation; 3D convolution layer + ResNet encoder: used to extract image features from the test single-word video or the standard single-word video to obtain image coding features; Feature enhancement module: used for extracting and enhancing features of the test single-word video or the standard single-word video to obtain optical flow field coding features, and merging the image coding features and the optical flow field coding features to obtain test video features or standard video features; GRU feature decoder: used to extract feature information of the test video feature or the standard video feature in the time dimension, and decode the feature information to obtain a test video feature vector or a standard video feature vector; Cosine similarity matcher: used to score the test video feature vectors and the standard video feature vectors output by the two twin networks to obtain a lip-sync score. Formula: Among them, Ftest is the test video feature vector output by the twin network, and Fstd is the standard video feature vector output by the twin network.

7. A Mandarin pronunciation evaluation system based on deep learning according to claim 6, characterized in that: The pronunciation evaluator comprises: A self-supervised learning model, two second twin networks, and the cosine similarity matcher; Self-supervised learning model: used to input the transcript and the audio into the self-supervised learning model to obtain a test pronunciation feature vector and a standard pronunciation feature vector; Second twin network: inputting the test pronunciation feature vector and the standard pronunciation feature vector into two second twin networks respectively, and the two second twin networks activate and standardize the test pronunciation feature vector or the standard pronunciation feature vector; Cosine similarity matcher: scores the activated and normalized test pronunciation feature vector and the standard pronunciation feature vector to obtain a pronunciation score.

Images