Speech training method and system based on artificial intelligence and application

Abstract

The invention discloses a speech training method and system based on artificial intelligence and application, and relates to the field of speech training, and the method comprises the steps: obtaining multi-modal data in real time when a user reads each training statement in each speech training process; obtaining a real-time multi-dimensional speech evaluation result according to the multi-modal data, and judging whether a next training statement exists in a speech training scheme in each speech training process or not; if not, generating a multi-dimensional speech evaluation result of each speech training process; if yes, the difficulty of the next training statement in the speech training scheme is adaptively and dynamically adjusted according to the real-time multi-dimensional speech evaluation result; and outputting the next training statement according to the adjusted speech training scheme, and repeating the steps until the next training statement does not exist in each speech training. The speech training scheme can be adaptively adjusted in real time in the speech training process, and the user experience, the participation degree and the training effect are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of speech training technology, and in particular to an artificial intelligence-based speech training method, system and application. Background Art

[0002] Speaking is an important part of learning any language. Speech training refers to the process of cultivating and improving oral expression skills through specific methods and techniques. Traditional speech training generally relies on teachers actively demonstrating and judging, which is highly subjective. Furthermore, with a large number of students, it is difficult to provide targeted teaching and assessment for each student or user. Existing speech training methods often use voice recognition to identify various situational dialogues to obtain test results for each student. However, this speech training method lacks feedback and multimodal interaction, making it difficult for students to receive multi-dimensional training guidance. The training is monotonous, and user engagement is low. Summary of the Invention

[0003] In order to solve the technical problems existing in the background technology, the present invention proposes a speech training method, system and application based on artificial intelligence.

[0004] In a first aspect, the present invention proposes a speech training method based on artificial intelligence, comprising:

[0005] S1. Acquire multimodal data of each training sentence read out by the user in real time during each speech training process; wherein reading includes one or more of singing, chanting, reciting, reciting, reading, sighing, humming, moaning, satirizing, and reciting;

[0006] S2. Based on the multimodal data, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence to obtain a real-time multi-dimensional speech evaluation result;

[0007] S4, determining whether the user has a next training sentence in the speech training plan during each speech training process;

[0008] If yes, go to S4; if no, go to S6;

[0009] S4. Adaptively and dynamically adjust the difficulty of the next training sentence in the speech training program based on the real-time multi-dimensional speech assessment results;

[0010] S5. Output the next training sentence according to the adjusted speech training plan and enter S1;

[0011] S6. Generate a multi-dimensional speech evaluation result for each speech training process based on the real-time multi-dimensional speech evaluation results corresponding to all training sentences in each speech training process.

[0012] Preferably, the multimodal data includes voice data and facial video;

[0013] In S2, based on the multimodal data, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence, and a real-time multi-dimensional speech evaluation result is obtained, specifically including:

[0014] Based on the multimodal data, a preset feature extraction model is used to extract features from the multimodal data in real time to obtain speech data, lip shape feature tracking results, facial micro-expression feature extraction results, and eye tracking results;

[0015] Based on speech data, lip shape feature tracking results, facial micro-expression feature extraction results and eye tracking results, a real-time multi-dimensional evaluation of the user's speech training effect on each training sentence is performed to obtain real-time multi-dimensional speech evaluation results.

[0016] Preferably, the feature extraction model includes a speech feature extraction model, a lip shape feature extraction model, a facial micro-expression feature extraction model and an eye tracking model.

[0017] Preferably, the speech feature extraction model includes a self-supervised speech recognition model and a generative AI model. The self-supervised speech recognition model is used to perform feature encoding and speech recognition on multimodal data to obtain speech recognition results; wherein the speech recognition results include speech feature vectors and speech recognition text; the generative AI model is used to generate speech based on the speech recognition results to obtain speech data.

[0018] Preferably, the facial micro-expression feature extraction model includes a deep convolutional neural network and a Transformer model; the deep convolutional network is used to perform feature encoding on facial video frames extracted from facial videos to obtain facial video features; the Transformer model is used to perform time-series encoding on facial video features to obtain facial feature vectors, and use the facial feature vectors as the facial micro-expression feature extraction results.

[0019] Preferably, the process of extracting features from facial videos by the lip shape feature extraction model includes: first capturing the human face in real time from the facial video and detecting the key points of the face to obtain the coordinate data of the key points of the lip area of ​​each frame; arranging the coordinate data of the key points of the lip area of ​​each frame in time sequence to obtain a real-time key point sequence; using Kalman filtering or exponential smoothing method to filter the key point sequence in real time; normalizing the real-time key point sequence after real-time filtering; inputting the normalized real-time key point sequence into the lip shape feature extraction model to obtain a lip shape dynamic feature vector as a result of lip shape feature tracking.

[0020] Preferably, the real-time multi-dimensional speech evaluation results include a comprehensive score of multi-dimensional training effects and a user engagement score;

[0021] Among them, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence based on speech data, lip feature tracking results, facial micro-expression feature extraction results and eye tracking results to obtain a real-time multi-dimensional speech evaluation result, specifically including: multi-modally fusing the speech data and the lip feature tracking results to form a first fusion feature; inputting the first fusion feature into a preset multi-dimensional training effect scoring model to obtain a multi-dimensional training effect comprehensive score; splicing the multi-dimensional training effect comprehensive score, facial micro-expression feature extraction results and eye tracking results to form a second fusion feature; inputting the second fusion feature into the user engagement evaluation model to obtain a user engagement score.

[0022] Preferably, the multidimensional training effect scoring model includes a pitch detection sub-model, a rhythm detection sub-model, a speaking rate detection sub-model, a pronunciation detection sub-model and a multidimensional training effect comprehensive scoring sub-model; the pitch detection sub-model, the rhythm detection sub-model, the speaking rate detection sub-model and the pronunciation detection sub-model are used to perform pitch, rhythm, speaking rate and pronunciation accuracy detection on the first fusion feature respectively to obtain pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking rate accuracy; the multidimensional training effect comprehensive scoring sub-model is used to perform a multidimensional comprehensive evaluation based on pitch accuracy, rhythm accuracy, pronunciation accuracy, speaking rate accuracy and preset weights to obtain a multidimensional training effect comprehensive score.

[0023] Preferably, the pronunciation detection submodel includes a self-supervised speech recognition model and an anomaly detection head; during the pronunciation detection process, the self-supervised speech recognition model is used to detect the pronunciation accuracy of the first fusion feature to obtain the initial pronunciation accuracy; it is determined whether the initial pronunciation accuracy is higher than the preset pronunciation accuracy, and if so, the initial pronunciation accuracy is used as the final pronunciation accuracy; if not, the first fusion feature is segmented using the anomaly detection head to obtain multiple intermediate segments; the multiple intermediate segments are then aligned segment by segment to obtain an alignment result; it is determined whether there is a key vowel or phoneme in the alignment result whose duration is lower than a preset range; if so, the key vowel or phoneme is marked as "suspected missing" or "abnormal pronunciation"; the pathological data and the key vowels or phonemes marked as "suspected missing" or "abnormal pronunciation" are weighted according to the preset penalty coefficient or the actual proportion to obtain the final pronunciation accuracy.

[0024] Preferably, the final pronunciation accuracy = (normal phoneme count - α × abnormal phoneme count) / total phoneme count; the abnormal phoneme count is the number of key vowels or phonemes marked as "suspected missing" or "abnormal pronunciation", the total phoneme count represents the total number of phonemes in the speech data, the normal phoneme count represents the number of normal phonemes in the speech data, and α represents the weight of the abnormal phoneme count.

[0025] Preferably, after each first preset number of speech training processes, the method further includes:

[0026] A reinforcement learning algorithm is used to adjust the weights of each indicator of pitch accuracy, rhythm accuracy, pronunciation accuracy, and speaking speed accuracy based on the pitch accuracy, rhythm accuracy, pronunciation accuracy, speaking speed accuracy and the comprehensive score of multi-dimensional training effects during the first preset number of speech training times.

[0027] Preferably, during the weight adjustment process, if the degree of improvement of one of the indicators among pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking speed accuracy is lower than a preset improvement threshold, the weight corresponding to the indicator is increased;

[0028] If the degree of improvement of one of the indicators among pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking speed accuracy is higher than or equal to the preset improvement threshold, the weight corresponding to the indicator is reduced or maintained.

[0029] Preferably, after each second preset number of speech trainings, it also includes: dynamically adjusting the speech training plan based on a preset training plan dynamic adjustment strategy or / and a reinforcement learning algorithm according to the speech training plan, multi-dimensional speech evaluation results in the current speech training process and the speech training plan and multi-dimensional speech evaluation results in the second preset number of speech training processes.

[0030] Preferably, in the adjustment process before the preset cumulative training times, the speech training plan is adjusted according to the preset training plan dynamic adjustment strategy; in the adjustment process after the preset cumulative training times, the speech training plan is dynamically adjusted using Deep Q-Learning or a multi-armed bandit algorithm based on the speech training plan in the current speech training process, the multi-dimensional speech evaluation results, and the speech training plan and multi-dimensional speech evaluation results during the preset historical number of speech training times.

[0031] Preferably, in the dynamic adjustment strategy of the training program, if after a second preset number of speech training sessions, the user's multi-dimensional speech evaluation result is greater than or equal to the ideal speech training effect and the user's participation is greater than or equal to a preset participation threshold, the difficulty of the speech training program is increased;

[0032] If, after a preset number of historical speech training sessions, the user's multi-dimensional speech assessment results are greater than or equal to the ideal speech training effect and the user's engagement is less than a preset engagement threshold, the difficulty of the speech training program is increased and personalized parameters are adjusted;

[0033] If, after a preset number of historical speech training sessions, the user's multi-dimensional speech evaluation results are less than the ideal speech training effect and the user's engagement is greater than or equal to a preset engagement threshold, the difficulty of the speech training program is reduced;

[0034] If after a preset number of historical speech training sessions, the user's multi-dimensional speech evaluation results are less than the ideal speech training effect and the user's participation is less than a preset participation threshold, the difficulty of the speech training program will be reduced and personalized parameters will be adjusted or / and an emotional encouragement program will be generated.

[0035] Preferably, the difficulty of the next training sentence in the speech training program is adaptively and dynamically adjusted according to the real-time multi-dimensional speech evaluation results, specifically including:

[0036] According to the real-time multi-dimensional speech evaluation results, the difficulty of the next training sentence in the speech training plan of the user during each speech training is dynamically adjusted in real time to obtain an intermediate speech training plan;

[0037] The intermediate speech training program is personalized based on the personalized parameters pre-entered or selected by the user.

[0038] In a second aspect, the present invention further proposes a speech training system based on artificial intelligence, comprising:

[0039] An acquisition module is used to acquire multimodal data of each training sentence read out by the user in real time during each speech training process;

[0040] The evaluation module is used to perform real-time multi-dimensional evaluation of the user's speech training effect on each training sentence based on multimodal data to obtain real-time multi-dimensional speech evaluation results;

[0041] A judgment module is used to judge whether the user has the next training sentence in the speech training plan during each speech training process;

[0042] A program generation and adjustment module is used to adaptively and dynamically adjust the difficulty of the next training sentence in the speech training program according to the real-time multi-dimensional speech evaluation results when the next training sentence exists in the speech training program;

[0043] The training module is used to output the next training sentence according to the adjusted speech training program;

[0044] The evaluation module is also used to generate a multidimensional speech evaluation result for each speech training process based on the real-time multidimensional speech evaluation results corresponding to all training sentences in each speech training process when there is no next training sentence in the speech training plan.

[0045] In the third aspect, the present invention also proposes an application of an artificial intelligence-based speech training system, which is applied to the field of speech rehabilitation training to perform speech rehabilitation training for users with aphasia or speech disorders.

[0046] The artificial intelligence-based speech training method, system and application proposed in the present invention can realize real-time detection of multimodal data of each training sentence read out by the user during the speech training process to obtain multimodal data; based on the multimodal data, perform real-time multi-dimensional evaluation of the user's speech training effect on each training sentence to obtain a real-time multi-dimensional speech evaluation result; determine whether the user has the next training sentence in the speech training plan in each speech training process; if not, enter the real-time multi-dimensional speech evaluation results corresponding to all training sentences in each speech training process to generate a multi-dimensional speech evaluation result for each speech training process; if so, adaptively and dynamically adjust the difficulty of the next training sentence in the speech training plan according to the real-time multi-dimensional speech evaluation results; output the next training sentence according to the adjusted speech training plan, and iteratively repeat the above steps until the next training sentence does not exist in the speech training plan in each training process.

[0047] The present invention effectively improves the accuracy of the multi-dimensional speech evaluation results obtained by performing multi-dimensional evaluation on the multimodal data of each training sentence of the user in each speech training process; moreover, the difficulty of the next training sentence of the user in each speech training is adaptively updated and dynamically adjusted in real time according to the multi-dimensional speech evaluation results, so that the finally generated speech training program meets the user's current situation, which is conducive to improving the user's experience, participation and training effect when using the generated speech training program for subsequent training in each training process, and can continuously optimize the speech training program during the training of subsequent training sentences to achieve more efficient and personalized speech training. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 Schematic diagram of the flow of an artificial intelligence-based speech training method in one embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] First, as Figure 1 As shown, the present invention also proposes a speech training method based on artificial intelligence, comprising:

[0051] S1. Acquire multimodal data of each training sentence read out by the user in real time during each speech training process; wherein reading includes one or more of singing, chanting, reciting, reciting, reading, sighing, humming, moaning, satirizing, and reciting;

[0052] S2. Based on the multimodal data, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence to obtain a real-time multi-dimensional speech evaluation result;

[0053] S3, determining whether the user has a next training sentence in the speech training plan during each speech training process;

[0054] If yes, go to S4; if no, go to S6;

[0055] S3. Adaptively and dynamically adjust the difficulty of the next training sentence in the speech training program based on the real-time multi-dimensional speech assessment results;

[0056] S5. Output the next training sentence according to the adjusted speech training plan and enter S1;

[0057] S6. Generate a multi-dimensional speech evaluation result for each speech training process based on the real-time multi-dimensional speech evaluation results corresponding to all training sentences in each speech training process.

[0058] The multimodal data in this embodiment includes voice data and facial video.

[0059] The present invention effectively improves the accuracy of the multi-dimensional speech evaluation results obtained by performing multi-dimensional evaluation on the multimodal data of each training sentence of the user in each speech training process; moreover, the difficulty of the next training sentence of the user in each speech training is adaptively updated and dynamically adjusted in real time according to the multi-dimensional speech evaluation results, so that the finally generated speech training program meets the user's current situation, which is conducive to improving the user's experience, participation and training effect when using the generated speech training program for subsequent training in each training process, and can continuously optimize the speech training program during the training of subsequent training sentences to achieve more efficient and personalized speech training.

[0060] In S2, based on the multimodal data, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence, and a real-time multi-dimensional speech evaluation result is obtained, specifically including:

[0061] Based on the multimodal data, a preset feature extraction model is used to extract features from the multimodal data in real time to obtain speech data, lip shape feature tracking results, facial micro-expression feature extraction results, and eye tracking results;

[0062] Based on speech data, lip shape feature tracking results, facial micro-expression feature extraction results and eye tracking results, a real-time multi-dimensional evaluation of the user's speech training effect on each training sentence is performed to obtain real-time multi-dimensional speech evaluation results.

[0063] This embodiment is configured in this way to obtain real-time multi-dimensional speech evaluation results.

[0064] In this embodiment, the feature extraction model includes a speech feature extraction model, a lip shape feature extraction model, a facial micro-expression feature extraction model, and an eye tracking model.

[0065] Among them, the speech feature extraction model in this embodiment includes a self-supervised speech recognition model and a generative AI model. The self-supervised speech recognition model is used to perform feature encoding and speech recognition on multimodal data to obtain speech recognition results; wherein, the speech recognition results include speech feature vectors and speech recognition text; the generative AI model is used to generate speech based on the speech recognition results to obtain speech data.

[0066] The facial micro-expression feature extraction model in this embodiment includes a deep convolutional neural network and a Transformer model; wherein the deep convolutional network is used to perform feature encoding on facial video frames extracted from facial videos to obtain facial video features; the Transformer model is used to perform time-series encoding on facial video features to obtain facial feature vectors, and use the facial feature vectors as the facial micro-expression feature extraction results.

[0067] This embodiment uses a deep convolutional network (CNN) to recognize local expressions (such as those around the eyes and the corners of the mouth) and a Transformer to extract facial micro-expression features. Since CNNs primarily extract local visual features, while Transformers excel at temporal or global contextual association, the combination of the two can more accurately capture short-term, extremely small dynamic micro-expressions.

[0068] In one specific embodiment, a facial video stream is input into a facial micro-expression feature extraction model; key frames are extracted from the facial video stream to obtain facial video frames; face detection and positioning are performed on each frame, the facial area is intercepted or cropped, and the cropped graphics are uniformly scaled to a preset size, and illumination and contrast normalization are performed; each frame image is fine-tuned using a pre-trained ResNet-50 or MobileNet to obtain a d-dimensional embedding; the embeddings of T frames are stacked into a [Batch, T, d] tensor; positional encoding (PE) is introduced to identify the frame sequence index of the [Batch, T, d] tensor, and the [Batch, T, d] tensor is input into a Transformer Encoder to output a temporal representation sequence, i.e., a facial feature vector.

[0069] Among them, the process of feature extraction of facial videos by the lip shape feature extraction model includes: first capturing the human face in real time from the facial video and detecting the key points of the face to obtain the coordinate data of the key points of the lip area of ​​each frame; arranging the coordinate data of the key points of the lip area of ​​each frame in time sequence to obtain a real-time key point sequence; using Kalman filtering or exponential smoothing method to filter the key point sequence in real time; normalizing the real-time key point sequence after real-time filtering; inputting the normalized real-time key point sequence into the lip shape feature extraction model to obtain the lip shape dynamic feature vector as the result of lip shape feature tracking.

[0070] Among them, the dynamic feature vector of the mouth shape in this embodiment includes the characteristics of the lip opening and closing amplitude, the characteristics of the change in the distance between the lip corners, and the characteristics of the lip protrusion or retraction, so as to target the particularly important mouth shape features in the speech training scenario, effectively detect and locate the abnormalities or deficiencies in the mouth shape of specific phonemes, and identify whether the user has truly mastered the action pattern of pronunciation, thereby guiding the training program generation and adjustment module to automatically adjust the difficulty or conduct targeted review training in the training program.

[0071] The eye movement feature extraction results in this embodiment include gaze direction, gaze duration, blink frequency, etc., which are used to evaluate the patient's concentration, understanding and cooperation.

[0072] In this embodiment, the real-time multi-dimensional speech evaluation results include a comprehensive score of multi-dimensional training effects and a user engagement score.

[0073] Among them, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence based on speech data, lip feature tracking results, facial micro-expression feature extraction results and eye tracking results to obtain a real-time multi-dimensional speech evaluation result, specifically including: multi-modally fusing the speech data and the lip feature tracking results to form a first fusion feature; inputting the first fusion feature into a preset multi-dimensional training effect scoring model to obtain a multi-dimensional training effect comprehensive score; splicing the multi-dimensional training effect comprehensive score, facial micro-expression feature extraction results and eye tracking results to form a second fusion feature; inputting the second fusion feature into the user engagement evaluation model to obtain a user engagement score.

[0074] This embodiment obtains a first fusion feature through multimodal fusion, which can more accurately reflect the patient's actual pronunciation quality, thereby generating a more reliable multi-dimensional training effect comprehensive score; and obtains a second fusion feature based on the multi-dimensional training effect comprehensive score, facial micro-expression feature extraction results and eye tracking results; and then inputs the second fusion feature into the user engagement evaluation model to obtain a user engagement score, which greatly improves the accuracy and robustness of the user engagement evaluation model.

[0075] Specifically, multimodal fusion strategies include cross-modal attention mechanisms and cascaded feature fusion strategies. The cross-modal attention mechanism utilizes a Transformer-based model to perform cross-attention interactions between speech features and lip shape features to more accurately capture the correlation between lip shape and speech. The cascaded feature fusion strategy, on the other hand, directly concatenates feature vectors from different modalities.

[0076] It should be understood that the user engagement evaluation model in this embodiment is a facial emotion recognition model or a facial expression coding system trained with the FER2013 dataset to accurately evaluate the user's facial emotions and thus accurately obtain the user's engagement.

[0077] Of course, the multi-dimensional speech evaluation results in this embodiment include a multi-dimensional training effect comprehensive score and a user participation score. The multi-dimensional speech evaluation results are a summary of the real-time multi-dimensional speech evaluation results of all training sentences in each speech training process.

[0078] The multidimensional training effect scoring model in this embodiment includes a pitch detection sub-model, a rhythm detection sub-model, a speaking rate detection sub-model, a pronunciation detection sub-model and a multidimensional training effect comprehensive scoring sub-model; the pitch detection sub-model, the rhythm detection sub-model, the speaking rate detection sub-model and the pronunciation detection sub-model are used to perform pitch, rhythm, speaking rate and pronunciation accuracy detection on the first fusion feature respectively to obtain pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking rate accuracy; the multidimensional training effect comprehensive scoring sub-model is used to perform a multidimensional comprehensive evaluation based on pitch accuracy, rhythm accuracy, pronunciation accuracy, speaking rate accuracy and preset weights to obtain a multidimensional training effect comprehensive score.

[0079] This configuration of the present embodiment can evaluate the user's voice quality from multiple dimensions, making the evaluation more objective and accurate.

[0080] In order to accurately calculate the pronunciation accuracy, the pronunciation detection submodel in this embodiment includes a self-supervised speech recognition model and an anomaly detection head; during the pronunciation detection process, the self-supervised speech recognition model is used to detect the pronunciation accuracy of the first fusion feature to obtain the initial pronunciation accuracy; it is determined whether the initial pronunciation accuracy is higher than the preset pronunciation accuracy. If so, the initial pronunciation accuracy is used as the final pronunciation accuracy; if not, the first fusion feature is segmented using the anomaly detection head to obtain multiple intermediate segments; the multiple intermediate segments are then aligned segment by segment to obtain the alignment result; it is determined whether there is a key vowel or phoneme in the alignment result whose duration is lower than the preset range; if so, the key vowel or phoneme is marked as "suspected missing" or "abnormal pronunciation"; the pathological data and the key vowels or phonemes marked as "suspected missing" or "abnormal pronunciation" are weighted according to the preset penalty coefficient or the actual proportion to obtain the final pronunciation accuracy.

[0081] Among them, the final pronunciation accuracy = (normal phoneme count - α × abnormal phoneme count) / total phoneme count; the abnormal phoneme count is the number of key vowels or phonemes marked as "suspected missing" or "abnormal pronunciation", the total phoneme count represents the total number of phonemes in the speech data, the normal phoneme count represents the number of normal phonemes in the speech data, and α represents the weight of the abnormal phoneme count.

[0082] The speech training program in this embodiment is a personalized speech training program.

[0083] In this embodiment, the difficulty of the next training sentence in the speech training program is adaptively and dynamically adjusted based on the real-time multi-dimensional speech evaluation results, specifically including:

[0084] According to the real-time multi-dimensional speech evaluation results, the difficulty of the next training sentence in the speech training plan of the user during each speech training is dynamically adjusted in real time to obtain an intermediate speech training plan;

[0085] The intermediate speech training program is personalized based on the personalized parameters pre-entered or selected by the user.

[0086] This embodiment is configured in such a way that the finally generated speech training program can be consistent with the user's current situation, interests and preferences, further improving the user's experience, participation and training effect when using the generated speech training program for subsequent training in each training process.

[0087] In this embodiment, after each first preset number of speech training processes, the method further includes:

[0088] A reinforcement learning algorithm is used to adjust the weights of each indicator of pitch accuracy, rhythm accuracy, pronunciation accuracy, and speaking speed accuracy based on the pitch accuracy, rhythm accuracy, pronunciation accuracy, speaking speed accuracy and the comprehensive score of multi-dimensional training effects during the first preset number of speech training times.

[0089] In the weight adjustment process, if the improvement degree of any one of the indicators among pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking speed accuracy is lower than the preset improvement threshold, the weight corresponding to the indicator is increased;

[0090] If the degree of improvement of one of the indicators among pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking speed accuracy is higher than or equal to the preset improvement threshold, the weight corresponding to the indicator is reduced or maintained.

[0091] This embodiment is configured in such a way that the weights of various indicators can be regularly adjusted according to the individual conditions and progress of the users, thereby being more suitable for the users and further improving the user experience and participation.

[0092] In a further embodiment, after each second preset number of speech trainings, it also includes: dynamically adjusting the speech training plan based on a preset training plan dynamic adjustment strategy or / and a reinforcement learning algorithm according to the speech training plan, multi-dimensional speech evaluation results in the current speech training process and the speech training plan and multi-dimensional speech evaluation results in the second preset number of speech training processes.

[0093] In order to improve the adaptability of the dynamically adjusted speech training plan to users, in one specific embodiment, in the adjustment process before the preset cumulative training times, the speech training plan is adjusted according to the preset training plan dynamic adjustment strategy; in the adjustment process after the preset cumulative training times, the speech training plan is dynamically adjusted using Deep Q-Learning or a multi-armed bandit algorithm based on the speech training plan, the multi-dimensional speech evaluation results in the current speech training process, and the speech training plan and the multi-dimensional speech evaluation results during the preset historical number of speech training sessions.

[0094] In a further specific embodiment, in the dynamic adjustment strategy of the training program, if after a second preset number of speech training sessions, the user's multi-dimensional speech evaluation result is greater than or equal to the ideal speech training effect and the user's participation is greater than or equal to a preset participation threshold, the difficulty of the speech training program is increased;

[0095] If, after a preset number of historical speech training sessions, the user's multi-dimensional speech assessment results are greater than or equal to the ideal speech training effect and the user's engagement is less than a preset engagement threshold, the difficulty of the speech training program is increased and personalized parameters are adjusted;

[0096] If, after a preset number of historical speech training sessions, the user's multi-dimensional speech evaluation results are less than the ideal speech training effect and the user's engagement is greater than or equal to a preset engagement threshold, the difficulty of the speech training program is reduced;

[0097] If after a preset number of historical speech training sessions, the user's multi-dimensional speech evaluation results are less than the ideal speech training effect and the user's participation is less than a preset participation threshold, the difficulty of the speech training program will be reduced and personalized parameters will be adjusted or / and an emotional encouragement program will be generated.

[0098] This embodiment can dynamically adjust the difficulty of the speech training program while adjusting personalized parameters or / and generating an emotional encouragement program to meet the user's preferences or / and actively provide emotional comfort or interactive encouragement to increase the user's enthusiasm and participation in speech training.

[0099] Specifically, the results of multi-dimensional speech assessments are used as reward / penalty signals; based on the user's progress or regression in various indicators, the training content, difficulty, and pacing are dynamically adjusted. For example, if a patient's pitch accuracy improves significantly, the weight of this training item is reduced; conversely, the weight of pronunciation accuracy training is increased. Furthermore, the training plan generation and adjustment module is used to increase the number of incorrect question sets that appear at irregular intervals to determine whether the user has truly mastered the training content.

[0100] In a second aspect, the present invention further provides an artificial intelligence-based speech training system for implementing the artificial intelligence-based speech training method according to any one of the first aspects, the system comprising:

[0101] An acquisition module is used to acquire multimodal data of each training sentence read out by the user in real time during each speech training process;

[0102] The evaluation module is used to perform real-time multi-dimensional evaluation of the user's speech training effect on each training sentence based on multimodal data to obtain real-time multi-dimensional speech evaluation results;

[0103] A judgment module is used to judge whether the user has the next training sentence in the speech training plan during each speech training process;

[0104] A program generation and adjustment module is used to adaptively and dynamically adjust the difficulty of the next training sentence in the speech training program based on the real-time multi-dimensional speech evaluation results;

[0105] The training module is used to output the next training sentence according to the adjusted speech training plan when the next training sentence exists in the speech training plan;

[0106] The evaluation module is also used to generate a multidimensional speech evaluation result for each speech training process based on the real-time multidimensional speech evaluation results corresponding to all training sentences in each speech training process when there is no next training sentence in the speech training plan.

[0107] During the evaluation process of the multi-dimensional speech assessment results, the assessment module first uses a preset feature extraction model to perform real-time feature extraction on the multi-modal data based on the multi-modal data to obtain speech data, lip shape feature tracking results, facial micro-expression feature extraction results and eye tracking results; then, based on the speech data, lip shape feature tracking results, facial micro-expression feature extraction results and eye tracking results, a real-time multi-dimensional assessment of the user's speech training effect on each training sentence is performed to obtain real-time multi-dimensional speech assessment results.

[0108] The real-time multi-dimensional speech evaluation results in this embodiment include a multi-dimensional training effect comprehensive score and a user engagement score; wherein, the process of obtaining the real-time multi-dimensional speech evaluation results includes: multimodally fusing the speech voice data and the lip feature tracking results to form a first fusion feature; inputting the first fusion feature into a preset multi-dimensional training effect scoring model to obtain a multi-dimensional training effect comprehensive score; splicing the multi-dimensional training effect comprehensive score, the facial micro-expression feature extraction results and the eye tracking results to form a second fusion feature; inputting the second fusion feature into the user engagement evaluation model to obtain a user engagement score.

[0109] It should be noted that the feature extraction model, multi-dimensional training effect scoring model and user engagement evaluation model in this embodiment are the same as those described in the first aspect and will not be elaborated here.

[0110] After each first preset number of speech training processes, the evaluation module is also used to adopt a reinforcement learning algorithm to adjust the weights of each indicator of pitch accuracy, rhythm accuracy, pronunciation accuracy, and speaking speed accuracy based on the pitch accuracy, rhythm accuracy, pronunciation accuracy, speaking speed accuracy and the comprehensive score of multi-dimensional training effects in the first preset number of speech training processes.

[0111] During the weight adjustment process, if the improvement of any one of the indicators of pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking speed accuracy is lower than the preset improvement threshold, the weight corresponding to the indicator is increased;

[0112] If the degree of improvement of one of the indicators among pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking speed accuracy is higher than or equal to the preset improvement threshold, the weight corresponding to the indicator is reduced or maintained.

[0113] After each second preset number of speech trainings, the program generation and adjustment module is also used to dynamically adjust the speech training program based on the preset training program dynamic adjustment strategy or / and reinforcement learning algorithm according to the speech training program, multi-dimensional speech evaluation results in the current speech training process and the speech training program and multi-dimensional speech evaluation results in the second preset number of speech trainings.

[0114] Among them, in the adjustment process before the preset cumulative training times, the speech training plan is adjusted according to the preset training plan dynamic adjustment strategy; in the adjustment process after the preset cumulative training times, the speech training plan is dynamically adjusted according to the speech training plan, multi-dimensional speech evaluation results in the current speech training process and the speech training plan and multi-dimensional speech evaluation results during the preset historical number of speech training times.

[0115] Among them, in the dynamic adjustment strategy of the training program, if after the second preset number of speech training, the user's multi-dimensional speech evaluation result is greater than or equal to the ideal speech training effect and the user's participation is greater than or equal to the preset participation threshold, the difficulty of the speech training program is increased;

[0116] If, after a preset number of historical speech training sessions, the user's multi-dimensional speech assessment results are greater than or equal to the ideal speech training effect and the user's engagement is less than a preset engagement threshold, the difficulty of the speech training program is increased and personalized parameters are adjusted;

[0117] If, after a preset number of historical speech training sessions, the user's multi-dimensional speech evaluation results are less than the ideal speech training effect and the user's engagement is greater than or equal to a preset engagement threshold, the difficulty of the speech training program is reduced;

[0118] If after a preset number of historical speech training sessions, the user's multi-dimensional speech evaluation results are less than the ideal speech training effect and the user's participation is less than a preset participation threshold, the difficulty of the speech training program will be reduced and personalized parameters will be adjusted or / and an emotional encouragement program will be generated.

[0119] In this embodiment, the difficulty of the next training sentence in the speech training plan is adaptively and dynamically adjusted based on the real-time multi-dimensional speech evaluation results, specifically including: dynamically adjusting the difficulty of the next training sentence in the speech training plan of the user during each speech training in real time based on the real-time multi-dimensional speech evaluation results to obtain an intermediate speech training plan; and personalizing the intermediate speech training plan based on personalized parameters pre-input or selected by the user.

[0120] In this embodiment, it also includes:

[0121] Tracking module, used to track the user's voice and face in real time to obtain multimodal data;

[0122] The display module is used to visualize the training content of the speech training program and the multi-dimensional speech assessment results in real time.

[0123] Third, current rehabilitation training methods for aphasia and other neurologically related speech disorders primarily rely on traditional treatments, such as speech-language therapy, transcranial magnetic stimulation (TMS), melodic intonation therapy (MIT), rhythmic speech cues (RSC), and musical sensory stimulation training (MUSTIM), as well as some AI-based speech rehabilitation technologies. However, these traditional treatments rely on the therapist's expertise and subjective judgment, are limited in rehabilitation resources, and are difficult to promote. Furthermore, users lack objective speech feedback during training.

[0124] Existing AI-based speech rehabilitation systems lack feedback and multimodal interaction, making it difficult to obtain multi-dimensional training guidance. The training is monotonous and user participation is low.

[0125] Therefore, the present invention also proposes an application of an artificial intelligence-based speech training system, which is applied to the field of speech rehabilitation training to perform speech rehabilitation training for users with aphasia or speech disorders.

[0126] Therefore, the artificial intelligence-based speech training system applied to the field of speech rehabilitation training in this embodiment is also called an artificial intelligence-based speech rehabilitation training system.

[0127] In one specific embodiment, the artificial intelligence-based speech rehabilitation training system includes:

[0128] An acquisition module is used to acquire multimodal data of each training sentence read out by the user in real time during each speech rehabilitation training session;

[0129] An evaluation module is used to perform a multi-dimensional evaluation of the user's speech rehabilitation training effect on the training sentence in real time based on the multimodal data, and obtain a real-time multi-dimensional speech rehabilitation evaluation result;

[0130] A judgment module is used to judge whether there is a next training sentence in the personalized speech rehabilitation training program;

[0131] A program generation and adjustment module is used to adaptively and dynamically adjust the difficulty of the next training sentence of the user during each speech rehabilitation training session based on the real-time multi-dimensional speech rehabilitation assessment results when the next training sentence exists in the personalized speech rehabilitation training program, thereby obtaining a personalized speech rehabilitation training program;

[0132] The training module is used to output the next training sentence according to the personalized speech rehabilitation training program;

[0133] The evaluation module is also used to generate a multidimensional speech rehabilitation evaluation result for each speech rehabilitation training process based on the real-time multidimensional speech rehabilitation evaluation results corresponding to all training sentences in each speech rehabilitation training process when there is no next training sentence in the personalized speech rehabilitation training plan.

[0134] This embodiment can obtain multidimensional speech rehabilitation evaluation results in real time during the speech rehabilitation training process based on sentence-level multimodal data (i.e., multimodal data of each training sentence), and instantly update the difficulty of the next training sentence in the speech rehabilitation training plan based on the multidimensional speech rehabilitation evaluation results of each training sentence, so that the finally generated speech training plan is in line with the user's current situation, interests and preferences, which is conducive to improving the user's experience, participation and training effect when using the generated speech training plan for subsequent training in each training process, and can continuously optimize the speech training plan during the training of subsequent training sentences to achieve more efficient and personalized speech rehabilitation training.

[0135] Among them, the training module in this embodiment is used to convert the corpus text in the preset speech rehabilitation training corpus into training sentences of different musical styles used by users for speech rehabilitation training through speech-to-melody conversion technology. Moreover, the training module in this embodiment supports different neuromusic therapy speech-directed technologies, such as melodic tone therapy, rhythmic voice prompting method, and music sensory stimulation training method. Among them, the training types of the training module include music initials and finals reinforcement exercises, music word exercises, music short sentence exercises, music scene dialogue exercises, and de-music scene generalization applications.

[0136] The evaluation module in this embodiment is the same as the evaluation module in the second aspect, and will not be described in detail here. The solution generation and adjustment module in this embodiment is the same as the solution generation and adjustment module in the second aspect, and will not be described in detail here.

[0137] The personalized parameters in this embodiment include: music type, instrument, pitch, language type and dialect, tone variation and rhythm pattern.

[0138] Specifically, the self-supervised speech recognition model in this embodiment is a Wav2Vec, HuBERT or Whisper model, and the generative AI model is a diffusion model (Diffusion), a variational autoencoder (VAE) or a generative adversarial network (GANs).

[0139] During the training process of the speech feature extraction model in this embodiment, a first training set and a second training set are first constructed; wherein, the first training set only includes normal speech data, and the second training set only includes pathological data; the first training set is used to perform self-supervised pre-training on the speech feature extraction model to obtain an intermediate speech feature extraction model with good generalization; and then the second training set is used to perform secondary fine-tuning training on the intermediate speech feature extraction model to obtain a trained speech feature extraction model; wherein, during the secondary fine-tuning training process, the objective function is a weighted loss function, such as focal loss FocalLoss, to solve the data imbalance problem and ensure high sensitivity to abnormal speech.

[0140] The process of constructing the second training set in this embodiment specifically includes: obtaining a pathology dataset labeled with pathology speech types; performing data enhancement on the pathology dataset, and using the enhanced pathology dataset as the second training set. The data enhancement includes GAN-based pathology speech synthesis enhancement, diffusion model enhancement, or speech signal enhancement.

[0141] Of course, the selection of pathological speech in this embodiment needs to be based on representative speech samples of different aphasia subtypes, and detailed speech classification and labeling of representative speech samples of various aphasia subtypes is required to enhance the comprehensiveness and sensitivity of the speech feature extraction model to different abnormal categories.

[0142] During initial use, weight coefficients for pitch accuracy, rhythm accuracy, pronunciation accuracy, and speech rate accuracy are set based on clinical expert recommendations, i.e., a pre-set multi-dimensional training effect comprehensive scoring strategy. Later in use, a reinforcement learning approach can be used to adjust these weights based on a pre-set number of rehabilitation training sessions, including pitch accuracy, rhythm accuracy, pronunciation accuracy, speech rate accuracy, and the comprehensive multi-dimensional training effect score. Specifically, the reinforcement learning approach is Deep Q-Learning (DQL).

[0143] In the pre-set multi-dimensional training effect comprehensive scoring strategy, the weighting coefficients for pitch, rhythm, articulation, and speed are also dynamically adjusted based on the user's aphasia type or severity. For example, if a user's aphasia type is Broca's aphasia (non-fluent aphasia), then the user's main characteristics are: difficulty in pronunciation, significantly impaired rhythm, but relatively preserved comprehension ability. The pre-set multi-dimensional training effect comprehensive scoring strategy for this user would recommend increasing the weights for pronunciation and rhythm scores, thereby highlighting the importance of "pronunciation and rhythm accuracy" in the multi-dimensional training effect comprehensive scoring. For example, if a user's aphasia type is Wernicke's aphasia (fluent but error-prone), then the user's main characteristics are: normal or even fast speaking speed, but with a large number of incorrect sounds or words, while the rhythm may be good and the pitch variation may be normal. The pre-set multi-dimensional training effect comprehensive scoring strategy for this user would recommend increasing the weighting for pronunciation accuracy to emphasize sensitivity to language error detection. For example, if the user's aphasia type is mild aphasia or mild language impairment after a stroke, the main characteristics of this user are: most parameters are relatively normal, but there may only be slight deviations in speed or pitch. In the preset multi-dimensional training effect comprehensive scoring strategy, the recommendation for this user is to moderately increase the weight of speed and pitch. For example, when the speech rehabilitation assessment module conducts a preset number of rehabilitation trainings during a rehabilitation cycle, and then finds that the "pronunciation accuracy" continues to be below 50 points, based on the reinforcement learning method, the weight of "pronunciation accuracy" in the comprehensive scoring of the multi-dimensional training effect is automatically increased, and pronunciation correction training is added in subsequent training.

[0144] In this embodiment, it also includes:

[0145] Tracking module, used to track the user's voice and face in real time to obtain multimodal data;

[0146] The display module is used to visualize the training content of the speech training program and the multi-dimensional speech rehabilitation assessment results in real time.

[0147] Specifically, the display module can display the rhythm of the training content of the speech rehabilitation training program in the form of a visual metronome, and the pitch of the multi-dimensional speech rehabilitation assessment results in real time in the form of a curve, that is, a real-time graphical comparison of the pronunciation data and the standard data. When the user's pitch is too low or the rhythm is behind, the display module will display a striking mark or animation, so that the patient can clearly see where the pronunciation problem exists and how big the gap is, which is convenient for guiding the user to make immediate corrections, effectively improving interactivity, user experience and training effects.

[0148] In a further embodiment, a virtual human module is further included, which is used to perform real-time lip movements according to the speech rehabilitation training program and / or provide emotional encouragement according to the emotional encouragement program; and the display module is further used to display the virtual human in real time. In this embodiment, the virtual human lip movements allow users to see specific lip movements, which can enhance interactivity and user experience.

[0149] In this embodiment, the training module includes an auditory demonstration submodule, a visual demonstration submodule and a right brain functional area stimulation module; the visual demonstration submodule is used to use a virtual person to perform lip demonstration; the auditory demonstration submodule is used to perform sound demonstration; the right brain functional area stimulation submodule is used to use a visual metronome for rhythm guidance, and the pitch of the lip demonstration or sound demonstration is visually compared with the user's real-time voice through the display module.

[0150] During specific implementation, the virtual human switches between "demonstration" and "guidance" modes in real time according to the multi-dimensional speech rehabilitation assessment results during the rehabilitation training process, provides emotional encouragement according to the emotional encouragement plan, activates the virtual human's encouraging words, changes songs, plays mini-games or provides short breaks, and adjusts the training rhythm, thereby improving the user's subsequent participation and experience.

[0151] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any technician familiar with the technical field, within the technical scope disclosed by the present invention, who makes equivalent replacements or changes based on the technical solution and inventive concept of the present invention, should be covered by the scope of protection of the present invention.

Claims

1. A speech training method based on artificial intelligence, characterized in that: include: S1. Acquire multimodal data of each training sentence read out by the user in real time during each speech training process; wherein reading includes one or more of singing, chanting, reciting, reciting, reading, sighing, humming, moaning, satirizing, and reciting; S2. Based on the multimodal data, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence to obtain a real-time multi-dimensional speech evaluation result; S3, determining whether the user has a next training sentence in the speech training plan during each speech training process; If yes, go to S4; if no, go to S6; S4. Adaptively and dynamically adjust the difficulty of the next training sentence in the speech training program based on the real-time multi-dimensional speech assessment results; S5. Output the next training sentence according to the adjusted speech training plan and enter S1; S6. Generate a multi-dimensional speech evaluation result for each speech training process based on the real-time multi-dimensional speech evaluation results corresponding to all training sentences in each speech training process.

2. The artificial intelligence-based speech training method according to claim 1, characterized in that: Multimodal data includes voice data and facial videos; In S2, based on the multimodal data, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence, and a real-time multi-dimensional speech evaluation result is obtained, specifically including: Based on the multimodal data, a preset feature extraction model is used to extract features from the multimodal data in real time to obtain speech data, lip shape feature tracking results, facial micro-expression feature extraction results, and eye tracking results; Based on speech data, lip shape feature tracking results, facial micro-expression feature extraction results and eye tracking results, a real-time multi-dimensional evaluation of the user's speech training effect on each training sentence is performed to obtain real-time multi-dimensional speech evaluation results.

3. The speech training method based on artificial intelligence according to claim 2, characterized in that: Real-time multi-dimensional speech evaluation results include a comprehensive score of multi-dimensional training effects and a user engagement score; Among them, a real-time multi-dimensional evaluation is performed on the user's speech training effect on each training sentence based on speech data, lip feature tracking results, facial micro-expression feature extraction results and eye tracking results to obtain a real-time multi-dimensional speech evaluation result, specifically including: multi-modally fusing the speech data and the lip feature tracking results to form a first fusion feature; inputting the first fusion feature into a preset multi-dimensional training effect scoring model to obtain a multi-dimensional training effect comprehensive score; splicing the multi-dimensional training effect comprehensive score, facial micro-expression feature extraction results and eye tracking results to form a second fusion feature; inputting the second fusion feature into the user engagement evaluation model to obtain a user engagement score.

4. The speech training method based on artificial intelligence according to claim 1, characterized in that: After each first preset number of speech training processes, the method further includes: Using a reinforcement learning algorithm, the weights of the pitch accuracy, rhythm accuracy, pronunciation accuracy, and speaking speed accuracy indicators are adjusted based on the pitch accuracy, rhythm accuracy, pronunciation accuracy, and speaking speed accuracy during the first preset number of speech training sessions and the comprehensive score of the multi-dimensional training effects; During the weight adjustment process, if the improvement of any one of the indicators of pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking speed accuracy is lower than the preset improvement threshold, the weight corresponding to the indicator is increased; If the degree of improvement of one of the indicators among pitch accuracy, rhythm accuracy, pronunciation accuracy and speaking speed accuracy is higher than or equal to the preset improvement threshold, the weight corresponding to the indicator is reduced or maintained.

5. The artificial intelligence-based speech training method according to claim 1, characterized in that: Adaptively and dynamically adjust the difficulty of the next training sentence in the speech training program based on the real-time multi-dimensional speech assessment results, specifically including: According to the real-time multi-dimensional speech evaluation results, the difficulty of the next training sentence in the speech training plan of the user during each speech training is dynamically adjusted in real time to obtain an intermediate speech training plan; The intermediate speech training program is personalized based on the personalized parameters pre-entered or selected by the user.

6. The artificial intelligence-based speech training method according to claim 5, characterized in that: After each second preset number of speech trainings, it also includes: dynamically adjusting the speech training plan based on a preset training plan dynamic adjustment strategy or / and a reinforcement learning algorithm according to the speech training plan in the current speech training process, the multi-dimensional speech evaluation results, and the speech training plan and multi-dimensional speech evaluation results of the second preset number of speech training processes.

7. The artificial intelligence-based speech training method according to claim 6, characterized in that: During the adjustment process before the preset cumulative training times, the speech training plan is adjusted according to the preset training plan dynamic adjustment strategy; during the adjustment process after the preset cumulative training times, the speech training plan is dynamically adjusted using Deep Q-Learning or a multi-armed bandit algorithm based on the speech training plan, multi-dimensional speech evaluation results in the current speech training process, and the speech training plan and multi-dimensional speech evaluation results during the preset historical number of speech training times.

8. The artificial intelligence-based speech training method according to claim 6 or 7, characterized in that: In the dynamic adjustment strategy of the training program, if after the second preset number of speech training sessions, the user's multi-dimensional speech evaluation results are greater than or equal to the ideal speech training effect and the user's participation is greater than or equal to the preset participation threshold, the difficulty of the speech training program is increased; If, after a preset number of historical speech training sessions, the user's multi-dimensional speech assessment results are greater than or equal to the ideal speech training effect and the user's engagement is less than a preset engagement threshold, the difficulty of the speech training program is increased and personalized parameters are adjusted; If, after a preset number of historical speech training sessions, the user's multi-dimensional speech evaluation results are less than the ideal speech training effect and the user's engagement is greater than or equal to a preset engagement threshold, the difficulty of the speech training program is reduced; If after a preset number of historical speech training sessions, the user's multi-dimensional speech evaluation results are less than the ideal speech training effect and the user's participation is less than a preset participation threshold, the difficulty of the speech training program will be reduced and personalized parameters will be adjusted or / and an emotional encouragement program will be generated.

9. A speech training system based on artificial intelligence, characterized in that: include: An acquisition module is used to acquire multimodal data of each training sentence read out by the user in real time during each speech training process; The evaluation module is used to perform real-time multi-dimensional evaluation of the user's speech training effect on each training sentence based on multimodal data to obtain real-time multi-dimensional speech evaluation results; A judgment module is used to judge whether the user has the next training sentence in the speech training plan during each speech training process; A program generation and adjustment module is used to adaptively and dynamically adjust the difficulty of the next training sentence in the speech training program according to the real-time multi-dimensional speech evaluation results when the next training sentence exists in the speech training program; The training module is used to output the next training sentence according to the speech training plan; The evaluation module is also used to generate a multidimensional speech evaluation result for each speech training process based on the real-time multidimensional speech evaluation results corresponding to all training sentences in each speech training process when there is no next training sentence in the speech training plan.

10. An application of a speech training system based on artificial intelligence, characterized in that: The artificial intelligence-based speech training system as claimed in claim 9 is applied in the field of speech rehabilitation training.

Images