AI-based preschool special child language rehabilitation training guiding method

Abstract

The invention relates to an AI-based preschool special child language rehabilitation training guiding method. The method comprises the following steps: acquiring multi-modal data of a preschool special child, carrying out asynchronous alignment and feature fusion processing on the multi-modal data, and generating a time-synchronous fusion feature vector; performing pronunciation deviation correction modeling on the fused feature vector based on adversarial transfer learning to generate standardized speech features; inputting the standardized speech features into a pre-training model based on a federal starting optimization algorithm to generate a personalized training task; and training the preschool special child through the personalized training task to obtain real-time training data, and dynamically adjusting the difficulty level of the personalized training task by using a reinforcement learning strategy based on the data to obtain an updated personalized training task. According to the method, through multi-modal data fusion, pronunciation deviation correction, personalized task generation and dynamic task optimization, pertinence, accuracy and effectiveness of language rehabilitation training can be effectively improved, and the rehabilitation training effect is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of language rehabilitation training, and in particular relates to an AI-based language rehabilitation training guidance method for preschool special children. Background Art

[0002] With the cross-integration of artificial intelligence and medical rehabilitation technology, AI-based intelligent language rehabilitation training has gradually become a research hotspot in the field of rehabilitation for children with special needs. This technology provides new technical solutions for language rehabilitation training for preschool children with special needs through intelligent data analysis and model building. However, this technology is usually trained based on a general model and cannot be dynamically and finely adjusted according to the unique situation of each child, resulting in poor training results for some children and difficulty meeting personalized rehabilitation needs. In addition, this technology usually relies on voice signals for pronunciation assessment, calculates deviation values through matching preset standardized voice templates, and generates training tasks of fixed difficulty. It lacks a real-time tracking and feedback adjustment mechanism for progress and changes during rehabilitation training. Summary of the Invention

[0003] Based on this, it is necessary to provide an AI-based language rehabilitation training guidance method for preschool children with special needs in response to the above-mentioned technical problems, so as to improve the accuracy and personalization of training and enhance the effect of rehabilitation training.

[0004] In the first aspect, this application provides an AI-based language rehabilitation training guidance method for preschool children with special needs, including:

[0005] Acquire multimodal data of preschool children with special needs, perform asynchronous alignment and feature fusion processing on the multimodal data, and generate time-synchronized fused feature vectors. The multimodal data includes original speech signals, facial expression image sequences, and body movement trajectories.

[0006] Based on adversarial transfer learning, the fused feature vector is modeled for pronunciation deviation correction to generate standardized speech features.

[0007] Input standardized speech features into a pre-trained model based on the federated startup optimization algorithm to generate personalized training tasks;

[0008] Preschool children with special needs are trained through personalized training tasks to obtain real-time training data. The difficulty level of the personalized training tasks is dynamically adjusted using reinforcement learning strategies based on the real-time training data to obtain updated personalized training tasks. The real-time training data includes voice interaction data, facial movement data, body movement data, and emotional physiological data.

[0009] In one embodiment, multimodal data of preschool children with special needs is obtained, and asynchronous alignment and feature fusion processing are performed on the multimodal data to generate a time-synchronized fused feature vector, including:

[0010] Perform noise reduction and phoneme segmentation on the original speech signal to extract MFCC features and fundamental frequency contour features;

[0011] Perform key point tracking on facial expression image sequences to extract facial visual features, including eye gaze direction features, mouth corner curvature features, and head posture angle features;

[0012] A multimodal attention alignment loss function is constructed to perform timestamp compensation on MFCC features, eye gaze direction features, and limb movement trajectories to obtain optimized MFCC features, optimized eye gaze direction features, and optimized limb movement trajectories.

[0013] The optimized MFCC features, optimized eye gaze direction features and optimized limb movement trajectories are input into the preset cross-modal cross-attention fusion network for fusion, and the fused feature vector is output.

[0014] In one embodiment, the multimodal attention alignment loss function is calculated as:

[0015]

[0016] Among them, L align is the modality attention alignment loss function, S(t i ) is the original speech signal, V(t i ) is a facial expression image sequence, α i is the dynamic weight matrix, JSD is the Jensen-Shannon divergence, which is used to constrain the consistency of the joint distribution of speech-expression p(S,V) and the action distribution p(M), β is a hyperparameter, is a regularization term to prevent the model from overfitting, and γ is used to control the influence of the regularization term.

[0017] In one embodiment, pronunciation deviation correction modeling is performed on the fused feature vector based on adversarial transfer learning to generate standardized speech features, including:

[0018] The pathological speech features are separated from the fused feature vector through a feature decoupling network, and a pre-trained standard speech library is obtained. The standard speech library contains standard pronunciation templates for children of different age groups.

[0019] Construct a generative adversarial network, and based on a pre-trained standard speech library, input pathological speech features into the generative adversarial network to output pseudo-standard speech features;

[0020] A dynamic mask matrix is generated based on the dialect distribution of preschool children with special needs in their area. Pseudo-standard speech features and pathological speech features are weighted and fused according to the dynamic mask matrix to generate dialect-corrected speech features. The weights of the dynamic mask matrix are dynamically adjusted through end-to-end training.

[0021] The dialect-corrected speech features were optimized with vocal tract formant constraints to obtain standardized speech features and pronunciation scores. The pronunciation scores were calculated by comparing the similarity between the dialect-corrected speech features and the children's standard pronunciation template.

[0022] In one embodiment, standardized speech features are input into a pre-trained model based on a federated startup optimization algorithm to generate personalized training tasks, including:

[0023] Obtain model parameters for a pre-trained model, which is trained based on a cross-institutional historical rehabilitation dataset that includes speech feature mappings for children with autism, children with language delays, and children with hearing impairments.

[0024] A local personalized model is constructed, and local parameters of the local personalized model are initialized as a weighted average of the model parameters, where the weight of the weighted average is dynamically allocated according to the type of impairment of the preschool special children;

[0025] The standardized speech features and pronunciation scores are combined into local training data, and the local parameters are updated using the differentially private federated distillation algorithm to obtain the updated local parameters.

[0026] Generate a task parameter set based on the updated local parameters, which includes vocabulary threshold, speech rate range, and grammatical complexity level;

[0027] The initial task difficulty in the task parameter set is dynamically adjusted based on the pronunciation score, and personalized training tasks are generated through an adaptive task allocation strategy combined with the real-time attention data of preschool special children.

[0028] In one embodiment, preschool children with special needs are trained through personalized training tasks, real-time training data is obtained, and a reinforcement learning strategy is used based on the real-time training data to dynamically adjust the current task difficulty level of the personalized training task to obtain an updated personalized training task, including:

[0029] The speech interaction data is processed through the speech recognition model to calculate the pronunciation accuracy;

[0030] Key frame analysis is performed on facial action data to calculate the degree of attention concentration;

[0031] Calculate the emotional fluctuation index based on body movement data and emotional physiological data;

[0032] Construct reinforcement learning state vectors through pronunciation accuracy, attention concentration and emotion fluctuation index;

[0033] Based on the reinforcement learning state vector, the Q-learning algorithm is used to perform strategy update processing to obtain a difficulty level adjustment action. The difficulty level adjustment action is any one of increasing the difficulty level, maintaining the current difficulty level, and decreasing the difficulty level.

[0034] Through the difficulty level adjustment action, the vocabulary threshold and speech speed range of the personalized training task are dynamically adjusted to generate an updated personalized training task.

[0035] In one embodiment, the method further comprises:

[0036] When the pronunciation accuracy is greater than a first threshold and the attention concentration is greater than a second threshold, generating encouraging voice instructions;

[0037] When the emotional stability index is less than a third threshold, a soothing voice command is generated;

[0038] If the emotional stability index is lower than the safety threshold, an emergency intervention instruction is generated, which is used to stop the personalized training task.

[0039] Secondly, this application also provides an AI-based language rehabilitation training guidance system for preschool children with special needs, including:

[0040] The data acquisition and processing module is used to obtain multimodal data of preschool children with special needs, perform asynchronous alignment and feature fusion processing on the multimodal data, and generate time-synchronized fused feature vectors. The multimodal data includes original speech signals, facial expression image sequences, and body movement trajectories;

[0041] The original speech correction module is used to perform pronunciation deviation correction modeling on the fused feature vector based on adversarial transfer learning to generate standardized speech features;

[0042] A personalized task generation module is used to input standardized speech features into a pre-trained model based on the federated start optimization algorithm to generate personalized training tasks;

[0043] The real-time task adjustment module is used to train preschool children with special needs through personalized training tasks, obtain real-time training data, and dynamically adjust the difficulty level of personalized training tasks using reinforcement learning strategies based on the real-time training data to obtain updated personalized training tasks. The real-time training data includes voice interaction data, facial movement data, body movement data, and emotional physiological data.

[0044] In a third aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps in the first aspect when executing the computer program.

[0045] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the first aspect when executed by a processor.

[0046] This AI-based speech rehabilitation training guidance method for preschool children with special needs effectively addresses the time asynchrony and feature fusion issues inherent in traditional speech rehabilitation training by asynchronously aligning and fusion-enhancing multimodal data. It comprehensively integrates multi-dimensional information, including raw speech signals, facial expression image sequences, and body movement trajectories, providing a comprehensive data foundation for subsequent pronunciation deviation correction and training task generation. Furthermore, by using adversarial transfer learning to model pronunciation deviation correction for the fused feature vectors, it automatically corrects for common pronunciation deviations in preschool children with special needs and generates standardized speech features. Secondly, the standardized speech features are fed into a pre-trained model based on a federated start optimization algorithm. This model is then personalized using cross-institutional data to generate personalized training tasks, addressing the poor generalization and lack of personalization inherent in traditional centralized training models. Finally, by acquiring real-time training data and dynamically adjusting the difficulty level of personalized training tasks using a reinforcement learning strategy based on this data, the method monitors children's mood swings, attention span, and other aspects in real time, enabling timely adjustments to training tasks to prevent frustration and enhance the continuity and effectiveness of training.

[0047] Compared with traditional language rehabilitation training methods, this method improves the accuracy, adaptability and effectiveness of language rehabilitation training for preschool children with special needs through multimodal data processing, pronunciation deviation modeling, personalized task generation and dynamic adjustment based on real-time data, providing a more scientific and efficient technical solution for language rehabilitation training for preschool children with special needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 A flowchart of an AI-based language rehabilitation training guidance method for preschool children with special needs is provided as an exemplary embodiment of the present invention;

[0050] Figure 2 A schematic structural diagram of an AI-based language rehabilitation training guidance system for preschool children with special needs is provided as an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0052] In one embodiment, Figure 1 As shown, an AI-based language rehabilitation training guidance method for preschool children with special needs is provided. This embodiment uses the method applied to a terminal as an example. It is understandable that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0053] S101: Acquire multimodal data of preschool children with special needs, perform asynchronous alignment and feature fusion processing on the multimodal data, and generate time-synchronized fusion feature vectors. The multimodal data includes original speech signals, facial expression image sequences, and body movement trajectories.

[0054] Specifically, the original speech signals of preschool children with special needs can be collected through a microphone to further analyze their pronunciation accuracy, speech fluency, and pitch changes. Facial expression image sequences can be collected through a camera to analyze their emotional state, mouth shape, and facial muscle movements. Children's body movement data can be collected through a camera or motion capture device to further analyze their body language and movement coordination. And because data acquisition devices of different modalities may have different sampling frequencies and timestamps, these data can be asynchronously aligned to eliminate time differences. Schematically, data of different modalities can be aligned to the same time axis through a time synchronization algorithm to generate a time-synchronized fusion feature vector. This fusion feature vector can comprehensively reflect the child's language, expression, and movement status, providing a reliable data basis for subsequent pronunciation deviation correction and personalized training.

[0055] S102: Based on adversarial transfer learning, the fused feature vector is modeled for pronunciation deviation correction to generate standardized speech features.

[0056] Specifically, adversarial transfer learning is a deep learning method based on generative adversarial networks that can generate standardized speech features through adversarial training. Schematically, the speech features in the fused feature vector can be extracted first and input into the adversarial transfer learning model. The model consists of a generator and a discriminator. The generator is responsible for converting speech features with deviations into standardized speech features, while the discriminator is responsible for distinguishing the differences between the generated standardized speech features and the true standard speech features. Through multiple iterative training, the generator gradually learns how to convert deviated speech features into standardized speech features, thereby correcting pronunciation deviations. The standardized speech features can truly reflect children's standard language expression ability and provide a basis for further personalized training.

[0057] S103: Input the standardized speech features into a pre-trained model based on the federated startup optimization algorithm to generate a personalized training task.

[0058] Specifically, the federated startup optimization algorithm is a distributed optimization algorithm that can optimize model parameters through multi-device collaboration while protecting data privacy. The pre-training model can be a speech recognition model or a natural language processing model based on deep learning, which can then generate personalized training tasks suitable for the child’s current language ability based on standardized speech features. The task can include pronunciation practice, vocabulary learning, sentence construction, etc., to help children gradually improve their language expression ability. The design of this personalized training task fully takes into account the individual differences and special needs of different special children, such as the type of pronunciation disorder, language development level, and emotional state, so as to ensure the pertinence and effectiveness of the training task.

[0059] S104: Train preschool children with special needs through personalized training tasks to obtain real-time training data, and dynamically adjust the difficulty level of the personalized training tasks using reinforcement learning strategies based on the real-time training data to obtain updated personalized training tasks. The real-time training data includes voice interaction data, facial movement data, body movement data, and emotional physiological data.

[0060] Specifically, voice interaction data may include voice input and system feedback generated during the training process. Facial movement data may include facial expressions and lip shape changes during the training process. Body movement data may include body movement and posture change data during the training process. Emotional physiological data can be collected through wearable devices such as smart bracelets through physiological transmission to analyze the child’s emotional state. In addition, the reinforcement learning strategy can evaluate the child’s performance in training through a reward and punishment mechanism, and adjust the task difficulty based on the performance. For example, if a child performs well in a task, the task difficulty can be appropriately increased; if the child performs poorly, the task difficulty can be reduced or additional guidance can be provided. Through this dynamic adjustment mechanism, it can be ensured that the training task always adapts to the child’s current ability level, thereby improving the rehabilitation training effect.

[0061] The above-mentioned AI-based language rehabilitation training guidance method for preschool children with special needs effectively eliminates the temporal differences between different modal data through asynchronous alignment and feature fusion processing of multimodal data. Furthermore, cross-modal correlation analysis generates a time-synchronized fused feature vector, avoiding the information loss caused by data asynchrony and isolated analysis. This method can more comprehensively and accurately reflect the language and behavioral status of preschool children with special needs, providing a reliable data foundation for subsequent pronunciation deviation correction and personalized training. Secondly, based on adversarial transfer learning, the fused feature vector is modeled for pronunciation deviation correction to generate standardized speech features, eliminating pronunciation deviations caused by individual differences and language barriers, so that the standardized speech features can truly reflect the child's standard language expression ability. Furthermore, the standardized speech features are input into a pre-trained model based on the federated startup optimization algorithm to generate personalized training tasks, which are then used to train preschool children with special needs. This allows the content and difficulty of the training tasks to be adjusted according to the needs and abilities of each child, thereby improving the pertinence and effectiveness of rehabilitation training.

[0062] Finally, preschool special children are trained through personalized training tasks to obtain real-time training data. The difficulty level of the personalized training tasks is dynamically adjusted using reinforcement learning strategies based on the real-time training data to obtain updated personalized training tasks. This can continuously adapt to changes in children's language and behavior during rehabilitation training, adjust training strategies in a timely manner, avoid stagnation or excessive difficulty in the training process, maintain the continuity and challenge of training, and further improve the effectiveness of rehabilitation training.

[0063] In one embodiment, multimodal data of preschool children with special needs is obtained, and asynchronous alignment and feature fusion processing are performed on the multimodal data to generate a time-synchronized fused feature vector, including:

[0064] Perform noise reduction and phoneme segmentation on the original speech signal to extract MFCC features and fundamental frequency contour features;

[0065] Perform key point tracking on facial expression image sequences to extract facial visual features, including eye gaze direction features, mouth corner curvature features, and head posture angle features;

[0066] A multimodal attention alignment loss function is constructed to perform timestamp compensation on MFCC features, eye gaze direction features, and limb movement trajectories to obtain optimized MFCC features, optimized eye gaze direction features, and optimized limb movement trajectories.

[0067] The optimized MFCC features, optimized eye gaze direction features and optimized limb movement trajectories are input into the preset cross-modal cross-attention fusion network for fusion, and the fused feature vector is output.

[0068] Specifically, a denoising method based on wavelet transform can be used for the original speech signal. By performing wavelet decomposition on the original speech signal, the signal is decomposed into different frequency subbands. Based on the noise distribution characteristics in these subbands, threshold processing is performed on the noise subbands to remove the noise component. The denoised speech signal is then reconstructed using wavelet transforms. Subsequently, a phoneme segmentation algorithm based on a hidden Markov model can be employed. By training on a large amount of annotated speech data, the acoustic models of different phonemes are learned, enabling accurate identification of phoneme boundaries in the original speech signal. After phoneme segmentation, MFCC (Mel-Frequency Cepstral Coefficient) features and fundamental frequency contour features can be further extracted. MFCC features reflect the spectral envelope characteristics of the speech signal and are highly representative of speech timbre and articulation. The fundamental frequency contour features, on the other hand, reflect variations in the fundamental frequency of the speech signal and reflect information such as intonation and prosody. For example, the autocorrelation function method or a method based on harmonic peak detection can be used to accurately extract the fundamental frequency contour features of the speech signal.

[0069] Specifically, a convolutional neural network-based key point detection algorithm can be used to extract features from the input facial expression image sequence. Through multiple convolutional and pooling layers, high-level semantic features are gradually extracted from the image. Fully connected layers are then used to classify and locate these features, obtaining the coordinate information of facial key points. Based on this coordinate information, eye gaze direction features, mouth corner curvature features, and head posture angle features are extracted. Eye gaze direction features are determined by calculating the eye gaze vector and can reflect the direction of a child's attention during training. Mouth corner curvature features are obtained by measuring the degree of upturn or downturn of the mouth corners and are used to determine the child's emotional state, such as happiness or sadness. Head posture angle features are obtained by calculating the rotation angle of the head in three-dimensional space and help understand the child's body posture and participation during training.

[0070] Furthermore, to achieve precise temporal alignment of multimodal data, a multimodal attention alignment loss function can be constructed based on the attention mechanism to perform timestamp compensation on MFCC features, eye gaze direction features, and body movement trajectories. This attention mechanism automatically learns the importance weights between different modal data. Specifically, MFCC features, eye gaze direction features, and body movement trajectories are mapped to the same dimensional space using different linear transformations, and their similarities are then calculated. Attention weights are then calculated based on these similarities, with larger weights indicating greater importance of the modal data at the current moment. Using these attention weights, a weighted summation of the different modal data is performed to achieve timestamp compensation. Furthermore, to ensure that the compensated features better reflect the true characteristics of the data, a regularization term can be added to the loss function to prevent model overfitting. By continuously adjusting model parameters and minimizing the loss function, optimized MFCC features, optimized eye gaze direction features, and optimized body movement trajectories are ultimately obtained. Finally, this input is fused into a pre-set cross-modal cross-attention fusion network, which outputs a fused feature vector. This fusion network can adopt a multi-layered architecture, including multiple cross-attention mechanism layers. In each cross-attention mechanism layer, features of different modalities interact with each other, and cross-modal fusion of features is achieved by calculating the attention scores between modalities.

[0071] Schematically, the calculation formula of the multimodal attention alignment loss function is:

[0072]

[0073] Among them, L align is the modality attention alignment loss function, S(t i ) is the original speech signal, V(t i ) is a facial expression image sequence, α i is the dynamic weight matrix, JSD is the Jensen-Shannon divergence, which is used to constrain the consistency of the joint distribution of speech-expression p(S,V) and the action distribution p(M), β is a hyperparameter, is a regularization term to prevent the model from overfitting, and γ is used to control the influence of the regularization term.

[0074] In one embodiment, the fused feature vector is modeled for pronunciation deviation correction based on adversarial transfer learning to generate standardized speech features, including:

[0075] The pathological speech features are separated from the fused feature vector through a feature decoupling network, and a pre-trained standard speech library is obtained. The standard speech library contains standard pronunciation templates for children of different age groups.

[0076] Construct a generative adversarial network, and based on a pre-trained standard speech library, input pathological speech features into the generative adversarial network to output pseudo-standard speech features;

[0077] A dynamic mask matrix is generated based on the dialect distribution of preschool children with special needs in their area. Pseudo-standard speech features and pathological speech features are weighted and fused according to the dynamic mask matrix to generate dialect-corrected speech features. The weights of the dynamic mask matrix are dynamically adjusted through end-to-end training.

[0078] The dialect-corrected speech features were optimized with vocal tract formant constraints to obtain standardized speech features and pronunciation scores. The pronunciation scores were calculated by comparing the similarity between the dialect-corrected speech features and the children's standard pronunciation template.

[0079] Specifically, the feature decoupling network, based on the principles of deep learning, can deeply analyze each element in the fused feature vector and identify the characteristic portions related to pathological pronunciation. The pre-trained standard speech library includes standard pronunciation templates for children of different age groups. Moreover, these templates are based on a large amount of pronunciation data from normal children, rigorously screened and annotated, and are highly accurate and representative, providing a reliable reference standard for subsequent correction of pronunciation deviations. Secondly, the generative adversarial network is an adversarial neural network architecture consisting of a generator and a discriminator. In this embodiment, based on the pre-trained standard speech library, the separated pathological speech features can be input into the generator of the generative adversarial network. Based on these features, the generator can continuously optimize its generation capabilities by learning the speech patterns and feature distributions in the standard speech library, attempting to generate pseudo-standard speech features that are as close to standard speech as possible. The discriminator is responsible for determining the differences between the generated pseudo-standard speech features and the true standard speech features. Through this adversarial training method, the generator can ultimately output high-quality pseudo-standard speech features.

[0080] Furthermore, considering dialectal differences in the regions where preschool children with special needs live, a dynamic masking matrix can be generated based on the dialect distribution of that region. This dialect distribution data can be obtained through language surveys of a large number of local people, speech sample analysis, and other methods. This dynamic masking matrix allows for a weighted fusion of pseudo-standard speech features and pathological speech features to correct pronunciation deviations caused by dialect factors. Each element in the dynamic masking matrix represents the weight of the corresponding feature during the fusion process, and these weights are dynamically adjusted through end-to-end training. During training, the model automatically learns and adjusts the weights based on the current training data and objective function, ensuring that the fused speech features retain key information from the pathological speech features while maximally approximating standard speech features. This generates dialect-corrected speech features, effectively reducing the impact of dialect on pronunciation assessment and correction. Finally, vocal tract formants are important features of speech signals, reflecting the shape and acoustic properties of the vocal tract and closely related to speech timbre. Different pronunciations correspond to different formant patterns. Therefore, a standardized formant model can be established by analyzing a large number of standard pronunciation samples. When optimizing the dialect-corrected speech features, their formant patterns are compared and adjusted with the standard formant model. By adjusting parameters such as the frequency and amplitude of the speech signal, the formant patterns of the dialect-corrected speech features are made closer to the standard pattern, thereby obtaining standardized speech features. At the same time, the pronunciation score can be calculated by comparing the similarity between the dialect-corrected speech features and the children's standard pronunciation template. This similarity can be calculated based on the distance between speech feature sequences using the dynamic time warping algorithm, or by using cosine similarity to calculate the similarity between feature vectors. As a quantitative indicator, this pronunciation score can intuitively reflect the degree of closeness between the pronunciation of special children and the standard pronunciation, providing an important basis for subsequent training effect evaluation and training program adjustment.

[0081] In one embodiment, standardized speech features are input into a pre-trained model based on a federated startup optimization algorithm to generate personalized training tasks, including:

[0082] Obtain model parameters for a pre-trained model, which is trained based on a cross-institutional historical rehabilitation dataset that includes speech feature mappings for children with autism, children with language delays, and children with hearing impairments.

[0083] A local personalized model is constructed, and local parameters of the local personalized model are initialized as a weighted average of the model parameters, where the weight of the weighted average is dynamically allocated according to the type of impairment of the preschool special children;

[0084] The standardized speech features and pronunciation scores are combined into local training data, and the local parameters are updated using the differentially private federated distillation algorithm to obtain the updated local parameters.

[0085] Generate a task parameter set based on the updated local parameters, which includes vocabulary threshold, speech rate range, and grammatical complexity level;

[0086] The initial task difficulty in the task parameter set is dynamically adjusted based on the pronunciation score, and personalized training tasks are generated through an adaptive task allocation strategy combined with the real-time attention data of preschool special children.

[0087] Specifically, the model parameters of the pre-training model are obtained. The pre-training model is trained based on a historical rehabilitation data set across institutions. And the historical rehabilitation data set contains the speech feature mapping relationship of different types of special children such as autistic children, children with delayed language development and hearing-impaired children. The rehabilitation training data of special children from multiple rehabilitation institutions can be cleaned, labeled and other pre-processing operations, and then the pre-training model can be trained so that it can learn the general knowledge and patterns related to the speech features and rehabilitation training of different types of special children. Therefore, its model parameters contain information that is instructive for the language rehabilitation of various types of special children. Secondly, a local personalized model is constructed. And in order to make the model better adaptable to preschool special children who currently need rehabilitation training, its local parameters can be initialized to the weighted average of the pre-training model parameters. Among them, the weight of the weighted average is not fixed, but is dynamically allocated according to the type of obstacle of the preschool special child. For example, for children with autism, due to their unique obstacles in social communication and language use, when allocating weights, more emphasis will be placed on the model parameters related to social language, so that the local personalized model can be more in line with the specific obstacle characteristics of the current special children during the initialization phase. The standardized speech features and pronunciation scores can then be fused into local training data. The standardized speech features are feature representations close to the standard pronunciation obtained by processing the original speech of special children, and the pronunciation scores intuitively reflect the similarity between the standardized speech features and the standard pronunciation. Fusion of the two can provide more comprehensive training information for the local personalized model.

[0088] Specifically, the differential privacy federated distillation algorithm combines differential privacy technology with the model distillation concept of federated learning. Differential privacy technology allows the model to use a certain amount of raw data during training while protecting data privacy, thus preventing the risk of sensitive data leakage during training. The federated distillation algorithm transfers knowledge from a pre-trained model to a local personalized model. Without direct cross-institutional data exchange, the local personalized model can leverage the generalization capabilities of the pre-trained model to update its parameters. Therefore, during the update process, the local personalized model can optimize and adjust local parameters based on local training data, incorporating differential privacy protection mechanisms. This results in updated local parameters, making the model more relevant to the specific needs of the child. A task parameter set can then be generated based on these updated local parameters. These updated local parameters contain personalized information specific to the child, and are used to determine the appropriate training task parameters for that child. Illustratively, the task parameter set may include key parameters such as vocabulary threshold, speech rate range, and grammatical complexity level.

[0089] Finally, the initial task difficulty within the task parameter set can be dynamically adjusted based on the pronunciation score. Combined with the real-time attention data of preschool children with special needs, personalized training tasks can be generated through an adaptive task allocation strategy. For example, the pronunciation score reflects the child's current pronunciation proficiency. A high score indicates a good grasp of pronunciation, and task difficulty can be appropriately increased, such as by increasing the vocabulary threshold, raising the upper limit of the speech rate range, or increasing the level of grammatical complexity. Conversely, task difficulty can be reduced. This real-time attention data can be obtained through various means, such as using eye tracking technology to monitor the direction and duration of a child's gaze, or using sensors to monitor the child's body movements. Incorporating this real-time attention data, the adaptive task allocation strategy intelligently adjusts the presentation and order of training tasks. For example, when a child's attention is focused, tasks with slightly higher difficulty and requiring more thought can be assigned. When attention is declining, tasks with greater interest and interactivity can be switched to simpler tasks to maintain the child's engagement and motivation. This comprehensive approach, which considers pronunciation scores and real-time attention data, generates highly personalized training tasks that better meet the individual needs of preschool children with special needs during speech rehabilitation training.

[0090] In one embodiment, preschool children with special needs are trained through personalized training tasks, real-time training data is obtained, and a reinforcement learning strategy is used based on the real-time training data to dynamically adjust the current task difficulty level of the personalized training task to obtain an updated personalized training task, including:

[0091] The speech interaction data is processed through the speech recognition model to calculate the pronunciation accuracy;

[0092] Key frame analysis is performed on facial action data to calculate the degree of attention concentration;

[0093] Calculate the emotional fluctuation index based on body movement data and emotional physiological data;

[0094] Construct reinforcement learning state vectors through pronunciation accuracy, attention concentration and emotion fluctuation index;

[0095] Based on the reinforcement learning state vector, the Q-learning algorithm is used to perform strategy update processing to obtain a difficulty level adjustment action. The difficulty level adjustment action is any one of increasing the difficulty level, maintaining the current difficulty level, and decreasing the difficulty level.

[0096] Through the difficulty level adjustment action, the vocabulary threshold and speech speed range of the personalized training task are dynamically adjusted to generate an updated personalized training task.

[0097] Specifically, during the training of preschool children with special needs through personalized training tasks, voice interactions are conducted with the children, generating real-time voice interaction data. This data can then be processed using a speech recognition model, such as one based on a recurrent neural network and a long short-term memory network. This model compares the child's voice data with a standard pronunciation library, analyzing speech elements such as phonemes and syllables in detail to calculate pronunciation accuracy. Furthermore, during training, a camera can continuously capture the child's facial movement data. Keyframe analysis technology, based on computer vision principles, selects representative keyframes from this continuous facial movement data. Within these keyframes, a facial landmark detection algorithm can be applied to precisely locate key facial features, such as the eyes, eyebrows, and mouth. By analyzing changes in the position and shape of these landmarks between keyframes, attentional focus can be calculated. For example, by monitoring eye gaze direction, blink frequency, and eyebrow movement, it is possible to determine whether a child is focusing on the training content. If the eyes are fixed on a specific area in the training interface for a long time, the blinking frequency is within the normal range, and the eyebrows remain natural, it can be considered that the concentration is high.

[0098] Specifically, body movement data can be acquired using inertial sensors worn on children's limbs or video-based body movement tracking technology. Emotional physiological data can be acquired through wearable devices, such as smart bracelets, which collect physiological signals such as a child's heart rate, galvanic skin response, and respiratory rate. Body movement data can be analyzed for characteristics such as amplitude, frequency, and smoothness. For example, large, frequent, and irregular body movements may indicate a child is emotionally agitated or restless, while steady, slow, and regular movements may indicate a calmer mood. Regarding emotional physiological data, a sudden increase in heart rate, an increase in galvanic skin response, and an increase in respiratory rate are often associated with emotional fluctuations or tension. By applying specific mathematical models and algorithms to body movement and emotional physiological data, an emotional volatility index can be calculated. This index provides a direct reflection of a child's emotional stability during training; higher values indicate greater emotional fluctuations and increased emotional instability.

[0099] Subsequently, a reinforcement learning state vector can be constructed using pronunciation accuracy, attention concentration, and mood swing index. Reinforcement learning is a machine learning method that optimizes decision-making based on environmental feedback. In this embodiment, the three key indicators mentioned above can be integrated into a state vector as input to the reinforcement learning algorithm. This state vector can comprehensively describe the child's state information during training, providing a rich data foundation for subsequent strategy updates. Furthermore, based on the constructed reinforcement learning state vector, a Q-learning algorithm can be used to update the strategy. The Q-learning algorithm is a classic reinforcement learning algorithm that selects the optimal action by learning a state-action value function. The action space can be defined as increasing the difficulty level, maintaining the current difficulty level, and decreasing the difficulty level. Based on the current reinforcement learning state vector, the algorithm searches the Q-table (a table that stores state-action values) for the corresponding Q value and selects an action based on the magnitude of the Q value. Furthermore, during training, the algorithm can update the Q-table based on actual reward feedback, continuously optimizing the strategy and gradually learning the optimal difficulty-adjusting action for different child training states.

[0100] Finally, the vocabulary threshold and speech speed interval of the personalized training task can be dynamically adjusted based on the difficulty level adjustment action obtained, thereby generating an updated personalized training task. If the difficulty level adjustment action is to increase the difficulty level, the vocabulary threshold can be increased accordingly, and the speech speed interval can be increased. If the action is to reduce the difficulty level, the vocabulary threshold can be reduced and the speech speed interval can be reduced. Through this dynamic adjustment process, the personalized training task can always match the child's real-time training status, better promote the child's language rehabilitation training process, and improve the training effect.

[0101] In one embodiment, the method further comprises:

[0102] When the pronunciation accuracy is greater than a first threshold and the attention concentration is greater than a second threshold, generating encouraging voice instructions;

[0103] When the emotional stability index is less than a third threshold, a soothing voice command is generated;

[0104] If the emotional stability index is lower than the safety threshold, an emergency intervention instruction is generated, which is used to stop the personalized training task.

[0105] Specifically, during training, the child's pronunciation accuracy and attention span can be continuously monitored. When the pronunciation accuracy exceeds a first threshold, it indicates that the child has achieved a certain level of pronunciation accuracy. This first threshold is not set arbitrarily; it is determined based on a large amount of pronunciation data for normal children of the corresponding age group, combined with the expected progress of rehabilitation training for children with special needs. Furthermore, when the attention span exceeds a second threshold, it indicates that the child is able to maintain a high level of concentration during training. The second threshold is also set based on research on the attention span of normal children in similar learning scenarios and the distribution of attention span duration during training for children with special needs. If both the pronunciation accuracy and attention span meet the above conditions, an encouraging voice command can be generated. This encouraging voice command can not only include direct affirmation of the child's accurate pronunciation, such as "You just said the word with very standard pronunciation, great!", but can also further strengthen the child's learning motivation, such as "Keep up this performance, you will get better and better, and learn more words and sentences," thereby enhancing the child's self-confidence and learning enthusiasm.

[0106] Specifically, when the emotional stability index is less than the third threshold, it means that the child may have emotional fluctuations or instability. The third threshold can be set with reference to the normal data of emotional fluctuations of special children during rehabilitation training. If the emotional stability index is lower than the third threshold, a soothing voice instruction can be generated. The soothing voice instruction can adopt a gentle and soothing tone, aiming to alleviate the child’s negative emotions, such as "Baby, don't worry, let's talk slowly, there's no rush" or "You seem a little tired, let's take a break, adjust our state and continue, okay?" And it can be combined with some visual elements, such as displaying warm and calm pictures on the training interface, to further help children stabilize their emotions and ensure that the training can continue.

[0107] If the emotional stability index is lower than the safety threshold, it indicates that the child's emotional state has reached a more serious level of instability, which may have adverse effects on their physical and mental health. This safety threshold is determined based on research on the behavioral and physiological reactions that special children may have under extreme emotions. If the emotional stability index exceeds the safety threshold, an emergency intervention instruction can be generated immediately. This emergency intervention instruction is intended to stop the personalized training task to avoid aggravating the child's negative emotions due to continued training. In addition, a series of measures can be taken to ensure the safety and emotional recovery of the child. For example, children can be guided by voice to perform simple relaxation exercises such as deep breathing, and an alert message can be sent to the parent or rehabilitation trainer responsible for the child's training to inform the child of the current emotional emergency so that the child can be given more direct and effective comfort and help in a timely manner to ensure the safety and health of the child during the training process.

[0108] Based on the same inventive concept, Figure 2 As shown, the embodiment of the present application also provides an AI-based preschool special children language rehabilitation training guidance system 200. The implementation solution provided by the system is similar to the implementation solution described in the above method. Therefore, the specific limitations of one or more embodiments of the AI-based preschool special children language rehabilitation training guidance system provided below can be found in the above-mentioned limitations of the AI-based preschool special children language rehabilitation training guidance method, and will not be repeated here. The system includes:

[0109] Data acquisition and processing module 201 is used to acquire multimodal data of preschool children with special needs, perform asynchronous alignment and feature fusion processing on the multimodal data, and generate time-synchronized fused feature vectors. The multimodal data includes original speech signals, facial expression image sequences, and body movement trajectories.

[0110] The original speech correction module 202 is used to perform pronunciation deviation correction modeling on the fused feature vector based on adversarial transfer learning to generate standardized speech features;

[0111] A personalized task generation module 203 is configured to input standardized speech features into a pre-trained model based on a federated start optimization algorithm to generate a personalized training task;

[0112] The real-time task adjustment module 204 is used to train preschool children with special needs through personalized training tasks, obtain real-time training data, and dynamically adjust the difficulty level of the personalized training tasks using a reinforcement learning strategy based on the real-time training data to obtain updated personalized training tasks. The real-time training data includes voice interaction data, facial movement data, body movement data, and emotional physiological data.

[0113] In the above-mentioned AI-based language rehabilitation training guidance system 200 for preschool special children, the data acquisition and processing module 201 can acquire multimodal data of preschool special children and perform asynchronous alignment and feature fusion processing on it, which can effectively eliminate the time differences between various types of data and ensure the time synchronization of the data, thereby providing a comprehensive data foundation for subsequent pronunciation correction and training task generation. The original speech correction module 202 performs pronunciation deviation correction modeling on the fused feature vector based on adversarial transfer learning, which can effectively distinguish the pronunciation problems of special children and generate standardized speech features, avoiding the limitations of traditional pronunciation correction methods that cannot accurately locate and solve pronunciation problems. The personalized task generation module 203 inputs the standardized speech features into a pre-training model based on the federal startup optimization algorithm, which can adjust the content and difficulty of the training tasks according to the needs and abilities of each child and generate personalized training tasks. The real-time task adjustment module 204 can train children through personalized training tasks, obtain real-time training data including voice interaction data, facial movement data, body movement data and emotional physiological data, and dynamically adjust the difficulty level of personalized training tasks based on reinforcement learning strategies. It can adjust the training tasks in real time according to the children's learning status and emotional changes, avoiding children from feeling frustrated or losing interest due to inappropriate training difficulty, thereby enhancing the adaptability and effectiveness of training.

[0114] Through the collaborative work of the above-mentioned multiple modules, the system not only comprehensively improves the pertinence, accuracy and adaptability of language rehabilitation training for preschool children with special needs, but also can carry out efficient rehabilitation training based on the characteristics and real-time status of each child, significantly improving the quality and effectiveness of rehabilitation training, and providing strong support for language rehabilitation training for preschool children with special needs.

[0115] In an exemplary embodiment, the present invention further provides a computer device comprising a memory and a processor. The memory stores a computer program, and when the processor executes the computer program, it implements the steps of the AI-based preschool children's language rehabilitation training guidance method for special needs children. A multi-core processor is preferred to improve the system's parallel processing capabilities. Memory: Provides sufficient temporary storage space to support program execution and data processing. The memory capacity should be large enough to accommodate large amounts of supply information and computing tasks.

[0116] In an exemplary embodiment, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the AI-based preschool special children language rehabilitation training guidance method of the present application. The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a solid-state drive (SSD), or an optical disc. Among them, the random access memory may include a resistance random access memory (ReRAM) and a dynamic random access memory (DRAM).

[0117] The above-described embodiments merely represent several implementation methods of the embodiments of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person skilled in the art may make various modifications and improvements without departing from the concept of the embodiments of the present application, and these modifications and improvements fall within the scope of protection of the embodiments of the present application.

Claims

1. An AI-based language rehabilitation training guidance method for preschool children with special needs, characterized by: The method comprises: Acquiring multimodal data of preschool children with special needs, performing asynchronous alignment and feature fusion processing on the multimodal data to generate a time-synchronized fused feature vector, the multimodal data including original speech signals, facial expression image sequences, and body movement trajectories; Performing pronunciation deviation correction modeling on the fused feature vector based on adversarial transfer learning to generate standardized speech features; Inputting the standardized speech features into a pre-trained model based on a federated startup optimization algorithm to generate a personalized training task; The preschool special children are trained through the personalized training tasks to obtain real-time training data, and the difficulty level of the personalized training tasks is dynamically adjusted based on the real-time training data using a reinforcement learning strategy to obtain an updated personalized training task. The real-time training data includes voice interaction data, facial movement data, body movement data, and emotional physiological data.

2. The method according to claim 1, characterized in that The method of acquiring multimodal data of preschool children with special needs, performing asynchronous alignment and feature fusion processing on the multimodal data, and generating a time-synchronized fusion feature vector includes: Performing noise reduction and phoneme segmentation processing on the original speech signal to extract MFCC features and fundamental frequency contour features; Performing key point tracking processing on the facial expression image sequence to extract facial visual features, wherein the facial visual features include eye gaze direction features, mouth corner curvature features, and head posture angle features; Constructing a multimodal attention alignment loss function, performing timestamp compensation processing on the MFCC features, the eye gaze direction features, and the limb movement trajectory to obtain optimized MFCC features, optimized eye gaze direction features, and optimized limb movement trajectory; The optimized MFCC features, the optimized eye gaze direction features, and the optimized limb movement trajectory are input into a preset cross-modal cross-attention fusion network for fusion, and the fused feature vector is output.

3. The method according to claim 2, characterized in that The calculation formula of the multimodal attention alignment loss function is: Among them, L align is the modality attention alignment loss function, S(t i ) is the original speech signal, V(t i ) is the facial expression image sequence, α i is the dynamic weight matrix, JSD is the Jensen-Shannon divergence, which is used to constrain the consistency of the joint distribution of speech-expression p(S,V) and the action distribution p(M), β is a hyperparameter, is a regularization term to prevent the model from overfitting, and γ is used to control the influence of the regularization term.

4. The method according to claim 1, wherein The method of performing pronunciation deviation correction modeling on the fused feature vector based on adversarial transfer learning to generate standardized speech features includes: Separating pathological speech features from the fused feature vector through a feature decoupling network, and obtaining a pre-trained standard speech library, wherein the standard speech library contains standard pronunciation templates for children of different age groups; Constructing a generative adversarial network, and based on a pre-trained standard speech library, inputting the pathological speech features into the generative adversarial network to output pseudo-standard speech features; Generating a dynamic mask matrix based on the dialect distribution of the area where the preschool special children are located, and performing weighted fusion of the pseudo-standard speech features and the pathological speech features according to the dynamic mask matrix to generate dialect-corrected speech features, wherein the weights of the dynamic mask matrix are dynamically adjusted through end-to-end training; The dialect-corrected speech features are subjected to vocal tract formant constraint optimization to obtain the standardized speech features and pronunciation scores, wherein the pronunciation scores are calculated by comparing the similarity between the dialect-corrected speech features and the children's standard pronunciation template.

5. The method according to claim 4, characterized in that The step of inputting the standardized speech features into a pre-trained model based on a federated startup optimization algorithm to generate a personalized training task includes: Obtaining model parameters of the pre-trained model, where the pre-trained model is trained based on a cross-institutional historical rehabilitation dataset, where the historical rehabilitation dataset includes speech feature mapping relationships for children with autism, children with language delays, and children with hearing impairments; Constructing a local personalized model, initializing local parameters of the local personalized model to a weighted average of the model parameters, wherein the weight of the weighted average is dynamically assigned according to the type of disability of the preschool special child; fusing the standardized speech features and the pronunciation scores into local training data, and updating the local parameters using a differentially private federated distillation algorithm to obtain updated local parameters; generating a task parameter set according to the updated local parameters, wherein the task parameter set includes a vocabulary threshold, a speech rate range, and a grammatical complexity level; The initial task difficulty in the task parameter set is dynamically adjusted based on the pronunciation score, and the personalized training task is generated through an adaptive task allocation strategy in combination with the real-time attention data of the preschool special children.

6. The method according to claim 1, characterized in that The training of the preschool special needs child through the personalized training task, obtaining real-time training data, and dynamically adjusting the current task difficulty level of the personalized training task using a reinforcement learning strategy based on the real-time training data to obtain an updated personalized training task, includes: Processing the voice interaction data through a voice recognition model to calculate the pronunciation accuracy; Performing key frame analysis on the facial action data to calculate the degree of attention concentration; Calculating an emotion fluctuation index based on the body movement data and the emotion physiological data; Constructing a reinforcement learning state vector based on the pronunciation accuracy, the attention concentration, and the emotion fluctuation index; performing a strategy update process using a Q-learning algorithm based on the reinforcement learning state vector to obtain a difficulty level adjustment action, wherein the difficulty level adjustment action is any one of increasing the difficulty level, maintaining the current difficulty level, and decreasing the difficulty level; Through the difficulty level adjustment action, the vocabulary threshold and speech speed range of the personalized training task are dynamically adjusted to generate the updated personalized training task.

7. The method according to claim 6, characterized in that The method further comprises: When the pronunciation accuracy is greater than a first threshold and the attention concentration is greater than a second threshold, generating an encouraging voice instruction; When the emotional stability index is less than a third threshold, generating a soothing voice instruction; If the emotional stability index is less than a safety threshold, an emergency intervention instruction is generated, and the emergency intervention instruction is used to stop the personalized training task.

8. An AI-based language rehabilitation training guidance system for preschool children with special needs, characterized by: The system comprises: A data acquisition and processing module is used to acquire multimodal data of preschool children with special needs, perform asynchronous alignment and feature fusion processing on the multimodal data, and generate time-synchronized fused feature vectors. The multimodal data includes original speech signals, facial expression image sequences, and body movement trajectories. An original speech correction module is used to perform pronunciation deviation correction modeling on the fused feature vector based on adversarial transfer learning to generate standardized speech features; A personalized task generation module, configured to input the standardized speech features into a pre-training model based on a federated startup optimization algorithm to generate a personalized training task; A real-time task adjustment module is used to train the preschool special children through the personalized training tasks, obtain real-time training data, and dynamically adjust the difficulty level of the personalized training tasks using a reinforcement learning strategy based on the real-time training data to obtain updated personalized training tasks. The real-time training data includes voice interaction data, facial movement data, body movement data, and emotional physiological data.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images