A control method and system of a robot for autism children diagnosis and treatment
By analyzing children's states through multimodal interaction and deep learning models, and combining LSTM and CPT models to optimize decisions, personalized intervention strategies are generated. This solves the problem that existing robots cannot understand children's states in real time, and realizes efficient and personalized assistance in the diagnosis and treatment of children with autism.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAMEN UNIV
- Filing Date
- 2026-03-09
- Publication Date
- 2026-06-05
AI Technical Summary
Existing diagnostic robots for autism cannot perceive and understand changes in a child's state in real time, resulting in stiff human-computer interaction, which can easily lead to boredom or avoidance behavior. Furthermore, their ability to assist in diagnosis and treatment is low, and they cannot automatically adjust teaching strategies based on a specific child's historical data.
The system employs multimodal interactive content and deep learning models to analyze children's states in real time. It uses LSTM state prediction models and CPT action decision models to generate a list of recommended intervention actions, and optimizes decisions through HTN hierarchical task networks and compact prediction trees. Combined with therapists' historical operating habits, it generates personalized intervention strategies. The system has self-learning and adaptive capabilities.
It reduced children's boredom or avoidance behaviors, improved auxiliary diagnostic and treatment capabilities, enabled personalized interaction with children, reduced the workload of therapists, and improved the accuracy and flexibility of interventions.
Smart Images
Figure CN122158050A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a medical engineering technology, and in particular to a control method and system for a diagnostic and treatment robot for children with autism. Background Technology
[0002] Autism spectrum disorder (ASD), also known as autism spectrum disorder or autism spectrum disorder, is a group of neurodevelopmental disorders that occur in early childhood. Children with this disorder are usually referred to as autistic children.
[0003] Traditional diagnosis and treatment of children with autism relies entirely on therapists, resulting in a relatively heavy workload for them. To alleviate this workload, many robots have emerged on the market to assist therapists in diagnosis and treatment. However, most of these robots can only function as terminals that simply execute instructions and cannot perceive or understand the changes in a child's state during intervention in real time. This leads to a stiff and rigid human-computer interaction process, which can easily cause children to become bored or avoid the interaction. In addition, existing diagnostic and treatment robots rely too heavily on pre-programming and cannot accumulate "experience" like human teachers. They cannot automatically adjust teaching strategies based on a specific child's historical training data and require manual adjustments by therapists, resulting in relatively low diagnostic and treatment capabilities.
[0004] In view of this, the inventor has conducted in-depth research on the above-mentioned problems, which led to the present invention. Summary of the Invention
[0005] The purpose of this invention is to provide a control method and system for a diagnostic robot for autistic children that is less likely to cause children to become bored or avoidant and has relatively high auxiliary diagnostic and treatment capabilities.
[0006] To achieve the above objectives, the present invention provides the following technical solution: A method for controlling a diagnostic robot for children with autism includes the following steps: S1, the robot interacts with children by presenting multimodal interactive content. During the interaction, it records the child's facial video in real time and records the therapist's score on the child's performance. S2, using a deep learning model to analyze the face video, obtain the child's attention level and the valence and arousal of facial expressions, and generate the child's state code based on the score, attention level, and valence and arousal through weighted calculation; S3, Based on the multimodal interactive content and the child's state encoding, generate a recommended list of next intervention actions using the LSTM state prediction model and the CPT action decision model; S4, the therapist selects or modifies one of the recommended items from the list of recommended next intervention actions and executes it, forming new interactive data; S5, using the new interactive data, retrain and update the LSTM state prediction model and the CPT action decision model.
[0007] As an improvement of the present invention, the multimodal interactive content includes video, audio, graphics and / or text.
[0008] As an improvement of the present invention, the deep learning model is the BlazeFace and EmoFAN deep learning model. In step S2, the BlazeFace and EmoFAN deep learning model is used to extract facial features from the face video, the child's gaze region is calculated based on the facial features, and the child's focus is estimated based on the gaze region.
[0009] As an improvement of the present invention, in step S3, the LSTM state prediction model is first used to predict the child's possible emotions and concentration in the next round, and the CPT action decision model is used to predict the experience-recommended actions based on the therapist's historical operating habits. Then, the HTN hierarchical task network is used to plan a logically consistent set of candidate actions using DTT domain knowledge. Finally, a hierarchical strategy is used to merge and sort the candidate action set with the experience-recommended actions to generate a list of recommended intervention actions for the next step.
[0010] A control system for a diagnostic and treatment robot for children with autism includes: The execution module is used to receive control commands, interact with the child and therapist according to the control commands, record the child's facial video in real time, and record the therapist's score on the child's performance; A perception module is used to receive the facial video and the score, and to perform quantitative analysis on the child's real-time status based on the facial video and the score; The memory module stores short-term interaction data for the current lesson, each child's historical information set, skill level table, and the trained optimal computational model. It also provides the child's historical state sequence, the current optimal LSTM state prediction model and CPT action decision model, as well as a domain knowledge base containing intervention rules. It also provides historically accumulated sequence data for model updates. An evolution module is used to extract accumulated sequence data from the memory module according to a pre-set frequency or triggering conditions, retrain the LSTM state prediction model and the CPT action decision model, compare the performance of the old and new models on the validation set, automatically select the optimal computational model adapted to the current characteristics of the child, and update it to the memory module; and The decision-making module is used to read the child's historical state sequence, the current optimal LSTM state prediction model, and the CPT action decision model provided by the memory module, and generate a recommendation for the next intervention action based on the read information.
[0011] As an improvement of the present invention, the specific method by which the perception module performs quantitative analysis of the child's real-time state is as follows: extracting facial features from the face video using BlazeFace and EmoFAN deep learning models, calculating the child's gaze area based on the facial features, estimating the child's focus based on the gaze area, identifying the valence and arousal of facial expressions, and generating a child's state code based on the score, the focus, and the valence and arousal through weighted calculation.
[0012] As an improvement of the present invention, the specific method for the decision module to generate the next intervention action recommendation is as follows: First, the LSTM state prediction model is used to predict the child's possible emotions and concentration in the next round. At the same time, the CPT action decision model is used to predict the experience-recommended actions based on the therapist's historical operating habits. Then, the HTN hierarchical task network is used to plan a logically consistent set of candidate actions using DTT domain knowledge. Finally, a hierarchical strategy is used to merge and sort the candidate action set with the experience-recommended actions to generate a list of next intervention action recommendations for the therapist to choose from.
[0013] By adopting the above technical solution, the present invention has the following beneficial effects: 1. The method and system of this invention introduce the quantitative calculation of the valence and arousal of children's attention and facial expressions, and use LSTM for future state prediction. It can adjust strategies in advance when it is predicted that children are about to lose interest, which is less likely to cause children to become bored or avoidant. The LSTM state prediction model and the CPT action decision model are retrained and updated using the new interaction data. It has self-learning and adaptive capabilities and relatively high auxiliary diagnosis and treatment capabilities.
[0014] 2. The system of this invention possesses "interactive cognitive computing capabilities," achieving intelligent assisted decision-making through human-machine collaboration by simulating the "perception-decision-execution-reflection" process of a human therapist; reducing the repetitive operational burden on therapists in turn-based teaching (DTT), with the robot automatically recommending the next teaching content; improving the accuracy of intervention by capturing children's attention and emotions in real time through multimodal perception and dynamically adjusting the teaching pace; and enabling personalized intervention by allowing the robot to understand the child better and better over time through a model evolution mechanism.
[0015] 3. This invention adopts a hierarchical strategy of "HTN rule constraints + CPT experience recommendations", which takes into account both medical standardization and personalized flexibility. Attached Figure Description
[0016] Figure 1 This is a schematic diagram of the framework structure of the autism diagnosis and treatment robot control system of the present invention. Detailed Implementation
[0017] The present invention will be further described below with reference to the accompanying drawings and specific embodiments.
[0018] This embodiment provides a control method for a diagnostic robot for children with autism. The diagnostic robot is a conventional robot with information input devices such as a camera, microphone, and keyboard. The hardware structure of the robot is not the focus of this embodiment and will not be described in detail here.
[0019] The control method for this autism diagnosis and treatment robot includes the following steps: S1. The robot interacts with children by presenting them with multimodal interactive content, including video, audio, graphics, and / or text. During the interaction, the robot collects real-time data on the child's behavior through a camera and microphone, specifically recording the child's facial video and voice data in real time. The voice data is used to enable interaction with the child. Simultaneously, the therapist observes the child's reactions, scores and records the child's performance.
[0020] S2, the facial video obtained in the previous step is analyzed using a deep learning model to obtain the child's attention level and the valence and arousal of facial expressions. Specifically, in this embodiment, the learning model uses BlazeFace and EmoFAN deep learning models to extract facial features from the facial video, calculate the child's gaze region based on the facial features, estimate the child's attention level based on the child's gaze region, and simultaneously identify the valence and arousal of facial expressions. Valence and arousal are used as emotional features of the child to determine the child's emotions. The specific methods for estimating attention level and identifying valence and arousal are conventional and will not be detailed here.
[0021] Next, combined with the therapist's rating, the child's state is encoded into a multidimensional feature vector. Specifically, based on the rating, attention, valence and arousal, attention, emotional characteristics and the child's performance rating entered by the therapist are combined and weighted to generate the child's state code. The specific weights of each part in the weighted calculation can be obtained through simple estimation, but it is best to obtain them based on accurate calculations of statistical data. The specific statistical calculation method is conventional and will not be detailed here.
[0022] S3, based on multimodal interactive content and children's state encoding, uses an LSTM state prediction model (LSTM is a long short-term memory network, a type of recurrent neural network) and a CPT action decision model (CPT is a compact prediction tree) to generate a list of recommended intervention actions for the next step. The multimodal interactive content and children's state encoding can include only the data of the current round, or they can include data from the current round, data from multiple past rounds, and historical data of similar items. In this way, the LSTM state prediction model can be used to predict the child's possible emotions and attention in the next round. If it is predicted that the child is about to lose interest, the robot can adjust its strategy in advance.
[0023] Specifically, in this step, the LSTM state prediction model is first used to predict the child's possible emotions (based on valence and arousal) and attention in the next round. Simultaneously, the CPT action decision model is used to predict experience-recommended actions based on the therapist's historical operational habits (this can be omitted when no historical operation exists for the first use, i.e., the experience-recommended action is null). Then, the HTN hierarchical task network is used, leveraging DTT domain knowledge (i.e., autism intervention domain knowledge) to plan a logically consistent set of candidate actions. Finally, a hierarchical strategy is used to fuse and rank the candidate action set with the experience-recommended actions, generating a list of recommended intervention actions for the next step. In other words, in this embodiment, the robot's next action is not random, but generated through a two-layer mechanism: the first layer is the rule layer, which uses the HTN hierarchical task network, combined with DTT domain knowledge, to plan a set of candidate actions that conform to teaching logic, ensuring the safety and compliance of the decision; the second layer is the experience layer, which uses a compact prediction tree (CPT) to learn the therapist's historical operational habits for the child. Finally, the system calculates the recommendation probability of each candidate action and ranks the candidate set, thus achieving personalized and human-like decision-making.
[0024] In step S4, the system pushes the sorted recommendation list to the robot's display or the therapist's handheld terminal. The therapist selects or modifies one of the recommended items from the next intervention action recommendation list and executes it, generating new interactive data. This "recommendation-confirmation" model retains the human's ultimate control while greatly reducing the number of operational steps.
[0025] S5: After each consultation, the LSTM state prediction model and CPT action decision model are retrained and updated using new interaction data, enabling the robot to continuously adapt to the changes in the child.
[0026] This embodiment also provides a control system for autism diagnosis and treatment robots that can implement the above-described methods, such as... Figure 1As shown, the control system includes five modules: execution module, perception module, memory module, evolution module, and decision module. These five modules together constitute the cognitive computing framework of the control system. The functions and principles of each module will be explained in detail below.
[0027] The execution module receives control commands, including those from the therapist and those confirmed by the decision module. Based on these commands, it interacts with the child and therapist, recording the child's facial video and voice data in real time, and recording the therapist's scores of the child's performance. The recorded voice data primarily serves to comprehensively document the original interactions during the treatment process, allowing for subsequent manual review by the therapist and use in future model training to facilitate future functional expansion. As part of the execution process recording, it does not participate in the quantitative analysis of the child's real-time state. The execution module, based on control commands, drives the robot to access a local media library to present multimodal interactive content such as video, audio, and facial expressions to the child, enabling interaction. Interaction with the therapist mainly involves displaying a list of recommended next intervention actions and receiving information from the therapist's selections or modifications, while simultaneously recording the therapist's scores of the child's skill performance. The execution module communicates with both the perception and decision modules, and its recorded data, along with the execution process log, is transmitted to the perception module for processing.
[0028] The perception module receives facial video, voice data, scores, and execution process records from the execution module, and performs quantitative analysis of the child's real-time state based on the facial video and scores. Specifically, the perception module extracts facial features from the facial video using BlazeFace and EmoFAN deep learning models, calculates the child's gaze area based on these features, estimates the child's focus level based on the gaze area, identifies the valence and arousal of facial expressions, and generates a child state code based on a weighted calculation using the scores, focus level, and valence and arousal. The perception module communicates with the memory module, transmitting the received execution process records and the generated child state codes to the memory module for processing.
[0029] The memory module, serving as the system's database and knowledge base, is responsible for storing and providing data support. Specifically, it stores short-term interaction data for the current lesson, each child's historical information set, skill level table, and the trained optimal computational model, namely the LSTM state prediction model and the CPT action decision model. The memory module comprises two parts: short-term memory and long-term memory. The short-term memory primarily stores short-term interaction data for the current lesson, child state sequences (obtained from child state codes), and robot action sequences (obtained from execution process records). The long-term memory primarily stores each child's historical information set (i.e., raw data), skill level table, and the trained optimal computational model. It can provide the corresponding modules with the child's historical state sequences, the current optimal LSTM state prediction model and CPT action decision model, and a domain knowledge base containing intervention rules. It also provides historically accumulated sequence data for model updates. The memory module communicates with both the evolution module and the decision module to provide data to these modules.
[0030] The evolution module gives the robot the ability to learn and adapt on its own. It is used to extract the accumulated sequence data from the memory module according to a pre-set frequency or triggering conditions (such as when predictions fail continuously), retrain the LSTM state prediction model and the CPT action decision model, compare the performance of the new and old models on the validation set, automatically select the optimal computational model that is adapted to the current characteristics of children, and update it to the memory module.
[0031] The decision-making module reads the child's historical state sequence, the current optimal LSTM state prediction model, and the CPT action decision model provided by the memory module, and generates a recommendation for the next intervention action based on the information obtained. The specific method for generating the next intervention action recommendation is as follows: First, the LSTM state prediction model is used to predict the child's possible emotions and concentration in the next round. Simultaneously, the CPT action decision model is used to predict experience-based recommended actions based on the therapist's historical operating habits. Then, an HTN hierarchical task network is used, leveraging DTT domain knowledge to plan a logically consistent set of candidate actions. Finally, a hierarchical strategy is used to fuse and sort the candidate action set with the experience-based recommended actions, generating a list of next intervention action recommendations that is pushed back to the execution module for the therapist to choose from.
[0032] The present invention has been described in detail above with reference to the accompanying drawings. However, the embodiments of the present invention are not limited to the above embodiments. Those skilled in the art can make various modifications to the present invention based on the prior art, and these modifications all fall within the protection scope of the present invention.
Claims
1. A control method for a diagnostic and treatment robot for children with autism, characterized in that, Includes the following steps: S1, the robot interacts with children by presenting multimodal interactive content. During the interaction, it records the child's facial video in real time and records the therapist's score on the child's performance. S2, using a deep learning model to analyze the face video, obtain the child's attention level and the valence and arousal of facial expressions, and generate the child's state code based on the score, attention level, and valence and arousal through weighted calculation; S3, Based on the multimodal interactive content and the child's state encoding, generate a recommended list of next intervention actions using the LSTM state prediction model and the CPT action decision model; S4, the therapist selects or modifies one of the recommended items from the list of recommended next intervention actions and executes it, forming new interactive data; S5, using the new interactive data, retrain and update the LSTM state prediction model and the CPT action decision model.
2. The control method for the autism diagnosis and treatment robot as described in claim 1, characterized in that, The multimodal interactive content includes video, audio, graphics, and / or text.
3. The control method for the autism diagnosis and treatment robot as described in claim 1, characterized in that, The deep learning model is the BlazeFace and EmoFAN deep learning model. In step S2, the BlazeFace and EmoFAN deep learning models are used to extract facial features from the face video, the child's gaze region is calculated based on the facial features, and the child's focus is estimated based on the gaze region.
4. The control method for the autism diagnosis and treatment robot as described in claim 1, characterized in that, In step S3, the LSTM state prediction model is first used to predict the child's possible emotions and concentration in the next round. At the same time, the CPT action decision model is used to predict experience-recommended actions based on the therapist's historical operating habits. Then, the HTN hierarchical task network is used to plan a logical set of candidate actions using DTT domain knowledge. Finally, a hierarchical strategy is used to merge and sort the candidate action set with the experience-recommended actions to generate a list of recommended intervention actions for the next step.
5. A control system for a diagnostic and treatment robot for children with autism, characterized in that, include: The execution module is used to receive control commands, interact with the child and therapist according to the control commands, record the child's facial video in real time, and record the therapist's score on the child's performance; A perception module is used to receive the facial video and the score, and to perform quantitative analysis on the child's real-time status based on the facial video and the score; The memory module stores short-term interaction data for the current lesson, each child's historical information set, skill level table, and the trained optimal computational model. It also provides the child's historical state sequence, the current optimal LSTM state prediction model and CPT action decision model, as well as a domain knowledge base containing intervention rules. It also provides historically accumulated sequence data for model updates. The evolution module is used to extract accumulated sequence data from the memory module according to a preset frequency or triggering conditions, retrain the LSTM state prediction model and the CPT action decision model, compare the performance of the new and old models on the validation set, automatically select the optimal computational model that is suitable for the current characteristics of children, and update it to the memory module. as well as The decision-making module is used to read the child's historical state sequence, the current optimal LSTM state prediction model, and the CPT action decision model provided by the memory module, and generate a recommendation for the next intervention action based on the read information.
6. The autism diagnosis and treatment robot control system as described in claim 5, characterized in that, The specific method by which the perception module performs quantitative analysis of the child's real-time state is as follows: facial features are extracted from the face video using BlazeFace and EmoFAN deep learning models, the child's gaze area is calculated based on the facial features, the child's focus is estimated based on the gaze area, the valence and arousal of facial expressions are identified, and a child's state code is generated by weighted calculation based on the score, the focus, and the valence and arousal.
7. The autism diagnosis and treatment robot control system as described in claim 5, characterized in that, The specific method for the decision-making module to generate recommendations for the next intervention action is as follows: First, the LSTM state prediction model is used to predict the child's possible emotions and concentration in the next round. At the same time, the CPT action decision model is used to predict experience-recommended actions based on the therapist's historical operating habits. Then, the HTN hierarchical task network is used to plan a logically consistent set of candidate actions using DTT domain knowledge. Finally, a hierarchical strategy is used to merge and sort the candidate action set with the experience-recommended actions to generate a list of recommended next intervention actions for the therapist to choose from.