An ASD child individualized screening and intervention system based on multi-modal data
By integrating multimodal perception technology, the system automatically collects behavioral data of children with ASD, provides personalized intervention game programs, and dynamically adjusts strategies. This solves the problems of subjectivity and low efficiency in traditional ASD screening and intervention, achieving efficient and personalized screening and intervention results.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- SUZHOU KUYUE NETWORK TECH CO LTD
- Filing Date
- 2026-03-11
- Publication Date
- 2026-06-12
AI Technical Summary
Traditional ASD screening and intervention methods rely on manual observation and scale assessment, which are highly subjective, inefficient, lack personalization, fail to meet the needs of different children, and have low participation rates.
Employing multimodal perception technology, integrating motion capture, speech recognition, eye tracking, and emotion perception modalities, it automatically collects behavioral data, provides personalized intervention game solutions, and dynamically adjusts intervention strategies through data tracking and evaluation modules.
This approach enables personalized and objective screening and precise intervention for children with ASD, improving screening efficiency and intervention effectiveness, and increasing children's participation and the effectiveness of intervention.
Smart Images
Figure CN122201645A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of children's mental health and rehabilitation technology, specifically to a personalized screening and intervention system for children with ASD based on multimodal data. Background Technology
[0002] Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder characterized by impairments in social interaction, communication, and behavioral patterns. ASD symptoms are diverse, ranging from mild to severe, and are often accompanied by repetitive behaviors, restricted interests, and sensory hypersensitivity. Due to the complexity and heterogeneity of ASD, early screening and personalized intervention are crucial for improving the symptoms and quality of life of children with ASD.
[0003] However, traditional ASD screening and intervention methods often rely on manual observation and scale assessments. While these methods can help identify children with ASD to some extent, they have several limitations: First, they are highly subjective: because the results of manual observation and scale assessments are often influenced by the assessor's experience, judgment, and subjective feelings, differences may exist between different assessors, easily leading to inconsistencies and inaccuracies in the assessment results. Second, they are inefficient: traditional methods require a lot of time and human resources, the assessment process is cumbersome and time-consuming, making it difficult to screen and assess a large number of children in a short period of time.
[0004] Furthermore, traditional intervention methods often adopt a "one-size-fits-all" approach, lacking sufficient consideration for the individual differences among children with ASD and failing to meet the personalized needs of different children. They also lack fun and interactivity, resulting in low participation among children with ASD and affecting the effectiveness of the intervention.
[0005] With the development of multimodal perception technology, it has become possible to objectively and accurately screen and intervene in children with ASD using multimodal data such as motion capture, speech recognition, eye tracking, and emotion perception. Given the limitations of traditional screening and intervention methods and the advantages of multimodal perception technology, it is particularly necessary to develop a personalized screening and intervention system for children with ASD based on multimodal data. Summary of the Invention
[0006] The purpose of this invention is to provide a personalized screening and intervention system for children with ASD based on multimodal data, so as to solve the problems mentioned in the background art.
[0007] To achieve the above objectives, the present invention provides the following technical solution: a personalized screening and intervention system for children with ASD based on multimodal data, comprising: The contactless multimodal identification module utilizes four modalities—motion capture, speech recognition, eye tracking, and emotion perception—to automatically collect children's behavioral data during interactions and map it to the 65 core items of the SRS scale (Autism Spectrum Rating Scale) to accurately determine children's shortcomings, symptom severity, and interests. Personalized Intervention Game Module: Responsible for providing personalized intervention game solutions for children based on the output results of the non-contact multimodal identification module; Data Tracking and Evaluation Module: Responsible for tracking children's performance data in play in real time, generating quantitative reports, and dynamically adjusting intervention plans based on data feedback to ensure maximum intervention effectiveness. It also generates comprehensive evaluation reports to provide parents and therapists with detailed analysis of intervention effects.
[0008] Furthermore, the details of the acquisition of the four modalities—motion capture, speech recognition, eye tracking, and emotion perception—are as follows: Motion capture: Data such as children's motion response speed and coordination are captured using a flagpole interactive platform or similar devices; the accuracy of the motion capture modality is evaluated using the following formula: ; Speech recognition: Collecting data such as the content of the child's speech response and the completeness of their speech expression; the accuracy of the speech recognition modality is calculated using the following formula: Eye tracking: Recording children's eye movement data, including gaze trajectory and gaze stability, to assess their response to social cues; the stability of the eye tracking modality is evaluated using the following formula: Among them, stable fixation time refers to the total time that the fixation point stays in a certain area for more than a preset threshold. Emotion perception: Analyze children's emotional states, such as happiness, tension, and expressionlessness, through facial expression recognition technology.
[0009] Furthermore, the contactless multimodal identification module outputs the following results: a child type report is generated, which clarifies the dimensions, severity, and interests of the shortcomings, providing a basis for personalized intervention.
[0010] Furthermore, the personalized intervention game module integrates popular IP characters such as "Octonauts" to increase fun and participation, and covers multiple dimensions such as social cognition, social communication, social motivation, repetitive and stereotyped behaviors, and social anxiety. Social cognition dimension: such as cracking facial expression codes and transmitting gesture signals, to improve the ability to recognize emotional expressions and understand social gestures; Social communication dimension: such as relaying messages during adventures, expressing needs for treasure, etc., to improve the ability to communicate in turns and clearly express needs; Social motivation dimension: such as treasure sharing sessions, cooperative sheep-hunting teams, etc., to enhance the willingness to share and the motivation to participate in groups; Dimensions of repetitive and stereotyped behaviors: such as ever-changing prayer flag designs, and varied movements, to break fixed thinking and reduce stereotyped actions; Social anxiety dimension: such as one-on-one greetings, calm breathing techniques, etc., to reduce anxiety when interacting with strangers, and learn emotion regulation methods.
[0011] Furthermore, the personalized intervention game module is designed with 20 independent game modules (5 modules for social cognition, 4 modules for social communication, 3 modules for social motivation, 3 modules for repetitive and stereotyped behaviors, 3 modules for social anxiety, and 2 modules for multi-dimensional integration). Each module is trained on one or more items of the SRS scale. In practice, specific triggering conditions and adaptation designs are set in the game modules to meet the needs of different children with ASD, such as reducing the volume of sound effects for children with hearing sensitivity and assigning simple roles to children with social avoidance. Each game module supports three levels of difficulty: basic, intermediate, and challenging, to suit children with different symptom severity. Specifically, children participate in the game by combining real-life digital avatars with role-playing (Octonauts characters) to increase immersion. The personalized intervention game module generates a game participation report and records the child's performance data in the game, such as task completion rate, facial expression recognition accuracy, and gesture recognition accuracy.
[0012] Furthermore, the children are categorized according to the severity of their symptoms: 1) Basic type children: Identify core characteristics: 1-2 dimensions of weakness, mild symptoms, ≤5 items in a single dimension that need improvement, and fair level of initiative in interaction; Corresponding SRS scale performance (data mapping): Data compliance rate ≥70%, frequency of repeated behaviors <3 times, and social anxiety percentage <20%; Intervention and adaptation principles: 2-3 modules, basic difficulty level, focusing on core weaknesses; 2) Advanced children: Identify core characteristics: 3-4 dimensions of weakness, moderate symptoms, 6-10 items in a single dimension that need improvement, and some avoidance behaviors; Corresponding SRS scale performance: data compliance rate 50%-69%, repetitive behavior frequency 3-5 times, social anxiety percentage 20%-40%; Intervention and adaptation principles: 4-5 modules, with difficulty levels ranging from basic to advanced, providing a tiered challenge; 3) Challenging children: Identify key characteristics: weaknesses in all 5 dimensions, severe symptoms, ≥11 items in a single dimension that need improvement, and obvious avoidance behavior; Corresponding SRS scale performance: Data compliance rate <50%, repetitive behavior frequency ≥6 times, social anxiety percentage >40%; Intervention and adaptation principles: 6-7 modules, basic difficulty level supported by enhanced level, and priority given to matching interests.
[0013] Furthermore, the data tracking and evaluation module includes the following sub-modules: Data integration submodule: Automatically summarizes objective data (device-collected) and subjective assessments (parent / therapist ratings) from the game module; The scoring calculation submodule calculates the scores for each dimension and the total score according to a preset weighting formula. The rating submodule categorizes students into rating levels based on their total score (Excellent, Good, Average, Needs Improvement; Excellent: Total score ≥ 85; Good: 70 ≤ Total score < 85; Average: 55 ≤ Total score < 70; Needs Improvement: Total score < 55) to assess the intervention's effectiveness. Report Output Submodule: Generates a comprehensive PDF report, including basic information (child's name, age, participation duration, etc.), radar chart of sub-scores, core areas for improvement, areas to be strengthened, and personalized intervention suggestions.
[0014] Furthermore, the preset weight formula is as follows: ;in, To assess the number of dimensions, For the first Scores in each dimension For the first Preset weights for each dimension.
[0015] Furthermore, the data tracking and evaluation module outputs a comprehensive evaluation report, providing scientific intervention guidance for parents and therapists.
[0016] Furthermore, the system is applicable to personalized screening and intervention for children with ASD in medical institutions, rehabilitation centers, or home environments. Application steps: S1. System Deployment: Deploy contactless multimodal identification platforms and game intervention devices in medical institutions, rehabilitation centers, or home environments; S2. Child Identification: Children complete multimodal interactive tasks in a contactless environment, and the system automatically collects data and generates a child type report; S3. Personalized Intervention: Based on the child type report, the system pushes personalized intervention game plans, and the child completes game tasks accompanied by real digital humans and Octonauts characters; S4. Data Tracking and Evaluation: The system tracks children's performance data in real time during play, generates quantitative reports and comprehensive evaluation reports, and provides a basis for dynamically adjusting intervention programs; S5. Continuous Optimization: Based on data feedback and children's progress, continuously optimize game modules and intervention strategies to improve intervention effectiveness.
[0017] This invention provides a personalized screening and intervention system for children with ASD based on multimodal data, which has the following beneficial effects: By integrating multimodal perception technology, it enables personalized screening and intervention for children with ASD. The system comprises three core components: a contactless multimodal identification module, a personalized intervention game module, and a data tracking and evaluation module. It can automatically collect children's behavioral data during interactions, accurately determine their social skills deficiencies, and provide targeted intervention plans.
[0018] 1. Multimodal data fusion: Integrating multimodal data such as motion capture, speech recognition, eye tracking, and emotion perception to achieve a comprehensive and objective assessment of children with ASD.
[0019] 2. Personalized intervention plans: Based on the child's weaknesses and interests, we provide targeted intervention game plans to improve the accuracy and effectiveness of the intervention.
[0020] 3. Gamified intervention: Integrating children's favorite IP characters and gamified design increases the fun and participation of the intervention, thereby improving children's compliance.
[0021] 4. Dynamic adjustment mechanism: Based on data tracking and evaluation results, the intervention plan is dynamically adjusted to ensure the maximum effect of the intervention. Attached Figure Description
[0022] Figure 1 This is an architectural block diagram of a personalized screening and intervention system for children with ASD based on multimodal data, according to the present invention. Figure 2 This is a schematic diagram illustrating the classification criteria for the screening results of a personalized screening and intervention system for ASD children based on multimodal data, as described in this invention. Figure 3 This is a module diagram of the social cognition dimension of a personalized screening and intervention system for ASD children based on multimodal data according to the present invention; Figure 4 This is a module diagram of the social communication dimension of a personalized screening and intervention system for ASD children based on multimodal data, as described in this invention. Figure 5 This is a module diagram of the social motivation dimension of a personalized screening and intervention system for ASD children based on multimodal data, as described in this invention. Figure 6 This is a module diagram of the repetitive and stereotyped behavior dimensions of a personalized screening and intervention system for ASD children based on multimodal data, as described in this invention. Figure 7This is a module diagram of the social anxiety dimension of a personalized screening and intervention system for ASD children based on multimodal data, as described in this invention. Figure 8 This is a multi-dimensional fusion module diagram of a personalized screening and intervention system for ASD children based on multimodal data according to the present invention. Detailed Implementation
[0023] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.
[0024] Example 1: Personalized Screening and Intervention for Children with Basic ASD Children stand in front of the flagpole interactive platform. The contactless multimodal identification module generates a lifelike digital avatar through 3D scanning, allowing them to choose the role of Captain Barnacles from "Octonauts." After completing the digital avatar creation, they enter the Yamdrok Lake Exploration interactive session. The contactless multimodal identification module collects behavioral data on the children during the interaction through four modalities: motion capture, speech recognition, eye tracking, and emotion perception. Simultaneously, the data is automatically integrated, mapped to the 65 items of the SRS scale, and a child type report is generated. The report identifies the weakness dimension as social cognition (difficulty distinguishing facial expressions, not understanding gestures, not understanding turn-taking rules), the severity of the symptoms as mild, and the interest preference as exploratory.
[0025] The personalized intervention game module pushes personalized intervention game plans based on the child's type report, including three modules: Facial Expression Code Cracking, Gesture Signal Transmission, and Adventure Rules Learning. Facial Expression Code Cracking: Children identify the facial expressions of their Octonauts companions using flagpole gestures, improving their ability to recognize emotional expressions. Gesture Signal Transmission: Children imitate commonly used gestures from the Octonauts and complete designated tasks through gestures, understanding the meaning of social gestures. Adventure Rules Learning: Children complete simulated tasks through flagpole interaction, learning and understanding the social rules of taking turns exploring.
[0026] The data tracking and evaluation module monitors children's performance data in real time, such as facial expression matching accuracy, gesture recognition accuracy, and rule compliance rate. Weekly quantitative and comprehensive evaluation reports are generated, showing children's progress in social cognition, and intervention plans are dynamically adjusted based on data feedback.
[0027] Implementation results: 1. Children showed significant improvement in facial expression recognition, gesture comprehension, and adherence to turn-taking rules.
[0028] 2. The quantitative report shows that the accuracy rate of facial expression recognition reached 85%, the gesture comprehension rate reached 90%, and the rule compliance rate reached 85%.
[0029] 3. Feedback from parents and therapists indicates that children show greater interest and engagement in social interactions.
[0030] Example 2: Personalized Screening and Intervention for Children with Advanced ASD Children create their digital personas and choose the character Kwazii from "Octonauts." They then enter the Yamdrok Lake Exploration interactive session, where a contactless multimodal identification module collects multimodal data to generate a child type report. The report identifies weaknesses including social communication (inability to take turns, unclear expression, lack of polite language), social motivation (lack of sharing, refusal to accept invitations), and social anxiety (fear of strangers, avoidance of eye contact). The severity of these symptoms is classified as moderate, and the child's interests are categorized as social.
[0031] The personalized intervention game module pushes personalized intervention game plans based on the child type report, including five modules: adventure relay relay, treasure sharing session, one-on-one greeting, treasure needs expression, and Yamdrok Lake party preparation.
[0032] 1) Relay Message Challenge: Children team up with virtual partners to pass on instructions via voice commands through a flagpole, improving their ability to take turns communicating.
[0033] 2) Treasure Sharing Session: Children share the treasures they have collected, enhancing their willingness to share.
[0034] 3) One-on-one greetings: Children greet virtual herdsmen, reducing anxiety about interacting with strangers.
[0035] 4) Expressing Needs Through the Flagpole Voice: Children can clearly express their needs through the flagpole voice, thus improving their expression skills.
[0036] 5) Yamdrok Lake Party Preparation: Children form teams to prepare for the party, making comprehensive use of communication, sharing and invitation skills.
[0037] The data tracking and evaluation module monitors children's performance data in real time, generating quantitative and comprehensive evaluation reports. Based on data feedback, intervention programs are dynamically adjusted, such as increasing the difficulty of the social communication module and reducing the frequency of the social anxiety module.
[0038] Implementation results: 1. Children have made significant progress in taking turns communicating, sharing proactively, and interacting with strangers.
[0039] 2. The quantitative report shows that the rate of achieving the goal of taking turns communicating reached 70%, the number of times of proactive sharing increased, and anxiety about interacting with strangers was significantly reduced.
[0040] 3. Parents and therapists report that children are more confident and proactive in social situations.
[0041] Example 3: Personalized Screening and Intervention for Children with Challenging ASD Children create their digital personas and choose the character Dr. Pi from "Octonauts." They then enter the Yamdrok Lake Exploration Interactive Session, where a contactless multimodal identification module collects multimodal data to generate a child type report. The report identifies shortcomings across all dimensions, including social cognition, social communication, social motivation, repetitive and stereotyped behaviors, and social anxiety. The severity of the symptoms is classified as severe, and the child's interests are categorized as creative.
[0042] The personalized intervention game module pushes personalized intervention game plans based on the child's type report, including seven modules: prayer flag design, facial expression code cracking, one-on-one greeting, treasure hunting needs expression, calm breathing method, Yamdrok Lake party preparation, and the great adventure of protecting the sheep.
[0043] 1) Versatile prayer flag designs: Children design prayer flags, breaking fixed thinking and reducing stereotyped actions.
[0044] 2) Facial Expression Code Cracking: Improve the ability to recognize emotional expressions.
[0045] 3) One-on-one greetings: reduce anxiety when interacting with strangers.
[0046] 4) Expressing Needs Clearly: Clearly express your needs and improve your communication skills.
[0047] 5) Calm breathing technique: Learn emotion regulation methods to relieve social anxiety.
[0048] 6) Yamdrok Lake Party Preparation: Utilize social skills to prepare for the party.
[0049] 7) Protect the Sheep Adventure: Identify the sheep's emotions, flexibly adjust the route, and reduce stereotypical behaviors.
[0050] The data tracking and evaluation module monitors children's performance data in real time, generating quantitative and comprehensive evaluation reports. Based on data feedback, the intervention plan is dynamically adjusted, such as increasing the difficulty of the social cognition and social communication modules, and strengthening support for the social motivation and emotion regulation modules.
[0051] Implementation results: 1. Children showed improvement in all aspects of social skills, especially in creative modules, demonstrating high levels of participation and interest.
[0052] 2. The quantitative report shows that the participation rate in creative modules reached 90%, and social anxiety and stereotyped behaviors have decreased.
[0053] 3. Feedback from parents and therapists indicates that children are more flexible and confident in social interactions and are better able to cope with various social situations.
[0054] The embodiments of the present invention are given for illustrative and descriptive purposes only, and are not intended to be exhaustive or to limit the invention to the forms disclosed. Many modifications and variations will be apparent to those skilled in the art. The embodiments were chosen and described in order to better illustrate the principles and practical application of the invention, and to enable those skilled in the art to understand the invention and to design various embodiments with various modifications suitable for a particular purpose.
Claims
1. A personalized screening and intervention system for children with ASD based on multimodal data, characterized in that, include: The contactless multimodal identification module utilizes four modalities—motion capture, speech recognition, eye tracking, and emotion perception—to automatically collect children's behavioral data during interactions and map it to the 65 core items of the SRS scale, accurately determining children's weakness dimensions, symptom severity, and interest preferences. Personalized Intervention Game Module: Responsible for providing personalized intervention game solutions for children based on the output results of the non-contact multimodal identification module; Data Tracking and Evaluation Module: Responsible for tracking children's performance data in real time, generating quantitative reports, and dynamically adjusting intervention plans based on data feedback to generate comprehensive evaluation reports.
2. The personalized screening and intervention system for ASD children based on multimodal data according to claim 1, characterized in that, The details of the acquisition of the four modalities—motion capture, speech recognition, eye tracking, and emotion perception—are as follows: Motion capture: Data on children's motion response speed and coordination are captured using a flagpole interactive platform or similar device; the accuracy of the motion capture modality is evaluated using the following formula: ; Speech recognition: Collects data on the child's speech responses and the completeness of their speech expression; the accuracy of the speech recognition modality is calculated using the following formula: Eye tracking: Recording children's eye movement data, including gaze trajectory and gaze stability, to assess their response to social cues; the stability of the eye tracking modality is evaluated using the following formula: Among them, stable fixation time refers to the total time that the fixation point stays in a certain area for more than a preset threshold. Emotion perception: Analyze children's emotional state through facial expression recognition technology.
3. The personalized screening and intervention system for ASD children based on multimodal data according to claim 1, characterized in that, The contactless multimodal identification module outputs the following results: a child type report is generated, which clarifies the dimensions, severity, and interests of the child, providing a basis for personalized intervention.
4. The personalized screening and intervention system for ASD children based on multimodal data according to claim 1, characterized in that, The personalized intervention game module integrates IP characters that children love and covers dimensions such as social cognition, social communication, social motivation, repetitive and stereotyped behaviors, and social anxiety.
5. A personalized screening and intervention system for children with ASD based on multimodal data according to claim 1, characterized in that, The personalized intervention game module is designed with 20 independent game modules. Each module is trained on one or more items of the SRS scale, and each game module supports three levels of difficulty: basic, intermediate, and challenging, to suit children with different symptom severity. In practice, children participate in the game by combining real-life digital avatars with role-playing. The personalized intervention game module generates a game participation report and records the child's performance data in the game.
6. A personalized screening and intervention system for ASD children based on multimodal data according to claim 5, characterized in that, The children are categorized according to the severity of their symptoms: 1) Basic type children: Identify core characteristics: 1-2 dimensions of weakness, mild symptoms, ≤5 items in a single dimension that need improvement, and fair level of initiative in interaction; Corresponding SRS scale performance: data compliance rate ≥70%, frequency of repeated behaviors <3 times, and social anxiety percentage <20%; Intervention and adaptation principles: 2-3 modules, basic difficulty level, focusing on core weaknesses; 2) Advanced children: Identify core characteristics: 3-4 dimensions of weakness, moderate symptoms, 6-10 items in a single dimension that need improvement, and some avoidance behaviors; Corresponding SRS scale performance: data compliance rate 50%-69%, repetitive behavior frequency 3-5 times, social anxiety percentage 20%-40%; Intervention and adaptation principles: 4-5 modules, with difficulty levels ranging from basic to advanced, providing a tiered challenge; 3) Challenging children: Identify key characteristics: weaknesses in all 5 dimensions, severe symptoms, ≥11 items in a single dimension that need improvement, and obvious avoidance behavior; Corresponding SRS scale performance: Data compliance rate <50%, repetitive behavior frequency ≥6 times, social anxiety percentage >40%; Intervention and adaptation principles: 6-7 modules, basic difficulty level supported by enhanced level, and priority given to matching interests.
7. A personalized screening and intervention system for children with ASD based on multimodal data according to claim 1, characterized in that, The data tracking and evaluation module includes the following sub-modules: Data integration submodule: Automatically summarizes objective data and subjective evaluations from the game module; The scoring calculation submodule calculates the scores for each dimension and the total score according to a preset weighting formula. The rating submodule categorizes students into rating levels based on their total scores to assess the effectiveness of the intervention. Report Output Submodule: Generates a comprehensive PDF report, including basic information, radar charts of sub-scores, key areas for improvement, areas for reinforcement, and personalized intervention suggestions.
8. A personalized screening and intervention system for ASD children based on multimodal data according to claim 7, characterized in that, The preset weighting formula is: ;in, To assess the number of dimensions, For the first Scores in each dimension For the first Preset weights for each dimension.
9. A personalized screening and intervention system for children with ASD based on multimodal data according to claim 1, characterized in that, The data tracking and evaluation module outputs a comprehensive evaluation report, providing scientific intervention guidance for parents and therapists.
10. A personalized screening and intervention system for ASD children based on multimodal data according to any one of claims 1-9, characterized in that, The system is suitable for personalized screening and intervention of children with ASD in medical institutions, rehabilitation centers, or home environments. The application steps of the system are as follows: S1. System Deployment: Deploy contactless multimodal identification platforms and game intervention devices in medical institutions, rehabilitation centers, or home environments; S2. Child Identification: Children complete multimodal interactive tasks in a contactless environment, and the system automatically collects data and generates a child type report; S3. Personalized Intervention: Based on the child type report, the system pushes personalized intervention game plans, and the child completes game tasks with the accompaniment of real digital people and characters; S4. Data Tracking and Evaluation: The system tracks children's performance data in real time during play, generates quantitative reports and comprehensive evaluation reports, and provides a basis for dynamically adjusting intervention programs; S5. Continuous Optimization: Based on data feedback and children's progress, continuously optimize game modules and intervention strategies to improve intervention effectiveness.