Language evaluation method and device for autistic children and medium

Through multimodal data extraction and personalized interactive task generation, the problem of poor evaluation of existing systems in complex social interaction and non-verbal communication is solved, and more accurate and comprehensive language evaluation of children with autism is achieved, and the evaluation effect is improved. and the feasibility of personalized interventions.

CN119028594APending Publication Date: 2024-11-26CHINA UNIV OF PETROLEUM (EAST CHINA)

Patent Information

Application Number
CN202411534257.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-31
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

The existing language assessment system for children with autism is poor in dealing with complex social interactions and nonverbal communication, resulting in poor evaluation results.

Method used

By obtaining multimodal response data (language data, visual data and behavioral data) in children with autism, the pitch, tone, sound strength, vocabulary, sentence structure, visual focus, eye movement trajectory, facial expressions and body movements are extracted. and other features, generate personalized interactive tasks, and obtain comprehensive language data through multiple iterative interactions for evaluation.

Benefits of technology

It improves the accuracy and effectiveness of language evaluation in children with autism, can more comprehensively evaluate language ability and learning process, and provides personalized intervention suggestions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119028594A_ABST
    Figure CN119028594A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of language evaluation, and discloses a language evaluation method and device for autistic children and a medium. The method comprises the following steps: acquiring language data, visual data and behavior data of an autistic child when the autistic child interacts with the autistic child; extracting multi-modal reaction characteristics of the autistic children from the language data, the visual data and the behavior data; according to the multi-mode reaction characteristics of the autistic child, generating an interaction task conforming to the current language ability of the autistic child so as to continuously interact with the autistic child through the generated interaction task; according to the method, all the language data of the autistic children in the multi-round iterative interaction process with the autistic children are obtained and serve as the comprehensive language data, the language ability of the autistic children is evaluated through the comprehensive language data, and the language evaluation effect of the autistic children can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of language assessment, and in particular to a method, device and medium for language assessment of autistic children. Background Art

[0002] Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that widely affects social interaction, communication skills and behavior. Children with autism have persistent and significant deficits in social-emotional interaction, non-verbal communication behaviors, and the establishment and maintenance of relationships, which greatly affects their social and learning abilities, making effective language assessment and early intervention particularly important.

[0003] With the development of computer technology, more and more studies are trying to apply speech recognition, natural language processing and other technologies to the language assessment of autistic children. They use speech recognition and text analysis technology to assess children's language development and provide personalized intervention suggestions. However, the personalized design of existing systems is still insufficient, especially in dealing with complex social interactions and non-verbal communication. The performance of the system is often unsatisfactory, resulting in poor evaluation of the language of autistic children. Summary of the invention

[0004] The purpose of the present invention is to provide a method, device and medium for language assessment of autistic children, which can improve the evaluation effect of the language of autistic children.

[0005] To solve the above technical problems, an embodiment of the present invention provides a method for language assessment of autistic children, comprising the following steps: Acquiring multimodal response data of the autistic child when interacting with the autistic child, wherein the multimodal response data includes language data, visual data, and behavioral data; Extract pitch, tone, intensity, and duration of pronunciation from language data, as well as extract vocabulary, sentence structure, and semantic information; extract visual focus, eye movement trajectories, and facial expressions from visual data; extract body movements, head movements, and gaze behaviors from behavioral data to obtain multimodal response characteristics of children with autism; According to the multimodal response characteristics of autistic children, generate interactive tasks that are consistent with the current language ability of autistic children, so as to continue to interact with the autistic children through the generated interactive tasks; All language data of autistic children during multiple rounds of iterative interactions with them are obtained as comprehensive language data, and the language ability of autistic children is evaluated through the comprehensive language data.

[0006] In some optional embodiments, the language ability of the autistic child is evaluated by comprehensive language data, including: Based on the average pitch, pitch variation, and pitch stability in the comprehensive language data, the tone control and tone variation abilities of children with autism were evaluated. Based on the frequency fluctuation of pitch in the comprehensive language data, we evaluate whether children with autism can maintain stable pitch control under different emotions and in different expressions; By comprehensively analyzing the changes in sound intensity in language data and the distribution of sound intensity in different sentences and words, the sound control ability of children with autism in expression is evaluated; The pronunciation fluency and rhythm of children with autism were evaluated by measuring the duration of pronunciation of each sentence or word in the comprehensive language data; According to the accuracy of the words used by autistic children in the comprehensive language data, the vocabulary reserve and vocabulary application ability of autistic children are evaluated; The language organization ability and grammatical mastery level of autistic children were assessed based on the complexity of the sentence structures used by autistic children in the comprehensive language data; By comprehensively comparing the semantic similarity between the responses of autistic children to interactive tasks in language data and the preset standard responses, we evaluate whether autistic children can accurately convey their intentions in their expressions.

[0007] In some optional embodiments, the language ability of the autistic child is evaluated by comprehensive language data, including: According to the responses of autistic children to each interactive task in the comprehensive language data and the corresponding preset standard responses, the proportion of the number of interactive tasks to which autistic children correctly responded to account for the total number of interactive tasks was determined to assess the accuracy of the language expression of autistic children when faced with new vocabulary or new sentence patterns; The average reaction time of autistic children in responding to interactive tasks in the comprehensive language data was used to evaluate the reaction speed of autistic children in language comprehension and expression; According to the vocabulary used by autistic children in the interaction process in the comprehensive language data, the proportion of the number of new vocabulary correctly used by autistic children to the total number of new vocabulary was determined to evaluate the efficiency of autistic children in learning new vocabulary; Based on the sentence structures used by autistic children during the interaction in the comprehensive language data, determine the number of new sentence structures correctly used by autistic children to assess the ability of autistic children to learn new sentence structures; Based on the responses of autistic children to interactive tasks in the comprehensive language data and the preset standard responses, the proportion of successfully corrected errors to the total number of errors made by autistic children after they made incorrect responses to interactive tasks was determined to assess the ability of autistic children to cope with errors.

[0008] In some optional embodiments, after evaluating the language ability of the autistic child by comprehensive language data, the method further includes: Acquire all visual data of the autistic child during the continuous interaction with the autistic child as comprehensive visual data; By integrating the frequency, duration and eye movement trajectory of visual focus changes in visual data, the degree of attention of autistic children during the interaction was evaluated; The frequency, intensity, and duration of facial expression changes in comprehensive visual data were used to assess the emotional state of children with autism during interaction.

[0009] In some optional embodiments, after evaluating the language ability of the autistic child by comprehensive language data, the method further includes: Acquire all behavioral data of the autistic child during the continuous interaction with the autistic child as comprehensive behavioral data; By integrating the body movements, head movements and gaze behaviors of autistic children in interactive tasks with the same context in the comprehensive behavioral data, the behavioral stability and concentration of autistic children in the same context are evaluated.

[0010] In some optional embodiments, the step of obtaining multimodal response data of an autistic child when interacting with the autistic child includes: Virtual characters interact with autistic children in virtual scenes, and obtain multimodal response data of autistic children during the interaction.

[0011] In some optional embodiments, generating a personalized interactive task that meets the current language ability of the autistic child based on the multimodal response characteristics of the autistic child includes: Converting the multimodal response features of autistic children into prompt words that conform to the input format of the large model, wherein the prompt words include feature descriptions, content summaries, and context information of the multimodal response features; Generate personalized interactive tasks that match the current language abilities of children with autism through a large model.

[0012] In some optional embodiments, the multimodal response data also includes electroencephalogram and functional magnetic resonance imaging.

[0013] An embodiment of the present invention also provides a computer device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned language assessment method for autistic children.

[0014] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which implements the above-mentioned language assessment method for autistic children when executed by a processor.

[0015] The language assessment method for autistic children provided by the present invention has at least the following beneficial effects: The present invention evaluates the language of autistic children by using the language data of autistic children when interacting with the present invention, and when the autistic children interact, the interaction task based on which the interaction is based on is constantly changing based on the multimodal response data (including language data, visual data and behavioral data) of the autistic children, such as generating soothing language when the emotion detects that the child is anxious, or switching to a more attractive topic when a decrease in attention is detected, thereby ensuring that the interaction task is both challenging and within the ability of the child. In this case, by evaluating the language of autistic children by using the language data of autistic children when interacting with the present invention, more accurate evaluation results can be obtained.

[0016] The specific basis for the evaluation is all the language data of autistic children (i.e. comprehensive language data) during multiple rounds of iterative interactions with autistic children. This data not only reflects the language ability of autistic children, but also describes the language learning process of autistic children in continuous interaction, thus reflecting the language learning ability of autistic children and helping to improve the evaluation effect of autistic children's language. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] One or more embodiments are exemplarily described by the pictures in the corresponding drawings, and these exemplary descriptions do not constitute limitations on the embodiments.

[0018] Figure 1 is a flow chart of a method for language assessment of autistic children provided according to an embodiment of the present invention; Figure 2 It is a structural schematic diagram of a computer device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings. However, it will be appreciated by those skilled in the art that in the embodiments of the present invention, many technical details are proposed in order to enable the reader to better understand the present invention. However, even without these technical details and various changes and modifications based on the following embodiments, the technical scheme claimed in the present invention can be implemented. The division of the following embodiments is for the convenience of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined and referenced with each other without contradiction.

[0020] One embodiment of the present invention relates to a language assessment method for autistic children. The implementation details of the language assessment method for autistic children in this embodiment are specifically described below. The following content is only the implementation details provided for easy understanding and is not necessary for implementing this solution.

[0021] The specific process of the language assessment method for autistic children in this embodiment can be as follows: Figure 1 As shown, including: Step 101, obtaining multimodal response data of an autistic child when interacting with the autistic child, the multimodal response data including language data, visual data and behavioral data.

[0022] Specifically, the present embodiment is to interact with autistic children in a virtual scene through a virtual character, and then the multimodal response data of autistic children can be obtained when the virtual character interacts with autistic children in a virtual scene. In addition, the interaction is based on a preset interactive task. The present embodiment introduces multisensory stimulation in the interaction, including visual, auditory and tactile elements. The virtual character attracts the attention of children through rich expressions and variable intonation performances, reduces tension, and integrates realistic interactive elements such as touch feedback or eye tracking technology into the virtual scene to further enhance the sense of participation and interactive experience of children. This multisensory fusion design helps to improve the concentration of children and ensures that they can fully participate and perform during the test process. And the multimodal response data includes language data, visual data and behavioral data, which can accurately record the voice characteristics, emotional changes and behavioral responses of autistic children, and provide key data support for subsequent analysis and evaluation.

[0023] In one example, multimodal response data also includes biological signals such as electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI). Through the fusion of biological signals, we can gain a deeper understanding of the characteristics of neural activity in autistic children during language processing, which will not only help to further improve the accuracy of the assessment, but may also provide a scientific basis for the development of new treatment and intervention methods. Biological signals such as electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI) can be collected under special clinical or laboratory conditions, mainly used to improve the accuracy of the assessment, and may not be used in routine assessments.

[0024] Step 102, extracting pitch, tone, volume and pronunciation duration from the language data, and extracting vocabulary, sentence structure and semantic information therein, extracting visual focus, eye movement trajectory and facial expression from the visual data, and extracting body movements, head movements and gaze behaviors from the behavioral data, so as to obtain multimodal response characteristics of children with autism.

[0025] Specifically, firstly, the acquired multimodal response data is preprocessed. In this embodiment, a multi-level feature extraction technology is used to divide the multimodal response data into the following parts: (1) Speech features: including pitch, tone, intensity, duration of pronunciation, etc., through advanced acoustic analysis technology, evaluate the pronunciation accuracy, intonation control and emotional expression of children. The analysis of speech features not only includes basic acoustic parameters, but also involves more complex semantic analysis to determine the understanding and expression ability of autistic children of language content.

[0026] (2) Natural language content: Extract and analyze vocabulary, sentence structure, and semantic information in speech. By analyzing the complexity of sentence structure, the diversity of vocabulary, and the accuracy of semantics, the system can assess the language comprehension and expression ability of children with autism. This part of the content analysis also combines the assessment of contextual comprehension and pragmatic ability to more comprehensively reflect the language level of children with autism.

[0027] (3) Emotional and behavioral characteristics: Analyze children’s visual focus, eye movement, facial expressions, body movements, head movements, and gaze behaviors to assess their concentration, emotional state, and behavioral responses. Through eye tracking equipment and expression analysis systems, the system can capture subtle behavioral and emotional changes exhibited by children with autism during the test.

[0028] The above-mentioned speech features, natural language content, behavior and emotion features constitute the multimodal response characteristics of autistic children described in this embodiment.

[0029] In the specific implementation, this embodiment uses deep learning models such as convolutional neural networks to extract features from the multimodal response data of autistic children. For example, by analyzing the frequency and time domains of language data, language features such as pitch and intonation are extracted; by processing visual data with an eye tracking algorithm, visual features such as visual focus and facial expression changes are extracted; and by analyzing behavioral data with a posture recognition algorithm, behavioral features such as limb movements and head movements are extracted.

[0030] Step 103, generating an interactive task that meets the current language ability of the autistic child based on the multimodal response characteristics of the autistic child, so as to continue to interact with the autistic child through the generated interactive task.

[0031] Specifically, the multimodal response features of the autistic child are converted into prompt words that conform to the input format of the large model, and the prompt words include feature descriptions, content summaries, and context information of the multimodal response features, and then the large model is used to generate interactive tasks that conform to the current language ability of the autistic child. This embodiment is equivalent to adjusting the initial interactive task in step 101, so as to facilitate the subsequent interaction with the autistic child through the adjusted interactive task.

[0032] For example, the multimodal response features input to the large model are as follows: Speech characteristics: the pitch is high and slightly fluctuating, the tone is flat, the sound intensity is medium, and the pronunciation lasts for a long time; natural language content: keywords include "apple", "red", and "eat", and the sentence structure is simple, such as "I like to eat apples", which expresses the preference for food; behavioral and emotional characteristics: eye movement data show that children focus their eyes when mentioning "apple", and their facial expressions show positive emotions.

[0033] Then, the above input is converted into prompt words: the prompt words are "The child's voice shows medium intensity and flat tone, using a simple sentence 'I like to eat apples', and key words include 'apple', 'red', and 'eat'. Generate appropriate responses, express interest in food, maintain simple language structure, and guide children to discuss other related topics."

[0034] After receiving the prompt word, the large model uses a complex language generation algorithm and combines the child's multimodal data to output a response that matches their current language ability. The generated language content must not only be easy to understand, but also be guiding to encourage children to further express their ideas. Through the fusion of multimodal data, the model can dynamically adjust the content of the interaction, such as generating soothing language when the emotion detects that the child is anxious, or switching to a more attractive topic when a decrease in attention is detected.

[0035] In a specific implementation, the generation of the above-mentioned interactive tasks is based on the multimodal response characteristics of autistic children, and a natural language generation model (such as GPT) is used to create interactive tasks that adapt to current language abilities. This embodiment monitors the multimodal response data of children during the interactive tasks in real time, and dynamically adjusts the difficulty of the interactive tasks through a reinforcement learning algorithm, thereby ensuring that the interactive tasks are both challenging and suitable for the language development level of autistic children.

[0036] Step 104, obtaining all language data of the autistic child during the continuous interaction with the autistic child as comprehensive language data, and evaluating the language ability of the autistic child through the comprehensive language data.

[0037] Specifically, by repeatedly executing steps 101 to 103, all language data of the autistic child during multiple rounds of iterative interactions with the autistic child can be obtained, and the data can be used as comprehensive language data for evaluating the language ability of the autistic child.

[0038] In the specific implementation, in order to ensure that the data accumulated during the long-term interaction process is accurate and reliable, this embodiment introduces a data smoothing algorithm and anomaly detection technology. By processing the comprehensive language data, abnormal data points can be automatically identified and eliminated, noise interference can be reduced, and the stability and accuracy of the comprehensive evaluation results can be further improved.

[0039] In this embodiment, the language of autistic children is evaluated by using the language data of autistic children when interacting with them, and when autistic children interact, the interactive task based on which the interaction is based on the multimodal response data (including language data, visual data and behavioral data) of autistic children is constantly changing, such as generating soothing language when the emotion detects that the child is anxious, or switching to a more attractive topic when attention is detected to be decreased, which can ensure that the interactive task is both challenging and within the ability of the child. In this case, the language of autistic children is evaluated by using the language data of autistic children when interacting with them, and a more accurate evaluation result can be obtained. The evaluation is specifically based on all the language data (i.e., comprehensive language data) of autistic children during multiple rounds of iterative interaction with autistic children. This data can not only reflect the language ability of autistic children, but also describe the language learning process of autistic children in continuous interaction, thereby reflecting the language learning ability of autistic children and helping to improve the evaluation effect of autistic children's language.

[0040] In some embodiments, after obtaining comprehensive language data during continuous interaction with autistic children, the language ability of the children is assessed using two methods: standardized assessment and dynamic assessment.

[0041] Among them, the standardized assessment specifically includes the following assessment contents: assessing the tone control and tone change ability of autistic children based on the average value of pitch, the amplitude of pitch change and the stability of pitch in the comprehensive language data; assessing whether autistic children can maintain stable tone control under different emotions and in different expressions based on the frequency fluctuation of tone in the comprehensive language data; assessing the sound control ability of autistic children in expression through the changes in sound intensity in the comprehensive language data and the distribution of sound intensity in different sentences and words; assessing the pronunciation fluency and sense of rhythm of autistic children through the pronunciation duration of each sentence or word in the comprehensive language data; assessing the vocabulary reserve and vocabulary application ability of autistic children based on the accuracy of the vocabulary used by autistic children in the comprehensive language data; assessing the language organization ability and grammar mastery level of autistic children based on the complexity of the sentence structure used by autistic children in the comprehensive language data; assessing whether autistic children can accurately convey their intentions in expression through the semantic similarity between the response content of autistic children to interactive tasks in the comprehensive language data and the preset standard response content.

[0042] It can be seen that the above evaluation is actually an evaluation of the speech features and natural language content in the above embodiment: (1) Speech feature evaluation: Pitch: Assess the pitch characteristics of autistic children's speech, including the average pitch, the range of pitch variation, and pitch stability. These indicators are used to measure the tone control ability of autistic children when they pronounce words. For example, by calculating the difference between the actual pitch value and the standard pitch range, the pitch accuracy assessment is obtained.

[0043] Pitch: Pitch assessment focuses on the stability and change pattern of pitch in the voice of autistic children, analyzes the frequency changes of speech, and assesses whether autistic children can maintain stable pitch control under different emotions and expressions.

[0044] Sound intensity: Sound intensity analysis not only measures the average intensity of speech, but also focuses on the change pattern of sound intensity and its distribution in different sentences and words. For example, by comparing the difference between the mean sound intensity and the standard value, it is possible to evaluate the sound control of autistic children when expressing themselves.

[0045] Pronunciation duration: Calculate the pronunciation duration of each sentence or word and compare it with the standard pronunciation time to measure the pronunciation fluency and rhythm of children with autism. This indicator is particularly important for evaluating whether children with autism have problems with rhythm and fluency in language expression.

[0046] (2) Natural language content evaluation: Vocabulary use: This assessment is based on the number and quality of keywords that children with autism use correctly when answering questions. The system analyzes the corpus to determine whether the vocabulary used by children with autism meets the standard level for their age group and calculates the proportion of correct keywords to the total keywords to measure their vocabulary mastery.

[0047] Sentence structure: Analyze the complexity of the sentence structure used by autistic children, including sentence length, frequency of clause use, and grammatical correctness. In other words, by analyzing the complexity of syntactic structure, the system can assess the ability of autistic children to construct sentences.

[0048] Semantic accuracy: Semantic analysis is performed by comparing the semantic similarity between the answers of autistic children and the expected answers. Based on the semantic understanding evaluation model, the relevance and accuracy of the content are scored to evaluate whether the autistic children have accurately conveyed their intentions.

[0049] In one example, after obtaining comprehensive language data during continuous interaction with an autistic child, the child's language ability is assessed, and all visual data of the autistic child during the continuous interaction with the autistic child is also obtained as comprehensive visual data. At this time, the standardized assessment also includes the following assessment content: the frequency, duration, and eye movement trajectory of the change of visual focus in the comprehensive visual data are used to assess the concentration of the autistic child during the interaction; the frequency, intensity, and duration of facial expression changes in the comprehensive visual data are used to assess the emotional state of the autistic child during the interaction.

[0050] It can be seen that the above evaluation is actually an evaluation of the above-mentioned behavioral and emotional characteristics: (1) Visual attention assessment: Through eye tracking technology, it is possible to analyze the visual attention distribution of autistic children when answering questions, including the frequency and duration of changes in gaze points. The assessment model will compare the eye movement data of autistic children with standard eye movement data to assess their degree of concentration when answering questions.

[0051] (2) Emotional response assessment: By capturing the facial expressions of children with autism and judging their emotional state during the answering process, this part of the assessment will combine the frequency, intensity and duration of expression changes to assess the emotional responses of children with autism in different situations.

[0052] Dynamic assessment is designed to evaluate children's progress and adaptability in the language learning process. The assessment system focuses on autistic children's performance in task completion, mastery of new knowledge, and response to errors and challenges.

[0053] It specifically includes the following evaluation contents: according to the response content of autistic children to each interactive task in the comprehensive language data and the corresponding preset standard response content, determine the proportion of the number of interactive tasks to which autistic children correctly respond to the total number of interactive tasks, so as to evaluate the accuracy of the language expression of autistic children when facing new vocabulary or new sentence patterns; through the average reaction time of autistic children in responding to interactive tasks in the comprehensive language data, evaluate the reaction speed of autistic children in language comprehension and expression; according to the vocabulary used by autistic children in the interactive process in the comprehensive language data, determine the proportion of the number of new vocabulary correctly used by autistic children to the total number of new vocabulary, so as to evaluate the efficiency of autistic children in learning new vocabulary; according to the sentence structure used by autistic children in the interactive process in the comprehensive language data, determine the number of new sentence structures correctly used by autistic children, so as to evaluate the ability of autistic children to learn new sentence structures; according to the response content of autistic children to interactive tasks in the comprehensive language data and the preset standard response content, determine the proportion of the number of errors successfully corrected by autistic children after incorrectly responding to interactive tasks to the total number of errors, so as to evaluate the ability of autistic children to deal with errors.

[0054] It can be seen that it is evaluated from the following two aspects: (1) Task completion evaluation: Task accuracy: The system calculates the accuracy of autistic children in completing tasks in dynamic assessments. This assessment is based on the ratio of the number of correct answers given by autistic children in a given task to the total number of tasks, and is used to measure the accuracy of autistic children when faced with new vocabulary or new sentence patterns.

[0055] Reaction time: Evaluate the average reaction time of autistic children when completing tasks. By comparing the difference between the actual reaction time and the standard reaction time, it is possible to evaluate the reaction speed of autistic children in language comprehension and expression.

[0056] (2) Language learning progress assessment: New vocabulary mastery: Assess the ability of autistic children to learn and correctly use new vocabulary in tasks. The evaluation model calculates the proportion of new vocabulary correctly used by autistic children in tasks to the total number of new vocabulary to assess their efficiency in learning new vocabulary.

[0057] Sentence structure mastery: Pay attention to whether autistic children can correctly use the newly learned sentence structure in tasks, and analyze the frequency and accuracy of their sentence structure use in different tasks.

[0058] Error Correction Ability: Analyze the ability of children with autism to cope with errors in the task, including whether they can successfully correct the errors after feedback. The evaluation will be based on the ratio of the number of successfully corrected errors to the total number of errors to assess their ability to adapt and adjust.

[0059] The dynamic assessment also includes the assessment of behavioral and emotional characteristics, which specifically includes the following assessment contents: obtaining all behavioral data of autistic children in the process of continuous interaction with autistic children as comprehensive behavioral data; evaluating the emotional stability of autistic children when dealing with difficulties or pressure by comprehensively analyzing the changes in visual focus, eye movement trajectories and facial expressions of autistic children in response to different interactive tasks in the comprehensive visual data; evaluating the behavioral stability and concentration of autistic children in the same situation by comprehensively analyzing the body movements, head movements and gaze behaviors of autistic children in the same interactive tasks in the comprehensive behavioral data.

[0060] That is, behavioral and emotional characteristics are evaluated from the following two aspects: (1) Emotional stability: By analyzing the emotional fluctuations and facial expressions of children with autism during the task, their emotional stability when dealing with difficulties or stress is evaluated. The evaluation will combine facial expressions, voice intonation, and eye movement data to fully reflect the emotional response ability of children with autism.

[0061] (2) Behavioral response consistency: By analyzing the consistency of autistic children’s behavioral responses in different tasks, we evaluate whether they can maintain stable performance in similar situations. The evaluation model will focus on the limb movements, head movements, and gaze behaviors of autistic children, and conduct a comprehensive evaluation based on spatial positioning and posture tracking data.

[0062] In some embodiments, when evaluating the language ability of autistic children, a comprehensive evaluation can be conducted by combining the contents of standardized evaluation and dynamic evaluation. At this time, not only the evaluation results of the individual abilities of autistic children are taken into account, but also the overall performance, consistency and stability of autistic children are combined to ensure the comprehensiveness and accuracy of the evaluation results. The comprehensive evaluation of this embodiment includes the following main indicators: speech feature evaluation, natural language content evaluation, eye movement and behavior feature evaluation, task completion evaluation, language learning progress evaluation, and emotional and behavioral response evaluation.

[0063] In one example, a block evaluation method combined with ensemble learning is used to effectively handle incomplete evaluation data and utilize information from different data blocks to improve the reliability of the evaluation. Specifically, the evaluation data is divided into two main data blocks according to its availability: Block A: Contains speech feature evaluation, natural language content evaluation, and eye movement and behavioral feature evaluation. Block B: Contains task completion evaluation, language learning progress evaluation, and emotional and behavioral response evaluation. On each data block, one or more machine learning models are trained to predict their respective evaluations. These models will be evaluated for performance through methods such as cross-validation to ensure their best performance on their respective data blocks.

[0064] The model output results on each data block will be integrated through weighted averaging to generate the final comprehensive evaluation. The weight of each model (such as 0.6 and 0.4) is assigned according to its performance on the validation set. The specific steps are as follows: Model performance calculation: Calculate the performance indicators of the model on data block A and data block B respectively (such as accuracy, mean square error, etc.).

[0065] Weight distribution: Assume that the performance of model M1 on data block A is P1, and the performance of model M2 on data block B is P2. The weight can be calculated according to the following formula: the weight of model 1 is W1=P1 / (P1+P2), and the weight of model 2 is W2=P2 / (P1+P2). In this way, the model with better performance will get a higher weight.

[0066] Integrated evaluation: The comprehensive evaluation is obtained by weighted average of the model outputs of data block A and data block B: comprehensive evaluation = output of W1×M1 + output of W2×M2.

[0067] In order to further improve the accuracy of the evaluation, a continuous optimization mechanism was introduced: Model fine-tuning: According to the performance of the comprehensive evaluation, the model of each data block is fine-tuned as necessary. Weight adjustment: Through the feedback of new data, the weight of each model is dynamically adjusted to ensure the scientificity and applicability of the comprehensive evaluation. This evaluation method can not only make full use of existing data, but also maximize the reflection of children's language ability and progress when the data is incomplete. Through block evaluation and integrated learning, the evaluation system can adapt to different evaluation scenarios and provide more accurate and fair results.

[0068] In some embodiments, after the evaluation is completed, a detailed evaluation report will be automatically generated. The report content includes a comprehensive score, detailed scores of each indicator, performance analysis of autistic children, and personalized recommendations based on the evaluation results. The report will be in the form of a combination of charts and text descriptions to help parents and professionals clearly understand the evaluation results.

[0069] It is understandable that the language assessment method for autistic children of the present invention is not only suitable for the language ability assessment of autistic children, but can also be extended to the assessment and intervention of other language disorders and children with neurodevelopmental disorders. By adjusting the assessment parameters and data analysis models, the method can adapt to different types of assessment needs and provide extensive and accurate assessment services. For example, in an educational environment, the method can provide strong support for special education teachers and help formulate personalized education plans; in a treatment environment, the assessment results can provide a scientific basis for speech therapists and psychologists to optimize treatment plans; it can also provide scientific researchers with rich data resources to support in-depth research on neurodevelopmental disorders, which can not only be used to analyze the language development patterns of autistic children, but also help explore the relationship with other neurodevelopmental disorders (such as attention deficit hyperactivity disorder, language retardation, etc.).

[0070] It should also be noted that the present invention can be implemented through more common technical equipment in scenarios with limited resources, such as using cameras and microphones instead of complex physiological data acquisition devices to obtain multimodal response data of children with autism, and replacing deep learning algorithms with traditional machine learning models to extract multimodal response features of children with autism from the multimodal response data. At this time, a high degree of accuracy can still be achieved.

[0071] The step division of the various methods above is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of the present invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process without changing the core design of the algorithm and process are all within the protection scope of the invention.

[0072] Another embodiment of the present invention relates to a computer device, such as Figure 2 As shown, it includes: at least one processor 201; and a memory 202 that is communicatively connected to the at least one processor; wherein the memory 202 stores instructions that can be executed by the at least one processor 201, and the instructions are executed by the at least one processor 201 so that the at least one processor 201 can execute the language assessment method for autistic children in the above-mentioned embodiments.

[0073] Among them, the memory and the processor are connected in a bus manner, and the bus may include any number of interconnected buses and bridges, and the bus connects various circuits of one or more processors and memories together. The bus can also connect various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the transceiver. The transceiver can be one element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices on a transmission medium. The data processed by the processor is transmitted on a wireless medium via an antenna, and further, the antenna also receives data and transmits the data to the processor.

[0074] The processor is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. Memory can be used to store data used by the processor when performing operations.

[0075] Another embodiment of the present invention relates to a computer-readable storage medium storing a computer program, which implements the above method embodiment when executed by a processor.

[0076] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a program, and the program is stored in a storage medium, including a number of instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as: ROM), random access memory (Random Access Memory, referred to as: RAM), disk or optical disk and other media that can store program codes.

[0077] Those skilled in the art will appreciate that the above embodiments are specific embodiments for implementing the present invention, and in actual applications, various changes may be made thereto in form and detail without departing from the spirit and scope of the present invention.

Claims

1. A language assessment method for children with autism, characterized in that: include: Acquiring multimodal response data of the autistic child when interacting with the autistic child, wherein the multimodal response data includes language data, visual data, and behavioral data; Extract pitch, tone, intensity, and duration of pronunciation from language data, as well as extract vocabulary, sentence structure, and semantic information; extract visual focus, eye movement trajectories, and facial expressions from visual data; extract body movements, head movements, and gaze behaviors from behavioral data to obtain multimodal response characteristics of children with autism; According to the multimodal response characteristics of autistic children, generate interactive tasks that are consistent with the current language ability of autistic children, so as to continue to interact with the autistic children through the generated interactive tasks; All language data of autistic children during multiple rounds of iterative interactions with them are obtained as comprehensive language data, and the language ability of autistic children is evaluated through the comprehensive language data.

2. The method for language assessment of autistic children according to claim 1, characterized in that: The language ability of autistic children is evaluated through comprehensive language data, including: Based on the average pitch, pitch variation, and pitch stability in the comprehensive language data, the tone control and tone variation abilities of children with autism were evaluated. Based on the frequency fluctuation of pitch in the comprehensive language data, we evaluate whether children with autism can maintain stable pitch control under different emotions and in different expressions; By comprehensively analyzing the changes in sound intensity in language data and the distribution of sound intensity in different sentences and words, the sound control ability of children with autism in expression is evaluated; The pronunciation fluency and rhythm of children with autism were evaluated by measuring the duration of pronunciation of each sentence or word in the comprehensive language data; According to the accuracy of the words used by autistic children in the comprehensive language data, the vocabulary reserve and vocabulary application ability of autistic children are evaluated; The language organization ability and grammatical mastery level of autistic children were assessed based on the complexity of the sentence structures used by autistic children in the comprehensive language data; By comprehensively comparing the semantic similarity between the responses of autistic children to interactive tasks in language data and the preset standard responses, we evaluate whether autistic children can accurately convey their intentions in their expressions.

3. The method for language assessment of autistic children according to claim 1, characterized in that: The language ability of autistic children is evaluated through comprehensive language data, including: According to the response content of autistic children to each interactive task in the comprehensive language data and the corresponding preset standard response content, determine the proportion of the number of interactive tasks to which autistic children correctly respond to the total number of interactive tasks, so as to evaluate the accuracy of the language expression of autistic children when faced with new vocabulary or new sentence patterns; The average reaction time of autistic children in responding to interactive tasks in the comprehensive language data was used to evaluate the reaction speed of autistic children in language comprehension and expression; According to the vocabulary used by autistic children in the interaction process in the comprehensive language data, determine the proportion of the number of new vocabulary correctly used by autistic children to the total number of new vocabulary, so as to evaluate the efficiency of autistic children in learning new vocabulary; Based on the sentence structures used by autistic children during the interaction in the comprehensive language data, determine the number of new sentence structures correctly used by autistic children to assess the ability of autistic children to learn new sentence structures; Based on the responses of autistic children to interactive tasks in the comprehensive language data and the preset standard responses, the proportion of successfully corrected errors to the total number of errors made by autistic children after they made incorrect responses to interactive tasks was determined to assess the ability of autistic children to cope with errors.

4. The method for evaluating language of autistic children according to any one of claims 1 to 3, characterized in that: After evaluating the language ability of autistic children through comprehensive language data, it also includes: Acquire all visual data of the autistic child during the continuous interaction with the autistic child as comprehensive visual data; By integrating the frequency, duration and eye movement trajectory of visual focus changes in visual data, the degree of attention of autistic children during the interaction was evaluated; The frequency, intensity, and duration of facial expression changes in comprehensive visual data were used to assess the emotional state of children with autism during interaction.

5. The method for language assessment of autistic children according to claim 4, characterized in that: After evaluating the language ability of autistic children through comprehensive language data, it also includes: Acquire all behavioral data of the autistic child during the continuous interaction with the autistic child as comprehensive behavioral data; By integrating the body movements, head movements and gaze behaviors of autistic children in interactive tasks with the same context in the comprehensive behavioral data, the behavioral stability and concentration of autistic children in the same context are evaluated.

6. The method for language assessment of autistic children according to claim 1, characterized in that: The method of obtaining multimodal response data of autistic children when interacting with the autistic children includes: Virtual characters interact with autistic children in virtual scenes, and obtain multimodal response data of autistic children during the interaction.

7. The method for language assessment of autistic children according to claim 1, characterized in that: The method of generating interactive tasks that are consistent with the current language abilities of autistic children based on the multimodal response characteristics of the autistic children includes: Converting the multimodal response features of autistic children into prompt words that conform to the input format of the large model, wherein the prompt words include feature descriptions, content summaries, and context information of the multimodal response features; Generate interactive tasks that match the current language abilities of children with autism through a large model.

8. The method for language assessment of autistic children according to claim 1, characterized in that: The multimodal response data also includes electroencephalography and functional magnetic resonance imaging.

9. A computer device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the language assessment method for autistic children as described in any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the language assessment method for autistic children according to any one of claims 1 to 8 is implemented.

Citation Information

Patent Citations

  • Autism rehabilitation training and ability evaluation system and method based on virtual reality

    CN110890140A

  • Autism virtual teaching system, method and equipment based on virtual reality technology

    CN110930780A

  • Virtual immersion type autistic child treatment system based on electroencephalogram signals and eye tracker

    CN113082448A

  • Social communication function training method for children with autism spectrum disorder

    CN113284625A

  • Autism assessment system, electronic device, and computer readable storage medium

    CN118039151A

Cited By

  • Autistic child intervention treatment auxiliary system and method based on intelligent interaction

    CN119541839A

  • Autism Children Intervention and Treatment Assistance System and Method Based on Intelligent Interaction

    CN119541839B

  • Construction method of autism spectrum disorder recognition model based on deep learning

    CN119993467A

  • A Method for Constructing a Deep Learning-Based Autism Spectrum Disorder Identification Model

    CN119993467B

  • Multi-modal monitoring method for autistic children

    CN120234749A