Children autism social training intervention system based on large model and VB-MAPP

Abstract

The invention discloses a children autism social training intervention system based on a large model and a VB-MAPP. The children autism social training intervention system comprises a sensing module, an expression module and a control module. The sensing module is used for collecting testee information, including speaking content and audio; the presentation module is used for generating corresponding feedback content according to the information of the subject and the configuration of the presentation module; the control module is used for controlling the working process of the whole system and completing synchronization and recording; the presentation module constructs a system cue word based on a VB-MAPP normal form, and performs dialogue generation according to the system cue word by using a large model; the system cue word comprises three parts, wherein the first part is personal information of a subject; the second part is that the current large model needs to play a virtual image and comprises role description and capability limitation; and the third part is description of the current dialogue theme. According to the method, personalized dialogue contents can be generated for different autism children, so that the dialogue richness is higher, the attention of the children is more easily attracted, and the intervention effect is improved.

Description

Technical Field

[0001] The invention relates to the field of clinical intervention for children with autism, and in particular to a social training intervention system for children with autism based on a large model and VB-MAPP. Background Art

[0002] Autism spectrum disorder is a widespread neurodevelopmental disorder, the main symptoms of which are social disorders and repetitive, stereotyped behaviors. Other symptoms may also exist, such as emotional regulation, language and non-verbal communication disorders. Autistic patients are unable to take care of themselves and cannot integrate into social life normally, which places a great burden on their families and society. Reports show that the prevalence of autism among school-age children worldwide is as high as 1 / 59, and is on the rise. Early diagnosis and effective intervention treatment can improve the language, cognitive abilities and behavioral habits of autistic children to a certain extent, so how to conduct efficient autism diagnosis and intervention has become the focus.

[0003] Due to the insufficiency and imbalance of medical resources, intervention resources in some areas are very limited, so researchers hope to use computer technology to assist professional interventionists in intervention or develop new systems and paradigms to complete independent intervention, so that more children with autism can receive intervention support. The current mainstream research usually uses AR and robots as carriers, and guides autistic children to interact through manually designed intervention paradigms, and then collects indicators to evaluate the intervention effect.

[0004] Such as the document "Social Intervention Strategy of Augmented Reality Combinedwith Theater-Based Games to Improve the Performance of Autistic Children inSymbolic Play and Social Skills." and the document "Effectiveness of a Robot-AssistedPsychological Intervention for Children with Autism Spectrum Disorder."

[0005] However, the current methods lack a certain degree of professionalism. The design of the paradigm method is not based on clinical assessment and intervention methods, and the intervention form lacks personalized content for subjects with different themes. Summary of the invention

[0006] The present invention provides a social training intervention system for children with autism based on a large model and VB-MAPP, which can generate personalized dialogue content for different children with autism, making the dialogue richer, easier to attract children's attention, and improving the intervention effect.

[0007] A social training intervention system for children with autism based on a large model and VB-MAPP, comprising a perception module, a performance module and a control module; the perception module is used to collect information about the subject, including the content and audio of the speech; the performance module is used to generate corresponding feedback content according to the subject's information and its own configuration; the control module is used to control the workflow of the entire system and complete synchronization and recording;

[0008] The performance module constructs system prompt words based on the VB-MAPP paradigm, and uses the big model to generate dialogue according to the system prompt words; the system prompt words include three parts, the first part is the personal information of the subject; the second part is the description of the virtual image that the current big model needs to play, including role description and ability limitations; the third part is the description of the current dialogue topic;

[0009] The systematic intervention process is divided into the preparation phase, intervention phase and collection phase, as follows:

[0010] In the preparation stage, the personal information of the test child is input, and the intervention stage is officially started after the virtual image is selected; in the intervention stage, the performance module combines the child's personal information, the virtual image of the large model and the current topic to construct the system prompt words, and inputs them to the large model, so that the large model can start limited multi-round dialogue intervention with the autistic child under the limited topic; the collection stage runs through the entire intervention stage. After the intervention on each topic is completed, the system uses the control module to save the near-infrared, text, and audio data collected by the perception module for evaluating the child's performance and the effectiveness of the system.

[0011] VB-MAPP (Verbal Behavior Milestones Assessment and Placement Program) is a widely used assessment tool that divides items and assessment scores according to the development level of the subjects and provides guidance for intervention. The form of turn-taking dialogue is an effective form of intervention recommended by VB-MAPP and is also one of the task forms that the large model is good at. Therefore, based on the guidelines provided by VB-MAPP and the advice of practicing therapists, the present invention designs a communication paradigm with a predetermined theme. The present invention customizes five topics commonly used in daily communication: food, animals, toys, family and colors.

[0012] During the conversation, the form of questions is limited to three types of questions: "who", "what" and "where", while questions of the form of "why", "how" and "when" are excluded. The present invention takes into account that the first three types of questions are related to specific memory images, which are relatively easy for subjects with autistic language disorders to answer; while "why, how and when" involve logical thinking, abstract generalization and time perception ability, and the questions are more complicated. The thinking and answer willingness of children with autism will decrease, and the intervention effect will not be achieved.

[0013] In the present invention, the large model may adopt Chat-GPT or other large language models.

[0014] Furthermore, in the first part of the system prompt words, the subject's personal information includes the subject's age, gender, nickname, preferences, interests and recent experiences, and the content of the conversation intervention topic is personalized based on this personal information, thereby increasing children's enthusiasm for interaction.

[0015] During the intervention phase, the large model began to conduct limited multi-round dialogue interventions with autistic children under limited topics, specifically:

[0016] The expression module asks questions after greeting the autistic child, and then gives different feedback based on the child's response. When the child does not respond, additional prompt words are input to allow the expression module to encourage the child to speak. When the child speaks, praise is given and a new round of conversation begins until the preset time is up.

[0017] In the intervention phase, a fixed prompt word was added before each round of dialogue to strengthen the limited dialogue generation based on the VB-MAPP paradigm.

[0018] During the intervention phase, after the conversation time is up, an ending cue word will be input to allow the performance module to say goodbye to the autistic child.

[0019] During the intervention phase, in order to deal with the problem of autistic children not responding, the system uses sound detection to determine whether they are speaking. If there is silence for more than ten seconds, it will trigger a pre-set encouraging voice to encourage the child to speak.

[0020] During the collection phase, the text and audio of the conversation intervention were automatically saved by the system, and the near-infrared signal was collected by an 8-channel NirSmart system with a time resolution of 0.1 seconds, and the electrodes were located in the prefrontal cortex.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention designs the entire system paradigm and prompts based on VB-MAPP, one of the current clinical autism intervention assessment methods, and uses a large model as the system's dialogue generation center to improve the system's dialogue quality during the intervention phase. The proposed system has strong dialogue capabilities and supports personalized dialogue content customization. Through multimodal clinical data analysis, the effectiveness of the system's dialogue intervention has been proven, which can alleviate the problem of scarce resources for autistic children's intervention to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a schematic diagram of system prompt words based on VB-MAPP in an embodiment of the present invention;

[0024] Figure 2 is an intervention flow chart of the system of the present invention;

[0025] Figure 3 This is a diagram of the analysis results of the conversation text during the intervention process;

[0026] Figure 4 This is a diagram showing the analysis results of physiological signals during the intervention process. DETAILED DESCRIPTION

[0027] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be pointed out that the embodiments described below are intended to facilitate the understanding of the present invention and do not have any limiting effect on the present invention.

[0028] In order to verify the effectiveness of the designed paradigm and system, the present invention designs experiments and collects data based on the following assumptions: If the system can perform effective social intervention, then some data indicators of children's dialogue with the system should be similar to those of professional autism interventionists. Therefore, the present invention conducted experiments on 12 children diagnosed with autism, and the inclusion criteria of the experiment were that their therapists confirmed that the autistic children had the ability to have multiple rounds of dialogue. The present invention allowed the subjects to have a dialogue with the proposed system, and the dialogue revolved around five topics designed based on VB-MAPP, and the time limit for each topic was 2 minutes. Before the dialogue, the preferences and personalized data of the subjects' parents, such as favorite food or recent activities, were collected and added to the system prompt as personalized content, so that the dialogue was targeted and personalized, and it could also attract children's attention and improve the intervention effect. In addition, the present invention allowed professional interventionists to conduct dialogue interventions with these subjects for the same topics and restrictions, and the total duration was kept within 15 minutes. The effectiveness of the system can be proved by data analysis of near infrared, dialogue text, and dialogue audio during the intervention.

[0029] A social training intervention system for children with autism based on a large model and VB-MAPP. The specific implementation process is as follows:

[0030] (1) Implementation based on VB-MAPP paradigm: The paradigm is specifically applied to the construction of system prompts, such as Figure 1 The prompt is divided into three parts. The first part is the personal information of the subject, which mainly introduces the age, gender, and nickname of the subject. info It indicates the personalized information of children, such as "his favorite food is dried tofu, and he asks an aunt to cook for him; he doesn't like small animals very much and has never raised any; he has a large family, including his parents, grandparents, and younger brother, whose nickname is Jiabao, and everyone plays with him; his favorite toys are cars, and he has many, including sedans, fire trucks, etc.; his favorite color is red, and he has a red water cup and a red car toy." The second part is the role description and ability restrictions, which are used to standardize the content and restrictions generated by Chat-GPT dialogues; the third part is a description of the topic of the dialogue. topic Describe specific topic information such as food: "The topic of the conversation is food. Please develop your ideas around the topic. Here are some references: staple food, snacks, drinks, desserts and snacks."

[0031] (2) The intervention process can be divided into three parts, such as Figure 2 As shown. In the preparation stage, an interface will be used for information input and system preference selection. According to the information provided by parents and professional interventionists, the basic information and personal preferences of the child will be input, and the intervention stage will be officially entered after the virtual image is selected. In the intervention stage, the system will input the system prompt with the personalized information of the subject into Chat-GPT, so that it can play the role of an intervention therapist in multiple rounds of dialogue. The specific process is: the system will ask questions after greeting the autistic child, and then give different feedback according to the child's response. When the child does not respond, the pre-made recording will be played to encourage the child to speak. When the child speaks, praise will be expressed, and then a new round of dialogue will begin until the preset time (2 minutes) of a single topic is exhausted. The collection stage runs through the entire intervention stage. After the intervention of the five topics is completed, the system will save the collected near-infrared, text, and audio data for evaluating the child's performance and the effectiveness of the system. The text and recording of the dialogue intervention are automatically saved by the system, and the near-infrared signal is collected by the 8-channel NirSmart system with a time resolution of 0.1 seconds and the electrode is located in the vmPFC.

[0032] (3) Implementation details of the child autism social training intervention system of the present invention: When the system conducts each round of dialogue (one question and one answer) with the autistic child, in order to enhance the context, an idle prompt will be added before the start of each round of dialogue to strengthen the dialogue generation based on the VB-MAPP paradigm design restrictions: "Continue to communicate around the topic and the content he mentioned. Only ask one question at a time, such as what, who, where, do not involve why, how, when. Do not explain. If his speech is not clear, you can ask him to repeat it." After the dialogue time is up, there will be an endprompt input for ending, allowing the system to say goodbye to the autistic child: "The communication time is over, respond to my words, summarize our communication, and say goodbye to me." At the same time, in order to deal with the problem of autistic children not responding, the system uses a sound detection method to determine whether they have spoken. If there is silence for more than ten seconds, it will automatically generate "(I didn't speak)" as a response, and will trigger a pre-set encouraging voice to encourage the child to speak.

[0033] The system is written in python. Chat-GPT (gpt3.5-turbo-0125) is used as the conversation generation hub and is called through an API. Since Chat-GPT does not have the functions of voice-to-text and text-to-speech, the present invention uses the real-time speech recognition function of the iFLYTEK open platform to convert speech into text as a user prompt, and combines the system prompt input to Chat-GPT to complete the generation of conversation text, and then converts the text into speech through Microsoft's Azure service for output, and the voice role is selected as "zh-CN-YunxiNeural". When speech recognition is unsuccessful, "(unrecognizable speech, duration of about ${second} seconds.)" will be used as the filler of the user prompt, so that the system can reasonably respond to the speech content that cannot be correctly recognized.

[0034] (4) System effectiveness evaluation and result analysis: The experiment collected 12 autistic children aged 4-11 (11 boys and 1 girl). The intervention conversations between professional interventionists and autistic children were recorded by a voice recorder (XXJ1, HP Inc.), and text transcription was performed using paraformer and speaker recognition annotation was performed using cam++. To ensure the accuracy of the transcribed text, manual calibration and correction were performed based on the recognition results of the model. During the preprocessing process, the speech fragments were separated from the audio and outliers were discarded. The near-infrared signal first converted the raw intensity data into optical density (OD) changes, and then motion artifacts were detected and sample interpolation was performed. An FIR filter (0.01–0.2Hz) was used to extract low-frequency fluctuations, and the OD data was converted into hemoglobin concentration changes using the improved Beer-Lambert law.

[0035] The analysis of the text is as follows Figure 3 As shown in Figure 2. Engagement reflects the level of children's participation during the intervention. The higher the engagement, the more focused the child is. It indirectly reflects that ASD-Chat is able to attract children's attention and achieve the expected intervention effect. Engagement can be measured by text length and audio duration. We collected data on the average number of words (excluding punctuation) and average duration of each child's sentence in the five subjects. Figure 4 It can be seen that when facing ASD-Chat, the children spoke more words and for longer periods of time than when interacting with professional interventionists. On average, the number of words spoken when interacting with ASD-Chat was 13.11% higher and the time was 43.03% longer than when interacting with professional interventionists. One possible explanation is that it is easier for children to communicate with virtual cartoon characters than with adults. This is because they are used to having similar conversations with toys in their daily lives. Communicating with virtual characters may put them in a more relaxed state, thereby increasing their enthusiasm and participation in the conversation. The quality of response reflects whether children can provide appropriate responses rather than illogical nonsense. This item is measured by semantic similarity. The results show that in terms of response quality, professional interventionists performed slightly better, and the system scored 11.31% lower. The interventionists' rich clinical intervention experience enables them to guide children to provide appropriate responses through language and actions. However, the current system lacks this ability and still needs to be improved in this regard.

[0036] The analysis of the audio is shown in Table 1 below.

[0037] Table 1

[0038]

[0039] At the audio level, the analysis focuses on the subject's speech. The voiced segments are classified with reference to the frequency pattern of the formants. The fundamental frequency (F0) and the zero crossing rate (ZCR) can represent the speech characteristics of a specific individual to a certain extent, as well as the arousal level of a specific individual.

[0040] Table 1 lists the average F0 and ZCR of speech and voiced sounds, as well as the first three formant frequencies of voiced sounds. F1 is related to the degree of opening of the vowel, F2 is related to the front and back position of the tongue, and F3 is related to the tip of the tongue and the shape of the lips. Statistically, due to the different frequencies of various voiced sounds in the comparative experiment, there are differences in the mean values ​​of F0, ZCR, F1, F2, and F3. There are similar situations in speech. It can be seen that in these indicators, the performance of the same subject on the interventionist and the proposed system is close. Specifically, in the ASD-Chat system experiment, the speech F0 is 2% higher than the interventionist, the speech ZCR is 48%, and the voiced ZCR is 40%. In terms of voiced indicators, F0 is 12% higher, F1 is 11% lower, F2 is 5% higher, and F3 is 2% higher. Overall, they are very close, which verifies the effectiveness of the system.

[0041] For near-infrared data, the oxygenated hemoglobin (HbO) data of the first 1 second was discarded for steady-state control and normalized using the z-score method to eliminate the impact of data units and facilitate comparison under different conditions. Finally, the average Hbo change of each channel for each subject under five dialogue topics in the two task scenarios was calculated to verify the effectiveness of the system.

[0042] The present invention uses HbO amplitude induced by the same subject to illustrate the similarities between the ASD-Chat system and real clinical intervention scenarios. Blood oxygen amplitude is particularly related to social and communication characteristics, representing the core symptoms of autism spectrum disorder (ASD). Figure 4 As shown, the present invention calculates the difference in HbO amplitude for each channel of all autistic children in five topic conversations in two scenarios. Compared with the clinician intervention scenario, the HbO amplitude changes of half of the channels in the ASD-Chat scenario were greater, while the brain activation intensity of four channels was weaker. It is worth noting that although the conversations in the intervention scenario led to relatively stronger fNIRS signal changes on some channels, the difference range between the two was only about 0.1. The difference between the two is almost negligible, which means that there is almost no difference in the HbO amplitude changes caused by the two scenarios. This proves that the ASD-Chat intervention system proposed in the present invention has similar effects to traditional intervention methods. It is expected to replace traditional intervention programs and save resources.

[0043] The embodiments described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A social training intervention system for children with autism based on a large model and VB-MAPP, characterized by: It includes a perception module, a performance module and a control module; the perception module is used to collect information from the subject, including the content and audio of the speech; the performance module is used to generate corresponding feedback content according to the subject's information and its own configuration; the control module is used to control the workflow of the entire system and complete synchronization and recording; The performance module constructs system prompt words based on the VB-MAPP paradigm, and uses the big model to generate dialogue according to the system prompt words; the system prompt words include three parts, the first part is the personal information of the subject; the second part is the virtual image that the current big model needs to play, including role description and ability limitations; the third part is the description of the current dialogue topic; The systematic intervention process is divided into the preparation phase, intervention phase and collection phase, as follows: In the preparation stage, the personal information of the test child is input, and the intervention stage is officially started after the avatar is selected; in the intervention stage, the performance module combines the child's personal information, the avatar of the large model, and the current topic to construct the system prompt words, which are input to the large model, so that the large model can start a limited multi-round dialogue intervention with the autistic child under the limited topic; The collection phase runs throughout the entire intervention phase. After the intervention for each subject is completed, the system uses the control module to save the near-infrared, text, and audio data collected by the perception module for evaluating the children's performance and the effectiveness of the system.

2. The social training intervention system for children with autism based on a large model and VB-MAPP according to claim 1 is characterized in that: In the first part of the system prompts, the subject's personal information includes the subject's age, gender, nickname, preferences, interests and recent experiences. The content of the conversation intervention topic is personalized based on this personal information, thereby increasing children's enthusiasm for interaction.

3. The social training intervention system for children with autism based on a large model and VB-MAPP according to claim 1 is characterized in that: In the third part of the system prompt words, themes include food, animals, toys, family and colors.

4. The social training intervention system for children with autism based on a large model and VB-MAPP according to claim 1, characterized in that: During the intervention phase, the large model began to conduct limited multi-round dialogue interventions with autistic children under limited topics, specifically: The expression module asks questions after greeting the autistic child, and then gives different feedback based on the child's response. When the child does not respond, additional prompt words are input to allow the expression module to encourage the child to speak. When the child speaks, praise is given and a new round of conversation begins until the preset time is up.

5. The social training intervention system for children with autism based on a large model and VB-MAPP according to claim 1, characterized in that: In the intervention phase, a fixed prompt word was added before each round of dialogue to strengthen the limited dialogue generation based on the VB-MAPP paradigm.

6. The social training intervention system for children with autism based on a large model and VB-MAPP according to claim 5, characterized in that: During the intervention phase, after the conversation time is up, an ending cue word will be input to allow the performance module to say goodbye to the autistic child.

7. The social training intervention system for children with autism based on a large model and VB-MAPP according to claim 1, characterized in that: During the intervention phase, in order to deal with the problem of autistic children not responding, the system uses sound detection to determine whether they are speaking. If there is silence for more than ten seconds, it will trigger a pre-set encouraging voice to encourage the child to speak.

8. The social training intervention system for children with autism based on a large model and VB-MAPP according to claim 1, characterized in that: During the collection phase, the text and audio of the conversation intervention were automatically saved by the system, and the near-infrared signal was collected by an 8-channel NirSmart system with a time resolution of 0.1 seconds, and the electrodes were located in the prefrontal cortex.

Images