Active intervention training method and system for autistic children

Abstract

The invention relates to the technical field of infantile autism intervention training, and discloses an active intervention training method and system for infantile autism children, and the method comprises the following steps: employing a visual music intervention method to guide the attention of the infantile autism children; guiding the autistic child to review the content of the previous intervention training; guiding the autistic children to learn and consolidate social gestures; and guiding the autistic children to learn and consolidate emotional expression. The robot has multiple intervention training means, the requirement for different children with the autism is met, in the intervention training process, the robot can autonomously interact with the children with the autism, and the emotion of the children with the autism is fully aroused.

Description

Technical Field

[0001] The present invention relates to the technical field of autism intervention training, and in particular to an active intervention training method and system for autistic children. Background Art

[0002] With the development of science and technology, robots have achieved remarkable results in the diagnosis, rehabilitation training, treatment and education of children with autism (ASD). For example, CN108145717A discloses an autism rehabilitation training system based on robot interaction, and CN107283389B discloses a robot for assisting the treatment of autism. At present, the robot-based intervention training technology for children with autism can effectively carry out generalization training, but there are still the following shortcomings: first, the intervention training means are single and cannot satisfy different individual children with autism; second, the robot conducts intervention training for children with autism according to the established procedures, and lacks autonomous interaction with children with autism during the training process, which has a poor effect on mobilizing the emotions of children with autism; third, the established mode of the robot in the past can easily cause children to repeat stereotyped behaviors, leading to a vicious cycle of intervention training. Summary of the invention

[0003] The purpose of the present invention is to provide an active intervention training method and system for autistic children to solve the deficiencies in the above-mentioned prior art.

[0004] The present invention is achieved through the following technical solutions:

[0005] On the one hand, an active intervention training method for autistic children is provided, comprising the following steps: S11: using a visual music intervention method to guide the attention of the autistic child; S12: judging whether the autistic child has responded;

[0006] S13: If no response is made, return to S11; if a response is made, execute S14: S14: Use sensory stimulation method to reward the autistic child; S21: Guide the autistic child to review the content of the previous intervention training; S22: Execute the operation described in S12; S23: If no response is made, return to S21; if a response is made, execute the operation described in S14; S31: Guide the autistic child to learn and consolidate social gestures; S32: Execute the operation described in S12; S33: If no response is made, return to S31; if a response is made, execute the operation described in S14; S41: Guide the autistic child to learn and consolidate emotional expression; S42: Execute the operation described in S12; S43: If no response is made, return to S41; if a response is made, execute the operation described in S14.

[0007] Furthermore, the active intervention training method is based on a robot; the robot includes: a video monitoring module, which is used to monitor the face and limbs of the autistic child in real time and generate a monitoring video; a face recognition module, which is used to simulate the shape of the eyeball and the eyelid according to the monitoring video using 3D skeleton geometric modeling technology, and use a stereoscopic vision system to capture the stereoscopic movement of the eyeball and identify the activity state of the eyeball to obtain a first recognition result, and use the stereoscopic vision system to identify changes in facial expressions to obtain a second recognition result; a behavior recognition module, which is used to identify human behavior actions in the monitoring video frame by frame to obtain a third recognition result; a speech recognition module, which is used to identify the sound made by the autistic child to obtain a fourth recognition result; the robot also includes: an audio playback module, a light display module, an LED display module, a first drive module and a second drive module.

[0008] Furthermore, before S11, it also includes: S10.1: collecting the first recognition result in real time; S10.2: if the first recognition result is moving the eyeball or looking at the robot, executing S11: if the first recognition result is not moving the eyeball and not looking at the robot, returning to S10.1.

[0009] Furthermore, the S11 specifically includes: controlling the audio playback module to play music, controlling the light display module to display lights, or controlling the first driving module to drive the robot to perform dance.

[0010] Furthermore, the S12 specifically includes: collecting the second recognition result, the third recognition result and the fourth recognition result in real time, and judging whether the autistic child responds according to the second recognition result, the third recognition result and the fourth recognition result.

[0011] Furthermore, S14 specifically includes: rewarding the autistic child using the method described in S11.

[0012] Furthermore, the S21 specifically includes: controlling the second driving module to drive the remaining modules of the robot to work according to the content of the previous intervention training.

[0013] Furthermore, the S31 includes: S31.1: controlling the first driving module to drive the audio playback module to play the first audio, and at the same time driving the robot's arm to simulate the social gesture corresponding to the first audio; S31.2: executing the operation described in S12; S31.3: if no response is made, returning to S31.1; if a response is made, executing S31.4; S31.4: controlling the first driving module to drive the audio playback module to play the second audio.

[0014] Furthermore, the S41 includes: S41.1: controlling the first driving module to drive the audio playback module to play the third audio and the fourth audio, and at the same time driving the robot's arm to simulate social gestures corresponding to the third audio or the fourth audio, and driving the LED display module to display emoticons corresponding to the third audio or the fourth audio; S41.2: executing the operations described in S12; S41.3: if no response is made, returning to S41.1; if a response is made, executing S41.4; S41.4: executing the operations described in S31.4.

[0015] Furthermore, after S43, the following steps are also included: summarizing and storing the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire intervention training process; obtaining first statistical data based on the summarized first recognition result, and storing the first statistical result; the first statistical data includes: the number of times the autistic child looks at the robot and the length of time the child looks at the robot; obtaining second statistical data based on the summarized third recognition result, and storing the second statistical result; the second statistical data includes: the number of times the autistic child performs behavioral actions and the accuracy of the behavior; based on the first statistical data and the second statistical data, analyzing the interaction, course completion and intervention training progress of the autistic child, and storing the analysis results; storing the content of the current intervention training.

[0016] Furthermore, the active intervention training method also includes the following steps: guiding parents or institutional training personnel to simulate the real scene together with the robot by jointly executing S31 to S43; summarizing and storing the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire simulation display scene.

[0017] Furthermore, the face recognition module includes: an eyeball positioning recognition unit, which is used to simulate the shape of the eyeball and the eyelid shape according to the monitoring video using 3D skeleton geometry modeling technology, and to capture the stereoscopic movement of the eyeball and identify the activity state of the eyeball using a stereoscopic vision system to obtain the first recognition result; a facial expression recognition unit, which is used to identify facial expression changes according to the monitoring video using a stereoscopic vision system to obtain the second recognition result.

[0018] Furthermore, the speech recognition module includes: an audio script storage unit, used to store the music, the first audio, the second audio, the third audio and the fourth audio.

[0019] Furthermore, the robot also includes: a two-wheel differential movement mechanism and a Beidou module.

[0020] Furthermore, the body part of the robot is gray, and the LED display module is a light blue LED display screen.

[0021] Furthermore, a servo is installed at the connection between the robot's arm and the side of the body; a transmission mechanism and a servo are installed on the robot's neck, the servo located on the neck is fixed on a cylindrical structure inside the neck, the end of the cylindrical structure is fixed on a bearing, the length of the cylindrical structure is greater than the length of the neck and extends deep into the head of the robot.

[0022] In a second aspect, an active intervention training system for autistic children is provided, comprising a remote control terminal and the robot described in the first aspect; the remote control terminal comprises: an intervention training guidance module, used to guide the attention of the autistic child by using a visual music intervention method; a feedback state recognition module, used to determine whether the autistic child has responded; a first logic control module, used to control the intervention guidance module and the feedback recognition module to work when the autistic child has not responded, and to control the sensory stimulation module to work when the autistic child has responded; a sensory stimulation reward module, used to reward the autistic child by using a sensory stimulation method; an intervention training review module, used to guide the autistic child to review the content of the previous intervention training; a first function calling module, used to call the feedback recognition module to work; a second logic control module, Used to control the review guidance module to work when the autistic child does not respond, and control the sensory stimulation module to work when the autistic child responds; a social gesture learning module, used to guide the autistic child to learn social gestures; a second function calling module, used to call the feedback recognition module to work; a third logic control module, used to control the gesture learning module to work when the autistic child does not respond, and control the sensory stimulation module to work when the autistic child responds; an emotion expression training module, used to guide the autistic child to learn emotion expression; the third function calling module is used to call the feedback recognition module to work; a fourth logic control module, used to control the expression training module to work when the autistic child does not respond, and control the sensory stimulation module to work when the autistic child responds.

[0023] Furthermore, the active intervention training system also includes: a first data acquisition module, used to collect the first recognition result in real time; a fifth logic control module, used to control the intervention guidance module to work when the first recognition result is moving the eyeballs or looking at the robot, and control the first data acquisition module to work when the first recognition result is not moving the eyeballs and not looking at the robot.

[0024] Furthermore, the intervention training guidance module includes: a music playing unit for playing music; a light display unit for displaying lights; and a first control unit for controlling the first driving module to drive the robot to perform a dance.

[0025] Furthermore, the feedback status recognition module includes: a second data acquisition unit, used to collect the second recognition result, the third recognition result and the fourth recognition result in real time; a feedback status analysis unit, used to determine whether the autistic child responds based on the second recognition result, the third recognition result and the fourth recognition result; the sensory stimulation reward module includes: the music playback unit, the light display unit and the first control unit.

[0026] Furthermore, the intervention training review module includes: a second control unit, used to control the second driving module to drive the remaining modules of the robot to work according to the content of the previous intervention training.

[0027] Furthermore, the social gesture learning module includes: a third control unit, used to control the first driving module to drive the audio playback module to play the first audio, and drive the robot's arm to simulate the social gesture corresponding to the first audio; a first calling unit, used to call the feedback recognition module to work; a fourth control unit, used to control the third control unit and the first calling unit to work when the autistic child does not respond, and control the fifth control unit to work when the autistic child responds; a fifth control unit, used to control the first driving module to drive the audio playback module to play the second audio.

[0028] Furthermore, the emotion expression training module includes: a sixth control unit, used to control the first driving module to drive the audio playback module to play the third audio and the fourth audio, and drive the robot's arm to simulate the social gesture corresponding to the third audio or the fourth audio, and drive the LED display module to display the emoticon corresponding to the third audio or the fourth audio; a second calling unit, used to call the feedback recognition module to work; a seventh control unit, used to control the sixth control unit and the second calling unit to work when the autistic child does not respond, and control the third calling unit to work when the autistic child responds; a third calling unit, used to call the fifth control unit to work.

[0029] Furthermore, the active intervention training system also includes: a first data storage module, which is used to summarize and store the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire intervention training process, store the content of the current intervention training, and continue to store the data sent by the first data processing module, the data sent by the second data processing module and the data sent by the third data processing module; the first data processing module is used to obtain first statistical data based on the summarized first recognition result, and send the first statistical data to the data storage module; the first statistical data includes: the number of times the autistic child looks at the robot and the length of time the autistic child looks at the robot; the second data processing module is used to obtain second statistical data based on the summarized third recognition result, and send the second statistical data to the data storage module; the second statistical data includes: the number of times the autistic child performs behavioral actions and the accuracy of the behavior; the third data processing module is used to analyze the interaction, course completion and intervention training progress of the autistic child based on the first statistical data and the second statistical data, and send the analysis results to the data storage module.

[0030] Furthermore, the active intervention training system is characterized in that it also includes: an intervention training simulation module, which is used to call the social gesture learning module, the second function calling module, the third logic control module, the emotional expression training module, the third function calling module and the third function calling module in sequence to guide parents or institutional training personnel to work with the robot to simulate real scenes together; a second data storage module, which is used to summarize and store the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire simulation display scene; a task punch-in marking module, which is used to execute the punch-in task after completing the simulation display scene.

[0031] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0032] 1. The robot can be remotely controlled in real time and actively, so that the robot can behave and respond autonomously. In addition, the active intervention of visual music can attract the attention of autistic children and thus carry out the next step of intervention.

[0033] 2. The intervention training method uses multiple functional modules. By monitoring the response behavior, intervention progress and feedback records of autistic children, the robot's movement and speech behaviors can be changed in real time to personalize the robot program. When the intervention reaches a certain stage, the current program can be adjusted by testing and evaluating autistic children so that the program can target different symptoms of different autistic children;

[0034] 3. During the intervention process, the robot combines social gestures, role-playing dialogues, story reading and other scientific intervention modes to increase the interaction ability, concentration, response to others' communication and emotional adaptation of autistic children, making up for the shortcomings of most robots with single functions.

[0035] 4. The robot's body is gray and the display screen is light blue, which is in line with the color preferences of autistic children. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the exemplary embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A workflow diagram of active intervention training for autistic children provided by an embodiment of the present invention;

[0038] Figure 2 A schematic flow chart of an active intervention training method for autistic children provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0040] Embodiment: Provide an active intervention training method for autistic children, using the robot's Wizard of Oz solution, which can remotely control the robot in real time and actively to induce the robot's autonomous behavior and response. The robot monitors the autistic children in real time, recognizes the autistic children's attention, facial expressions and behavioral actions based on the monitoring video data, and sends the recognition results to the remote control end. The application program of the remote control end controls the robot to provide emotional and social prompts to the autistic children in the form of audio-visual expression.

[0041] Based on the Wizard of Oz solution, this method uses a robot with built-in video monitoring module, face recognition module, behavior recognition module and speech recognition module.

[0042] 1. Video surveillance module

[0043] The video monitoring module is used to monitor the face and limbs of autistic children in real time and generate monitoring videos.

[0044] 2. Face recognition module

[0045] The face recognition module is used to recognize the eyeball activity state according to the monitoring video to obtain a first recognition result, and recognize the facial expression change to obtain a second recognition result. The face recognition module includes: an eyeball positioning recognition unit and a facial expression recognition unit.

[0046] The first recognition result includes: moving the eyeball, not moving the eyeball, looking at the designated target, and not looking at the designated target, where the designated target is a robot. The second recognition result includes expressions such as happy, sad, angry, excited, and confused.

[0047] (1) Eye positioning recognition unit

[0048] Children with autism lack eye contact and eye gaze when communicating with people face to face. People's attention and emotions can be conveyed through eye contact. The eyeball positioning and recognition unit simulates the shape of the eyeball and the eyelid shape based on the monitoring video using 3D skeleton geometric modeling technology, and uses a stereoscopic vision system to capture the stereoscopic movement of the eyeball and recognize the activity state of the eyeball to obtain the first recognition result.

[0049] (2) Facial Expression Recognition Unit

[0050] Children with autism lack effective emotional expression when communicating with people face to face. The facial expression recognition unit performs real-time facial expression recognition based on the facial emotion recognition method of 2D-Gabor, uniform local binary pattern (LBP) operator and multi-class extreme learning machine classifier. For specific technical content, please refer to "Zhentao Liu, Min Wu, Weihua Cao, Luefeng Chen, Jianping Xu, Ri Zhang, Mengtian Zhou, Junwei Mao. A Facial Expression Emotion Recognition Based Human-robot Interaction System[J]. Acta Automatica Sinica: English Edition, 2017, 0(4): 668-676.".

[0051] Furthermore, using Python technology, combined with the surveillance video collected by the video surveillance module, an emotion visual tracking model is established, and combined with the facial feature detection method, sample analysis of emotions is performed; a database is established, and sample analysis of the robot emotion database is performed through fuzzy information fusion, and a robot emotion visual feature extraction model is established. Information tracking and facial expression detection are performed through feature extraction results. For specific technical content, please refer to "Dong Yuxia, Su Rongcong, Zheng Xinliang. Research on Intelligent Robot Emotion Recognition Technology Based on Python Technology [J]. Journal of Heilongjiang Institute of Technology: Comprehensive Edition, 2020, 20(10): 60-64.".

[0052] 3. Behavior recognition module

[0053] The behavior recognition module is used to recognize the behavior action according to the monitoring video to obtain the third recognition result. The third recognition result specifically includes walking, wiping sweat, hugging, running, punching, saying goodbye, waving, shaking hands, waving, crossing hands, raising hands above the head and waving, etc.

[0054] According to the data type of the collected surveillance video, it can be divided into three types: RGB image, 2D depth map, and 3D skeleton. The behavior recognition module is based on machine vision behavior recognition, which uses images or video streams to detect the behavioral activities of autistic children. It uses image processing technology as the core, processes frames by frame, and describes and identifies human actions in video sequences. For specific technical content, please refer to "Yang Guanci, Li Yang, Zhao Le, Liu Saisai, He Ling, Liu Dan. A review of user behavior recognition methods based on sensor data [J]. Packaging Engineering, 2021, 42(18): 94-102133, I0009.".

[0055] 4. Speech Recognition Module

[0056] The speech recognition module is used to recognize the sound emitted by the autistic child to obtain a fourth recognition result. The specific content of the fourth recognition result includes the sounds of moving forward, moving backward, turning left, turning right, stopping, telling a story, etc.

[0057] The speech recognition module uses speech processing and recognition technology to recognize the sounds made by autistic children and can correctly understand simple sentences. It should be noted that regarding the pronunciation and pauses of polyphones, the speech recognition module adds erhua sounds, sentence-end punctuation, and changes in intonation to improve the naturalness of the entire synthesis effect. The accuracy of English pronunciation in Chinese-English mixed TTS is improved by using hybrid deep learning and statistical cross-language models.

[0058] 5. Other functional modules

[0059] The robot described in this embodiment further includes an audio playback module, a light display module, an LED display module, a first driving module and a second driving module.

[0060] (1) The audio playback module plays various pre-stored audio scripts according to the staged tasks, including: playing music, reading stories, reading sentences, etc.;

[0061] (2) The light display module can display lights;

[0062] (3) The LED display module is used to simulate eyes and mouth to express emotions such as happiness, sadness, surprise, anger, and shyness;

[0063] (4) The first driving module and the second driving module are both used to drive the above-mentioned functional modules to work.

[0064] In addition, the robot described in this embodiment also includes: a two-wheel differential moving mechanism and a Beidou module; the body part of the robot is gray, and the LED display module is a light blue LED display screen; a servo is installed at the connection between the robot's arm and the side of the body, and a transmission mechanism and a servo are installed on the neck. The servo located on the neck is fixed on a cylindrical component inside the neck, and the end of the cylindrical structure is fixed on a bearing. The length of the cylindrical structure is greater than the length of the neck and penetrates deep into the head of the robot.

[0065] Specifically: On the one hand, for a robot with two-wheel differential movement, by establishing an appropriate kinematic model for the robot, its movement process can be simulated more accurately. The Beidou module is ATGM336H, a high-performance positioning and navigation module that supports a variety of satellite navigation systems and realizes joint positioning, navigation and timing. It has the advantages of high sensitivity and low power consumption. On the other hand, a study on the color preferences of autistic children found that autistic children generally prefer cold colors, and the order of their color preferences is: green, blue, red, purple, orange, and yellow. The color preference order is significantly different from that of ordinary children. Therefore, the body of this robot is gray and the display screen is light blue. On the other hand, a wiring servo is installed on the arm of the robot, and the servo is fixed to the side of the robot body to realize the rotation of the arm with freedom; because the servo cannot bear axial force, a set of transmission mechanical structure is designed in the neck. The servo is fixed to the neck and hidden in the robot body. The servo shaft is fixed to a cylindrical member inside the neck, the cylindrical member is inserted into the cylindrical neck, and the end is fixed to the bearing. The bearing is placed in a vacant space with a larger inner diameter inside the cylindrical neck to prevent the bearing from falling off. The cylindrical component is slightly longer than the neck, extends into the head and is fixed, so that all the weight of the head is borne by the neck and the bearing inside the neck, thereby realizing head transmission without axial force.

[0066] Based on the above robot, this embodiment provides an active intervention training method for autistic children, such as Figure 1 As shown in FIG, the method is divided into the guiding stage, the review stage, the learning and consolidation stage, and the storage stage. The purpose of the guiding stage is to attract the joint attention of autistic children through visual music introduction, the review stage is to learn the training plan of the previous intervention training, the main task of the learning and consolidation stage is to guide autistic children to learn and consolidate social gestures and emotional expressions, and the storage stage mainly stores the feedback records, intervention progress, and training effect evaluation results during the intervention training process, so as to adjust the intervention training plan.

[0067] The specific implementation steps of this method are as follows: Figure 2 shown.

[0068] The boot phase consists of the following steps:

[0069] Step 1: Collect the first recognition result in real time.

[0070] Step 2: Determine whether the robot successfully guides the attention of the autistic child based on the first recognition result.

[0071] If the first recognition result is that the eyeballs are moved or the child looks at the robot, it can be determined that the robot has attracted the attention of the autistic child, and step 3 can be executed. If the first recognition result is that the eyeballs are not moved and the child does not look at the robot, it can be determined that the robot has not attracted the attention of the autistic child, and the judgment method described in steps 1 and 2 needs to be continued.

[0072] Step 3: Using a visual music intervention method to guide the attention of the autistic children.

[0073] Music is beneficial to improving the mood, social skills, behavior, perception and language of children with autism. Visual music therapy is a new method of music therapy. It uses digital signal processing technology and voice-controlled animation technology to organically combine vision, hearing and movement to achieve multi-sensory stimulation under the comprehensive intervention of the three treatment elements of music, animation and lighting. Visual music therapy can significantly increase the number of children's active responses, significantly reduce self-stimulatory behavior, and significantly reduce aggressive behavior.

[0074] This method uses the comprehensive stimulation elements of hearing (music), vision (pictures), touch (movement) and other visual music therapy to continuously give autistic children new sensory stimulation, prolong their concentration time, and thus reduce the frequency of their bad behaviors. Through continuous training, it generalizes it to life and learning, and ultimately achieves the goal of eliminating their bad behaviors. When the robot recognizes that the autistic child is moving his eyes or looking at the robot, the robot will give feedback, play music or dance, greet, emit light, etc. This can be used as a reward for the child's active communication, attracting the child's attention and allowing for the next step of intervention.

[0075] Based on the above analysis, the specific implementation method of step 3 is: controlling the audio playback module to play music, controlling the light display module to display lights, or controlling the first driving module to drive the robot to perform dance.

[0076] Step 4: Determine whether the autistic child has responded.

[0077] The purpose of this step is to determine the response of autistic children to the visual music intervention. Specifically, after the visual music intervention, whether the autistic children have facial emotion changes, behavioral movements or sound feedback is used to determine whether they have responded to the visual music intervention.

[0078] The specific implementation method is as follows: first, the second recognition result, the third recognition result and the fourth recognition result are collected in real time; then, the second recognition result, the third recognition result and the fourth recognition result are respectively extracted to obtain a second feature vector corresponding to the second recognition result, a third feature vector corresponding to the third recognition result and a fourth feature vector corresponding to the fourth recognition result, wherein the second feature vector is used to represent a variety of expression change features, the third feature vector is used to represent the action frequency, action type or the association between the action and the stimulus, and the fourth feature vector is used to represent the tone, volume or vocalization frequency; next, the second feature vector, the third feature vector and the fourth feature vector are respectively input into a pre-trained machine learning model (such as a support vector machine) for classification, and the second classification result, the third classification result and the fourth classification result are output, and the second classification result, the third classification result and the fourth classification result are all probability values; finally, the second classification result, the third classification result and the fourth classification result are respectively compared with a response threshold set in advance, if there is at least one probability value greater than the response threshold, it means that the autistic child has responded to the stimulus, and if the output probability values ​​are all less than the response threshold, it means that the autistic child has not responded to the stimulus.

[0079] If no response is given, return to step 3 and continue to use the visual music intervention method to guide the attention of the autistic child; if a response is given, execute step 5.

[0080] Step 5: Reward the autistic child using sensory stimulation methods.

[0081] The sensory stimulation method, i.e., the visual music intervention method in step 3, can be used to guide the attention of autistic children and also to reward the responses of autistic children. Thus, the guidance stage task is completed.

[0082] The review phase includes the following steps:

[0083] Step 6: Guide the autistic child to review the content of the previous intervention training.

[0084] For the second and subsequent intervention training processes, the content of the previous intervention training needs to be reviewed. The specific implementation method is: controlling the second driving module to drive the remaining modules of the robot to work according to the content of the previous intervention training.

[0085] Step 7: Determine whether the autistic child has responded.

[0086] If no response is made, return to step 6 and continue to guide the autistic child to review the content of the previous intervention training; if a response is made, the autistic child is rewarded in the manner described in step 5 to complete the task of the review stage and enter the learning and consolidation stage.

[0087] The tasks in the learning and consolidation stage include social gesture learning and consolidation, and emotional expression learning and consolidation.

[0088] Among them, the social gesture learning and consolidation part includes the following specific implementation steps:

[0089] Step 8: Guiding the autistic child to learn and consolidate social gestures.

[0090] Referring to "Human-recognizable robotic gestures. IEEE Transactions on Autonomous Mental Development, 4(4):305-314" published by Kabibihan et al., a total of 14 non-transitive gestures commonly used in daily life were taught. These gestures can be used in robot interactive dialogues, including: "hello", "goodbye", "wrong", "cheers", "yes", "no", "hungry", "me", "irritated", "angry", "wait", "welcome", "come" and "where". "Lv Yanchen. Study on gesture imitation of autistic children under different demonstration conditions based on NAO robot [D]. East China Normal University, 2019." Referring to previous studies, 19 robot gestures were determined to ensure that the actions made by the robot can be understood by us.

[0091] This method uses the 14 non-transitive gestures of Kabibihan et al. (2012) to achieve social gesture imitation in children with ASD. This is an alternative and augmentative communication (AAC) method that can enhance the expressive communication of children with ASD, especially those with limited language skills.

[0092] Based on the above research results, step 8 specifically includes:

[0093] Step 8.1: Control the first driving module to drive the audio playing module to play the first audio, and at the same time drive the robot's arm to simulate a social gesture corresponding to the first audio.

[0094] The following are some examples:

[0095] (1) The robot plays the first audio “Hello, teacher” and makes a handshake gesture;

[0096] (2) The robot plays the first audio “Goodbye, teacher” and makes a waving gesture;

[0097] (3) The robot plays the first audio message “The answer is incorrect. Please think again,” and makes a hand-crossing gesture;

[0098] (4) The robot plays the first audio “Welcome” and makes a waving gesture with both hands raised above the head.

[0099] Step 8.2: Determine whether the autistic child responds to the audio and gestures emitted by the robot in step 8.1.

[0100] If no response is made, return to step 8.1; if a response is made, execute step 8.3 to consolidate the learning content of step 8.1.

[0101] Step 8.3: Control the first driving module to drive the audio playing module to play the second audio.

[0102] In the social gesture consolidation stage, the robot asked questions to autistic children by playing the second audio. For example, goodbye - waving gesture; story introduction: get out of class is over; robot: "Goodbye, teacher", and waved.

[0103] Step 9: Determine whether the autistic child responds to the second audio signal emitted by the robot in step 8.3.

[0104] If no response is made, return to step 8.3; if a response is made, reward the autistic child in the manner described in step 5 to complete the social gesture learning and consolidation task and enter the emotional expression learning and consolidation stage.

[0105] The emotional expression learning and consolidation part includes the following specific gesture steps:

[0106] Step 10: Guide the autistic children to learn and consolidate emotional expression.

[0107] This method mainly improves the emotion recognition and expression ability of autistic children through the story reading function of the robot. Story reading mainly involves two stages. The first stage: emotion recognition activities, the robot reads a story and makes sounds such as angry (whoosh), happy (haha), etc., with corresponding expressions and actions (angry: crying expression and hands hanging on the display, happy: happy expression and arms raised above the head on the display). The purpose of this part is to enable autistic children to recognize basic emotions. The second stage: emotion and situation association activities, the activities focus on the game of "When I feel how...", that is, for the presentation of social situations, the robot reads the story again and repeats the sounds, expressions and actions in the story. The robot gives instructions: "I am very happy in the story now, how do you feel?" Ask the child to experience the robot's emotions. The purpose of this part is to teach autistic children how to present appropriate emotions related to the situation.

[0108] The specific implementation steps of step 10 are as follows, including:

[0109] Step 10.1: Control the first driving module to drive the audio playback module to play the third audio and the fourth audio, and at the same time drive the robot's arm to simulate a social gesture corresponding to the third audio or the fourth audio, and drive the LED display module to display emoticons corresponding to the third audio or the fourth audio.

[0110] For example:

[0111] (1) The robot plays the third audio “My mother gave me a gift today, so I am very happy” and the fourth audio “Hahaha laughter”, makes a clapping gesture, and displays a happy expression (smile);

[0112] (2) The robot plays the third audio “I feel sad because I didn’t assemble the car” and the fourth audio “Whinging sound”, and makes a gesture of wiping tears while showing a sad expression (crying);

[0113] (3) The robot plays the third audio “My toy was taken away by someone else, so I am very angry” and the fourth audio “Whining sound”, and makes a akimbo gesture while showing an angry expression (anger);

[0114] (4) The robot plays the third audio “Mom and Dad are going to take me out to play, so I’m very excited” and the fourth audio “Woo-hoo sound”, and makes a gesture of raising both arms above the head while showing an excited expression (looking up and laughing);

[0115] (5) The robot plays the third audio “The room is dark with the lights off, so I’m scared” and the fourth audio “The sound of teeth chattering”, and makes a gesture of crossing its arms and shoulders, while showing a scared expression (eyes closed);

[0116] (6) The robot plays the third audio “I don’t know how to solve this problem, so I’m confused” and the fourth audio “Hmm?” and makes a head scratching gesture while showing a puzzled expression (frown).

[0117] Step 10.2: Determine whether the autistic child responds to the audio and gestures emitted by the robot in step 8.1.

[0118] If no response is made, return to step 9.1; if a response is made, execute step 9.4;

[0119] Step 10.3: Control the first driving module to drive the audio playing module to play the second audio.

[0120] In the social gesture consolidation stage, the robot asked questions to autistic children by playing the second audio. For example, goodbye - waving gesture; story introduction: get out of class is over; robot: "Goodbye, teacher." and made a waving gesture.

[0121] Step 11: Determine whether the autistic child responds to the second audio signal emitted by the robot in step 10.3.

[0122] If no response is made, return to step 10.3; if a response is made, reward the autistic child in the manner described in step 5, complete the task of learning and consolidating emotional expression, and enter the storage stage.

[0123] Storage stage:

[0124] The storage phase mainly includes the following contents:

[0125] First, the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire intervention training process are summarized and stored.

[0126] Second, first statistical data is obtained based on the summarized first recognition results, and the first statistical results are stored; the first statistical data includes: the number of times the autistic child looks at the robot and the duration of looking at the robot. Specifically, the first recognition results are analyzed using eye tracking technology to capture the changes in the child's gaze point on the screen and record the child's eye movements, including the gaze point, the number of gazes, and the duration.

[0127] The third is to obtain second statistical data based on the aggregated third recognition results, and store the second statistical results; the second statistical data includes: the number of behavioral actions performed by the autistic child and the behavioral accuracy rate. Specifically, the number of interactions of the child in a specific task or game, as well as the number of correct tasks performed, are recorded through video monitoring or professional observers; the video is encoded using behavioral analysis software such as Noldus Observer XT to analyze the number of interactions and behavioral patterns; the accuracy rate is calculated: behavioral accuracy rate = number of correct interactions / total number of interactions*100%.

[0128] Fourth, based on the first statistical data and the second statistical data, the interaction, course completion and intervention training progress of the autistic children are analyzed, and the analysis results are stored. Specifically, the analysis of the interaction can be done by teachers or therapists recording the details of each interaction (including the type of interaction, participation and children's response), and by regularly distributing questionnaires to parents and therapists to collect feedback on the quality of interaction; the analysis of course completion can use online learning platforms such as Blackboard or Moodle to track the progress of course completion; the analysis of the progress of intervention training can be done by conducting a baseline assessment of the children before the intervention to record the initial ability level, and can also set time points (such as every 3 months) for re-evaluation to compare the children's progress in various skills. Progress reports can also be generated based on the assessment results, including charts and data analysis, to intuitively display the children's progress.

[0129] 5. Store the content of the current intervention training.

[0130] This is achieved through the following functional modules:

[0131] A first data processing module is used to obtain first statistical data according to the summarized first recognition results, and send the first statistical data to the data storage module; the first statistical data includes: the number of times the autistic child looks at the robot and the length of time the child looks at the robot;

[0132] A second data processing module is used to obtain second statistical data according to the summarized third recognition results, and send the second statistical data to the data storage module; the second statistical data includes: the number of times the autistic child performs behavioral actions and the accuracy rate of the behavior;

[0133] The third data processing module is used to analyze the interaction, course completion and intervention training progress of the autistic children according to the first statistical data and the second statistical data, and send the analysis results to the data storage module.

[0134] In addition, the method also includes guiding the parents or training personnel of the institution to complete the simulation together with the robot, and the specific contents refer to the relevant operations of the social gesture learning and consolidation stage, and the relevant operations of the emotional expression learning and consolidation stage. The relevant data of the entire simulation process is stored with reference to the implementation method of the storage stage.

[0135] It should be noted that, in the process of executing the above steps 3 to 11, the intervention steps can be randomly selected to solve the dependence of autistic children on the above specific order.

[0136] Corresponding to the above-mentioned active intervention training method, the second aspect of this embodiment further provides an active intervention training system for autistic children, including a remote control terminal and the robot as described in the above-mentioned first aspect; the remote control terminal includes: an intervention training guidance module, which is used to guide the attention of the autistic child by adopting a visual music intervention method; a feedback state recognition module, which is used to determine whether the autistic child has responded; a first logic control module, which is used to control the intervention guidance module and the feedback recognition module to work when the autistic child has not responded, and control the sensory stimulation module to work when the autistic child has responded; a sensory stimulation reward module, which is used to reward the autistic child by adopting a sensory stimulation method; an intervention training review module, which is used to guide the autistic child to review the content of the previous intervention training; a first function calling module, which is used to call the feedback recognition module to work ; a second logic control module, used to control the review guidance module to work when the autistic child does not respond, and to control the sensory stimulation module to work when the autistic child responds; a social gesture learning module, used to guide the autistic child to learn social gestures; a second function calling module, used to call the feedback recognition module to work; a third logic control module, used to control the gesture learning module to work when the autistic child does not respond, and to control the sensory stimulation module to work when the autistic child responds; an emotion expression training module, used to guide the autistic child to learn emotional expression; a third function calling module, used to call the feedback recognition module to work; a fourth logic control module, used to control the expression training module to work when the autistic child does not respond, and to control the sensory stimulation module to work when the autistic child responds.

[0137] Furthermore, the active intervention training system also includes: a first data acquisition module, used to collect the first recognition result in real time; a fifth logic control module, used to control the intervention guidance module to work when the first recognition result is moving the eyeballs or looking at the robot, and control the first data acquisition module to work when the first recognition result is not moving the eyeballs and not looking at the robot.

[0138] Furthermore, the intervention training guidance module includes: a music playing unit for playing music; a light display unit for displaying lights; and a first control unit for controlling the first driving module to drive the robot to perform a dance.

[0139] Furthermore, the feedback status recognition module includes: a second data acquisition unit, used to collect the second recognition result, the third recognition result and the fourth recognition result in real time; a feedback status analysis unit, used to determine whether the autistic child responds based on the second recognition result, the third recognition result and the fourth recognition result; the sensory stimulation reward module includes: the music playback unit, the light display unit and the first control unit.

[0140] Furthermore, the intervention training review module includes: a second control unit, used to control the second driving module to drive the remaining modules of the robot to work according to the content of the previous intervention training.

[0141] Furthermore, the social gesture learning module includes: a third control unit, used to control the first driving module to drive the audio playback module to play the first audio, and drive the robot's arm to simulate the social gesture corresponding to the first audio; a first calling unit, used to call the feedback recognition module to work; a fourth control unit, used to control the third control unit and the first calling unit to work when the autistic child does not respond, and control the fifth control unit to work when the autistic child responds; a fifth control unit, used to control the first driving module to drive the audio playback module to play the second audio.

[0142] Furthermore, the emotion expression training module includes: a sixth control unit, used to control the first driving module to drive the audio playback module to play the third audio and the fourth audio, and drive the robot's arm to simulate the social gesture corresponding to the third audio or the fourth audio, and drive the LED display module to display the emoticon corresponding to the third audio or the fourth audio; a second calling unit, used to call the feedback recognition module to work; a seventh control unit, used to control the sixth control unit and the second calling unit to work when the autistic child does not respond, and control the third calling unit to work when the autistic child responds; a third calling unit, used to call the fifth control unit to work.

[0143] Furthermore, the active intervention training system also includes: a first data storage module, which is used to summarize and store the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire intervention training process, store the content of the current intervention training, and continue to store the data sent by the first data processing module, the data sent by the second data processing module and the data sent by the third data processing module; the first data processing module is used to obtain first statistical data based on the summarized first recognition result, and send the first statistical data to the data storage module; the first statistical data includes: the number of times the autistic child looks at the robot and the length of time the autistic child looks at the robot; the second data processing module is used to obtain second statistical data based on the summarized third recognition result, and send the second statistical data to the data storage module; the second statistical data includes: the number of times the autistic child performs behavioral actions and the accuracy of the behavior; the third data processing module is used to analyze the interaction, course completion and intervention training progress of the autistic child based on the first statistical data and the second statistical data, and send the analysis results to the data storage module.

[0144] Furthermore, the active intervention training system is characterized in that it also includes: an intervention training simulation module, which is used to call the social gesture learning module, the second function calling module, the third logic control module, the emotional expression training module, the third function calling module and the third function calling module in sequence to guide parents or institutional training personnel to work with the robot to simulate real scenes together; a second data storage module, which is used to summarize and store the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire simulation display scene; a task punch-in marking module, which is used to execute the punch-in task after completing the simulation display scene.

[0145] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. An active intervention training method for autistic children, characterized in that: The following steps are involved: S11: Using visual music intervention method to guide the attention of the autistic children; S12: determining whether the autistic child has responded; S13: If no response is made, return to S11; if a response is made, execute S14: S14: rewarding the autistic child using a sensory stimulation method; S21: guiding the autistic child to review the content of the previous intervention training; S22: Execute the operation described in S12; S23: If no response is made, return to S21; if a response is made, perform the operation described in S14; S31: guiding the autistic child to learn and consolidate social gestures; S32: Execute the operation described in S12; S33: If no response is made, return to S31; if a response is made, perform the operation described in S14; S41: guiding the autistic child to learn and consolidate emotional expression; S42: Execute the operation described in S12; S43: If no response is made, return to S41; If a response is made, the operation described in S14 is executed.

2. The active intervention training method for autistic children according to claim 1, characterized in that: The active intervention training method is based on a robot; the robot comprises: A video monitoring module, used to monitor the face and limbs of the autistic child in real time and generate a monitoring video; A face recognition module is used to simulate the shape of the eyeball and the eyelid using 3D skeleton geometric modeling technology according to the surveillance video, capture the stereoscopic movement of the eyeball and identify the activity state of the eyeball using a stereoscopic vision system to obtain a first recognition result, and identify the change of facial expression using the stereoscopic vision system to obtain a second recognition result; A behavior recognition module is used to recognize human behavior actions in the surveillance video frame by frame to obtain a third recognition result; A speech recognition module, used for recognizing the sound emitted by the autistic child to obtain a fourth recognition result; The robot also includes: an audio playing module, a light display module, an LED display module, a first driving module and a second driving module.

3. The active intervention training method for autistic children according to claim 2, characterized in that: Before S11, it also includes: S10.1: Collect the first recognition result in real time; S10.2: If the first recognition result is that the eyeballs are moved or the user is looking at the robot, then execute S11: If the first recognition result is that the eyeballs are not moved and the user is not looking at the robot, then return to S10.1; The S11 specifically includes: controlling the audio playback module to play music, controlling the light display module to display light, or controlling the first driving module to drive the robot to perform dance; The S12 specifically includes: collecting the second recognition result, the third recognition result and the fourth recognition result in real time, and judging whether the autistic child responds according to the second recognition result, the third recognition result and the fourth recognition result; The S14 specifically includes: rewarding the autistic child using the method described in S11; The S21 specifically includes: controlling the second driving module to drive the remaining modules of the robot to work according to the content of the previous intervention training; The S31 includes: S31.1: Control the first driving module to drive the audio playing module to play a first audio, and at the same time drive the robot's arm to simulate a social gesture corresponding to the first audio; S31.2: Execute the operation described in S12; S31.3: If no response is made, return to S31.1; if a response is made, execute S31.4; S31.4: Control the first driving module to drive the audio playing module to play the second audio; The S41 includes: S41.1: Control the first driving module to drive the audio playback module to play the third audio and the fourth audio, and at the same time drive the robot's arm to simulate a social gesture corresponding to the third audio or the fourth audio, and drive the LED display module to display an emoticon corresponding to the third audio or the fourth audio; S41.2: Execute the operation described in S12; S41.3: If no response is made, return to S41.1; if a response is made, execute S41.4; S41.4: Control the first driving module to drive the audio playing module to play the second audio.

4. The active intervention training method for autistic children according to claim 2, characterized in that: After S43, the following steps are also included: Summarizing and storing the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire intervention training process; Acquire first statistical data according to the summarized first recognition results, and store the first statistical results; the first statistical data includes: the number of times the autistic child looks at the robot and the length of time the child looks at the robot; Obtaining second statistical data according to the summarized third recognition results, and storing the second statistical results; the second statistical data includes: the number of times the autistic child performs behavioral actions and the accuracy rate of the behavior; Analyzing the interaction situation, course completion situation and intervention training progress of the autistic children according to the first statistical data and the second statistical data, and storing the analysis results; The content of the current intervention training is stored.

5. An active intervention training method for autistic children according to any one of claims 2 to 4, characterized in that: The following steps are also included: Guide the parents or institution training personnel to simulate the real scene together with the robot by jointly executing S31 to S43; The first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire simulation display scene are summarized and stored.

6. An active intervention training method for autistic children according to claim 3 or 4, characterized in that: The face recognition module comprises: An eyeball positioning and recognition unit, configured to simulate the shape of the eyeball and the eyelid using 3D skeleton geometric modeling technology according to the surveillance video, capture the stereoscopic movement of the eyeball using a stereoscopic vision system and recognize the activity state of the eyeball, and obtain the first recognition result; A facial expression recognition unit, configured to recognize facial expression changes using a stereoscopic vision system according to the surveillance video to obtain the second recognition result; The speech recognition module comprises: An audio script storage unit, used for storing the music, the first audio, the second audio, the third audio and the fourth audio; The robot also includes: a two-wheel differential movement mechanism and a Beidou module; The body part of the robot is gray, and the LED display module is a light blue LED display screen; A servo is installed at the connection between the robot's arm and the side of the body; a transmission mechanism and a servo are installed on the robot's neck, the servo located on the neck is fixed on a cylindrical component inside the neck, the end of the cylindrical structure is fixed on a bearing, the length of the cylindrical structure is greater than the length of the neck and penetrates deep into the head of the robot.

7. An active intervention training system for autistic children, characterized in that: comprising a remote control terminal and the robot according to claim 2; The remote control terminal comprises: An intervention training guidance module, used to guide the attention of the autistic child using a visual music intervention method; A feedback status identification module, used to determine whether the autistic child has responded; A first logic control module, configured to control the intervention guidance module and the feedback identification module to work when the autistic child does not respond, and to control the sensory stimulation module to work when the autistic child responds; A sensory stimulation reward module, used to reward the autistic child using a sensory stimulation method; An intervention training review module, used to guide the autistic child to review the content of the previous intervention training; A first function calling module, used for calling the feedback identification module to work; A second logic control module is used to control the review guidance module to work when the autistic child does not respond, and to control the sensory stimulation module to work when the autistic child responds; A social gesture learning module, used to guide the autistic child to learn social gestures; A second function calling module is used to call the feedback identification module to work; A third logic control module is used to control the gesture learning module to work when the autistic child does not respond, and to control the sensory stimulation module to work when the autistic child responds; An emotion expression training module, used to guide the autistic child to learn emotion expression; A third function calling module is used to call the feedback identification module to work; The fourth logic control module is used to control the expression training module to work when the autistic child does not respond, and to control the sensory stimulation module to work when the autistic child responds.

8. The active intervention training system for autistic children according to claim 7, characterized in that: Also includes: A first data collection module, used for collecting the first recognition result in real time; a fifth logic control module, configured to control the intervention guidance module to operate when the first recognition result is that the eyeball is moved or the user is looking at the robot, and to control the first data acquisition module to operate when the first recognition result is that the eyeball is not moved and the user is not looking at the robot; The intervention training guidance module includes: A music playing unit, used for playing music; A lighting display unit, used for displaying lights; A first control unit, used for controlling the first driving module to drive the robot to perform a dance; The feedback state identification module comprises: A second data collection unit, used for collecting the second recognition result, the third recognition result and the fourth recognition result in real time; a feedback status parsing unit, configured to determine whether the autistic child responds according to the second recognition result, the third recognition result and the fourth recognition result; The sensory stimulation reward module includes: the music playing unit, the light display unit and the first control unit; The intervention training review module includes: A second control unit, used for controlling the second driving module to drive the remaining modules of the robot to work according to the content of the previous intervention training; The social gesture learning module includes: a third control unit, configured to control the first driving module to drive the audio playing module to play the first audio, and drive the arm of the robot to simulate a social gesture corresponding to the first audio; A first calling unit, used for calling the feedback identification module to work; a fourth control unit, configured to control the third control unit and the first calling unit to operate when the autistic child does not respond, and to control the fifth control unit to operate when the autistic child responds; A fifth control unit, used for controlling the first driving module to drive the audio playing module to play a second audio; The emotion expression training module includes: a sixth control unit, configured to control the first driving module to drive the audio playing module to play the third audio and the fourth audio, drive the arm of the robot to simulate a social gesture corresponding to the third audio or the fourth audio, and drive the LED display module to display an emoticon corresponding to the third audio or the fourth audio; A second calling unit, used for calling the feedback identification module to work; a seventh control unit, configured to control the sixth control unit and the second calling unit to operate when the autistic child does not respond, and to control the third calling unit to operate when the autistic child responds; The third calling unit is used to call the fifth control unit to operate.

9. An active intervention training system for autistic children according to claim 7 or 8, characterized in that: Also includes: The first data storage module is used to summarize and store the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire intervention training process, store the content of the current intervention training, and continue to store the data sent by the first data processing module, the data sent by the second data processing module and the data sent by the third data processing module; A first data processing module, configured to obtain first statistical data according to the summarized first recognition results, and send the first statistical data to the data storage module; The first statistical data includes: the number of times the autistic child looks at the robot and the duration of looking at the robot; A second data processing module is used to obtain second statistical data according to the summarized third recognition results, and send the second statistical data to the data storage module; the second statistical data includes: the number of times the autistic child performs behavioral actions and the accuracy rate of the behavior; The third data processing module is used to analyze the interaction, course completion and intervention training progress of the autistic children according to the first statistical data and the second statistical data, and send the analysis results to the data storage module.

10. An active intervention training system for autistic children according to claim 7 or 8, characterized in that: Also includes: An intervention training simulation module is used to sequentially call the social gesture learning module, the second function calling module, the third logic control module, the emotion expression training module, the third function calling module and the third function calling module to guide parents or institutional training personnel to work together with the robot to simulate real scenes; A second data storage module is used to summarize and store the first recognition result, the second recognition result, the third recognition result and the fourth recognition result collected during the entire simulation display scene process; The task punch-in marking module is used to execute the punch-in task after completing the simulation display scene.

Images