Multi-modal interactive closed-loop training system for autism spectrum disorder patient

Abstract

The invention discloses a multi-mode interactive closed-loop training system for autism spectrum disorder patients. A hardware part of the system comprises data acquisition equipment, a display screen and sound equipment; the software part comprises a database, a data preprocessing module, a data calling module and a data analyzing and processing module, and the data analyzing and processing module is used for analyzing whether a subject makes a correct choice within a limited time or not according to eyeball movement data and pupil size data acquired by the data acquisition equipment after information display is performed on the display screen and the sound box; and the training difficulty of the subject is adjusted according to the analysis result. The eye movement data of the testee is collected through the data collection device so as to judge whether the testee completes the training task within the specified time threshold value or not, the instruction for improving the training difficulty or reducing the training difficulty is made according to the judgment result, and an ABA therapist can be helped to conduct intervention treatment on autism children to a certain degree.

Description

Technical Field

[0001] The present invention relates to the technical field of rehabilitation training, and in particular to a multimodal interactive closed-loop training system for patients with autism spectrum disorders. Background Art

[0002] Autism Spectrum Disorder (ASD) means that patients have serious obstacles in basic social communication skills such as social behavior and social language. ASD is a behaviorally defined neurodevelopmental disorder, and its core symptoms are divided into two aspects: difficulties in social communication and interaction, and restricted, repetitive behaviors and interests. Communication difficulties are a characteristic of ASD and one of the hallmark features for diagnosing ASD. ASD patients have slower language processing speed than normal children. At the same time, ASD children produce fewer social gestures (indicative gestures, traditional gestures, iconic gestures) than normal children of the same age, and their ability to understand social gestures is also weaker than that of normal children of the same age. Understanding social gestures is also one of the main tasks of applied behavior analysis in intervening in ASD children.

[0003] At present, most of the rehabilitation of ASD patients is carried out through ABA intervention therapy in rehabilitation centers, but the number of ABA therapists in China is still in short supply, and most ABA therapists lack experience, which makes it difficult to make an appointment for the rehabilitation of ASD children, expensive, and inefficient, which creates a huge burden on the patients' families. Therefore, there is an urgent need for a system that can assist ABA therapists in intervening with autistic children. Summary of the invention

[0004] The purpose of the present invention is to provide a multimodal interactive closed-loop training system for patients with autism spectrum disorders, which can assist ABA therapists in performing intervention treatment on autistic children.

[0005] To achieve the above object, the present invention provides the following technical solution: a multimodal interactive closed-loop training system for patients with autism spectrum disorder, comprising a hardware part and a software part, wherein the hardware part comprises: Data collection equipment, used to collect the subject's eye movements and pupil size; A display screen, used to display picture information to the subjects; A speaker, used to send audio information to the subjects; The software part includes: Database, used to store image data and audio data; A data preprocessing module, used to divide the image data and audio data in the database into three data sets of difficulty one, difficulty two, and difficulty three according to the direction of increasing difficulty gradient, wherein each of the three data sets includes a plurality of data combinations, each of which includes four image data showing different social behaviors and audio data corresponding to the social behavior shown in one of the images; The data retrieval module is used to retrieve the data combination processed by the data preprocessing module from the database and display the information to the subjects through the display screen and the audio; The data analysis and processing module is used to analyze whether the subjects make correct choices within a limited time after the information is displayed on the display screen and the audio system based on the eye movement data and pupil size data collected by the data acquisition equipment, and adjust the training difficulty of the subjects based on the analysis results.

[0006] Preferably, the data acquisition device comprises an infrared light source, and the infrared light source emits infrared light to form reflected light on the corneas of the two eyes; It also includes an eye tracking camera for tracking the reflected light and feeding the data back to the data analysis and processing module.

[0007] Preferably, the infrared light source and the eye tracking camera are both installed below the display screen.

[0008] Preferably, the limited time includes several groups of time thresholds that decrease in sequence. If the subject pays attention to the target behavior within the time threshold, it is determined that the subject completes the matching task within the specified time, and the task difficulty is increased by reducing the time threshold. If the subject completes the matching task within the minimum time threshold, the difficulty of the data set is increased.

[0009] Preferably, if the subject fails to pay attention to the target behavior within the specified time, it is determined that the subject has not completed the matching task within the specified time, all characters in the data combination except the target are blurred, the audio is played again while highlighting the target, the subject is given intervention training, and then the subject is retested. If the subject fails to complete the matching task for three consecutive times, the training difficulty of the subject is reduced by increasing the time threshold or reducing the difficulty of the data set.

[0010] Preferably, the time threshold includes three time gradients of 3s, 2.4s and 1.8s.

[0011] The method for processing the image data and audio data in the database through the data preprocessing module is as follows: S1, train an image difficulty classification model; S2, use the trained image difficulty classification model to classify the new image set and distribute the new image set into three data sets according to the difficulty type; S3, uses a random algorithm to select several pictures from the same type of data set for stitching; S4, using a random algorithm to select one of the stitched images as the target image; S5, packaging the audio data of the target image obtained in S4 and the multiple images spliced ​​in S3 into a data combination; S6, repeating steps from S3 to S5, can obtain several data combinations of three difficulty gradients.

[0012] In S1, the steps for training the image difficulty classification model are as follows: S101, collecting and preparing image data for training the model, preprocessing the image data, and dividing the preprocessed data into a training set, a validation set, and a test set; S102, select the CNN convolutional neural network model, select the cross entropy loss function and Adam optimizer, and set the learning rate, momentum, and weight decay parameters; S103, input the training set into the model for training; use the validation set to monitor the training process of the model to avoid overfitting; adjust the model parameters and training strategy according to the performance of the validation set; S104, use the test set to evaluate the performance of the model; analyze the errors of the model and make improvements.

[0013] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention collects the eye movement data of the subject through a data acquisition device to determine whether the subject completes the training task within a specified time threshold, and makes instructions to increase or decrease the difficulty of training based on the judgment result, which can help ABA therapists to intervene in the treatment of autistic children to a certain extent.

[0014] Second, the present invention sets multiple data sets with different difficulty gradients and multiple different time thresholds, which is conducive to detailed division of training difficulty and can better select the training difficulty suitable for the subject. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the connection of various parts of the present invention; Figure 2 It is a schematic diagram of the difficulty gradient of the present invention; Figure 3 A schematic diagram of a cross gaze point is displayed on the screen of the display; Figure 4 For social behavior pictures; Figure 5 This is a schematic diagram of blurring a social behavior picture; Figure 6This is a simulation flow chart of the present invention when performing intervention training at difficulty level 2 of 3.0s. DETAILED DESCRIPTION

[0016] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solution of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] See also Figures 1 to 6 The present invention provides a technical solution: a multimodal interactive closed-loop training system for patients with autism spectrum disorder, including a hardware part and a software part, wherein the hardware part includes: a data acquisition device, a display screen and an audio system. The data acquisition device is used to collect the eye movement and pupil size of the subject; the display screen is used to display picture information to the subject; the audio system is used to send audio information to the subject; The software part includes: a database, a data preprocessing module, a data retrieval module and a data analysis and processing module. The database is used to store image data and audio data; the data preprocessing module is used to divide the image data and audio data in the database into three data sets of difficulty one, difficulty two and difficulty three according to the direction of increasing difficulty gradient, and the three data sets each contain a number of data combinations, each of which includes four image data showing different social behaviors and audio data corresponding to the social behavior shown in one of the images; the data retrieval module is used to retrieve the data combination processed by the data preprocessing module from the database, and display the information to the subject through the display screen and the audio; the data analysis and processing module is used to analyze whether the subject makes a correct choice within a limited time after the information is displayed on the display screen and the audio according to the eye movement data and pupil size data collected by the data acquisition device, and adjust the training difficulty of the subject according to the analysis results.

[0018] The present invention uses the Tobii Pro X3-120 screen-type (telemetry) eye tracker of the Swedish tobii company to collect eye movement data. This model of eye tracker is equipped with an eye tracking camera and an infrared light source. The screen resolution of the display screen is 1280*1024 pixels. Eye movements and pupil size are recorded at a rate of 60 data points per second (250hz). The system tracks the two eyes separately by emitting infrared light to create corneal reflections. The eye tracking camera and infrared light source are installed below the screen monitor, and there is no need to fix the subject's head. The distance between the center of the stimulus display screen and the subject's eyes is about 60cm. The experiment starts with a 9-point calibration procedure and is further verified by 4 points displayed on the screen. When the drift exceeds 1 degree, recalibration is required.

[0019] The database includes a social behavior picture library and a social language audio library. According to the color, background, rationality of the posture of the picture, and the sound quality and volume of the audio, a total of 108 social behavior pictures and 27 audios were selected, and divided into three data sets of difficulty one, difficulty two, and difficulty three through the data preprocessing module. The three data sets all contain several data combinations, and each of the data combinations includes four picture data showing different social behaviors and audio data corresponding to the social behavior shown in one of the pictures. Specifically, the difficulty is designed as follows: 30 ASD children and 30 normal children (60 children are independent of the experimental group) are shown all the picture data and audio data, and their eye movement data are collected. The weighted average of the first fixation time of the 60 children on each paradigm target picture is used to make a frequency distribution table, and social behavior is divided into three increasing difficulty gradients: difficulty one, difficulty two, and difficulty three.

[0020] Difficulty 1: waving, nodding, like, applauding, hello, thank you, you’re welcome, yawning, and bowing.

[0021] Difficulty 2: shaking hands, high-fiving, holding hands, touching heads, pinky promises, sleeping, raising hands, making a “yeah” sign, and saluting.

[0022] Difficulty three: hug, cheers, eat, congratulate, make a phone call, give you a little red flower, kiss, eat snacks, and make a heart shape with your hands.

[0023] During the test, a cross gaze point is first displayed on the screen. When the subject gazes at the cross gaze point for 200ms or more, four pictures of four different social behaviors performed by the same person will be randomly displayed in the four directions of the cross gaze point. One second after the picture is displayed, an audio recorded by a professional dubbing staff will be played. The subject needs to find the social behavior picture that matches the audio among the four pictures within a limited time.

[0024] The time limit includes several groups of time thresholds that decrease in sequence, and the time thresholds include three time gradients of 3s, 2.4s, and 1.8s. The design method is as follows: 30 ASD children and 30 normal children (60 children are independent of the experimental group) are shown all the picture data and audio data, and their eye movement data are collected. Take the weighted average of the first fixation time of each target picture of the 30 ASD children, and then take the weighted average of the first fixation time of each target picture of the 30 normal children. The time difference between the two is divided into three difficulty gradients, namely 3.0s, 2.4s, and 1.8s.

[0025] If the subject fixates on the target behavior within the time threshold, it is determined that the subject has completed the matching task within the specified time. The task difficulty is increased by reducing the time threshold. If the subject completes the matching task within the minimum time threshold, the difficulty of data collection is increased.

[0026] If the subject fails to focus on the target behavior within the specified time, it is determined that the subject has not completed the matching task within the specified time. All characters in the data combination except the target are blurred, the audio is played again while highlighting the target, the subject is given intervention training, and then re-tested. If the subject fails to complete the matching task for three consecutive times, the training difficulty of the subject is reduced by increasing the time threshold or reducing the difficulty of the data set.

[0027] Take 3.0s difficulty level 2 as an example: If the subject completes the matching task within the specified time, the difficulty is increased by reducing the time threshold to 2.4s and the difficulty level is increased to continue the test; if the subject still completes the matching task within the specified time, the difficulty is increased by reducing the time threshold to 1.8s and the difficulty level is increased to continue the test; if the subject fails to complete the difficulty level at 2.4s, all characters in the data combination except the target are blurred, the audio is played again while highlighting the target, the subject is given intervention training, and then the subject is re-tested; if the subject fails to complete the matching task for three consecutive times, the subject is trained by increasing the time threshold to 3.0s and the difficulty level is increased.

[0028] The method for processing the image data and audio data in the database through the data preprocessing module is as follows: S1, train an image difficulty classification model; S2, use the trained image difficulty classification model to classify the new image set and distribute the new image set into three data sets according to the difficulty type; S3, uses a random algorithm to select several pictures from the same type of data set for stitching; S4, using a random algorithm to select one of the stitched images as the target image; S5, packaging the audio data of the target image obtained in S4 and the multiple images spliced ​​in S3 into a data combination; S6, repeating steps from S3 to S5, can obtain several data combinations of three difficulty gradients.

[0029] In S1, the steps for training the image difficulty classification model are as follows: S101, collecting and preparing image data for training the model, preprocessing the image data, and dividing the preprocessed data into a training set, a validation set, and a test set; S102, select the CNN convolutional neural network model, select the cross entropy loss function and Adam optimizer, and set the learning rate, momentum, and weight decay parameters; S103, input the training set into the model for training; use the validation set to monitor the training process of the model to avoid overfitting; adjust the model parameters and training strategy according to the performance of the validation set; S104, use the test set to evaluate the performance of the model; analyze the errors of the model and make improvements.

[0030] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multimodal interactive closed-loop training system for patients with autism spectrum disorder, comprising a hardware part and a software part, characterized in that: The hardware part includes: Data collection equipment, used to collect the subject's eye movements and pupil size; A display screen, used to display picture information to the subjects; A speaker, used to send audio information to the subjects; The software part includes: Database, used to store image data and audio data; A data preprocessing module, used to divide the image data and audio data in the database into three data sets of difficulty one, difficulty two, and difficulty three according to the direction of increasing difficulty gradient, wherein each of the three data sets includes a plurality of data combinations, each of which includes four image data showing different social behaviors and audio data corresponding to the social behavior shown in one of the images; The data retrieval module is used to retrieve the data combination processed by the data preprocessing module from the database and display the information to the subjects through the display screen and the audio; The data analysis and processing module is used to analyze whether the subjects make correct choices within a limited time after the information is displayed on the display screen and the audio system based on the eye movement data and pupil size data collected by the data acquisition equipment, and adjust the training difficulty of the subjects based on the analysis results.

2. The multimodal interactive closed-loop training system for patients with autism spectrum disorder according to claim 1, characterized in that: The data acquisition device comprises an infrared light source, through which infrared light is emitted to form reflected light on the corneas of two eyes; It also includes an eye tracking camera for tracking the reflected light and feeding the data back to the data analysis and processing module.

3. The multimodal interactive closed-loop training system for patients with autism spectrum disorder according to claim 2, characterized in that: The infrared light source and the eye tracking camera are both installed below the display screen.

4. The multimodal interactive closed-loop training system for patients with autism spectrum disorder according to claim 1, characterized in that: The limited time includes several groups of time thresholds that decrease in sequence. If the subject pays attention to the target behavior within the time threshold, it is determined that the subject has completed the matching task within the specified time. The task difficulty is increased by reducing the time threshold. If the subject completes the matching task within the minimum time threshold, the difficulty of data collection is increased.

5. The multimodal interactive closed-loop training system for patients with autism spectrum disorder according to claim 4, characterized in that: If the subject fails to focus on the target behavior within the specified time, it is determined that the subject has not completed the matching task within the specified time. All characters in the data combination except the target are blurred, the audio is played again while highlighting the target, the subject is given intervention training, and then re-tested. If the subject fails to complete the matching task for three consecutive times, the training difficulty of the subject is reduced by increasing the time threshold or reducing the difficulty of the data set.

6. The multimodal interactive closed-loop training system for patients with autism spectrum disorder according to claim 4, characterized in that: The time threshold includes three time gradients: 3s, 2.4s, and 1.8s.

7. The multimodal interactive closed-loop training system for patients with autism spectrum disorder according to claim 1, characterized in that: The method for processing the image data and audio data in the database through the data preprocessing module is as follows: S1, train an image difficulty classification model; S2, use the trained image difficulty classification model to classify the new image set and distribute the new image set into three data sets according to the difficulty type; S3, uses a random algorithm to select several pictures from the same type of data set for stitching; S4, using a random algorithm to select one of the stitched images as the target image; S5, packaging the audio data of the target image obtained in S4 and the multiple images spliced ​​in S3 into a data combination; S6, repeating steps from S3 to S5, can obtain several data combinations of three difficulty gradients.

8. A multimodal interactive closed-loop training system for patients with autism spectrum disorder according to claim 7, characterized in that: In S1, the steps for training the image difficulty classification model are as follows: S101, collecting and preparing image data for training the model, preprocessing the image data, and dividing the preprocessed data into a training set, a validation set, and a test set; S102, select the CNN convolutional neural network model, select the cross entropy loss function and Adam optimizer, and set the learning rate, momentum, and weight decay parameters; S103, input the training set into the model for training; use the validation set to monitor the training process of the model to avoid overfitting; adjust the model parameters and training strategy according to the performance of the validation set; S104, use the test set to evaluate the performance of the model; analyze the errors of the model and make improvements.

Images