Autism assisted assessment system and method based on deep learning

Abstract

The present invention provides an auxiliary assessment system and method for autism based on deep learning. The system includes a data acquisition and feature extraction unit, a first neural network, a second neural network, a third neural network and a result output unit, wherein the data acquisition and feature extraction unit and the result output unit are respectively connected to the first neural network, the second neural network and the third neural network, wherein the data acquisition and feature extraction are used to collect the eye movement data of the subject watching the video, and obtain a heat map, a focus map and a scanning path map; the first neural network inputs the heat map to obtain a first classification result; the second neural network inputs the focus map to obtain a second classification result; the third neural network inputs the scanning path map to obtain a third classification result; the result output unit collects the first classification result, the second classification result and the third classification result to obtain the autism detection result of the subject. The present invention improves the prediction efficiency and prediction accuracy of autism.

Description

Technical Field

[0001] The present invention relates to the technical field of autism assessment, and in particular to an autism auxiliary assessment system and method based on deep learning. Background Art

[0002] Autism spectrum disorder (ASD), also known as "autism spectrum disorder", "autism" and "autism", is a heterogeneous neurodevelopmental disorder with social disorders and stereotyped behaviors as the core disorders. The incidence of ASD children has increased year by year worldwide and has become a social public health issue. According to the Autism and Developmental Disabilities Monitoring Network of the Centers for Disease Control and Prevention, 1 in 68 people in the United States suffers from autism, so the attention paid to ASD in today's society has increased significantly in the past few years. Rehabilitation training is the main treatment for children with ASD. More and more examples show that the earlier the intervention begins, the better the prognosis. Through early diagnosis and intervention of ASD children, the symptoms of ASD children can be alleviated, their potential can be maximized, their corresponding functional levels can be improved, and they can grow up healthily like normal children.

[0003] However, the current diagnosis of ASD lacks reliable biochemical diagnostic markers or imaging evidence, and is highly heterogeneous. The core symptoms of ASD, social communication and interaction disorders, are impaired to varying degrees and present in a variety of ways. In particular, it is difficult to promptly identify and correctly diagnose children with mild to moderate ASD, which can easily lead to missed or misdiagnosed children being labeled as “problem children” after they enter mainstream childcare institutions. They are easily blamed by their parents, punished by their teachers, and alienated by their classmates, which can lead to behavioral and emotional problems and are very detrimental to the child’s prognosis. Therefore, there is an urgent need for convenient and highly reliable methods for auxiliary diagnosis, which can facilitate early correct diagnosis and timely intervention, and reduce the increasingly heavy burden of rehabilitation costs for children with autism spectrum disorders.

[0004] The diagnostic criteria for children with ASD mainly include the American Psychiatric Association's Diagnostic and Statistical Manual of Mental Disorders and the World Health Organization's diagnostic criteria for mental and behavioral disorders. The current mainstream assessment and prediction method is to use the international mainstream diagnostic tools (Autism Diagnostic Observation Scale and Autism Diagnostic Interview Scale) based on the diagnostic criteria, targeting language communication disorders, social interaction disorders and repetitive stereotyped behaviors, and to diagnose autism through questionnaires and interviews combined with the child's usual growth and development history, medical history and mental examination. When direct or indirect observation based on behavioral signs and symptoms is no longer applicable, from the perspective of social cognition, the analysis of ASD visual characteristics around social disorders, a core specific symptom of ASD, is of special significance. Recent studies have provided evidence of different eye movement patterns in ASD individuals, and studied how ASD individuals scan faces through eye tracking technology. These studies unanimously agree that autistic individuals have less visual attention to faces than normal children. At present, the medical field has begun to study eye gaze behavior and use it as one of the diagnostic criteria for childhood autism. From this perspective, a class of methods has been formed that uses eye trackers to track the eye gaze behavior of ASD, and uses the eye movement data of ASD to analyze and extract features to evaluate and predict autism. On the other hand, with the development of research on brain functional imaging technology, some methods for diagnosing ASD with the help of brain functional imaging technology have emerged. The resting state functional magnetic resonance data is obtained through a nuclear magnetic resonance scanner, or the electroencephalogram (EEG) signals are collected by electroencephalogram (EEG) equipment. The collected data is pre-processed and functionally processed to obtain the required features to train the classification model for the classification of autistic and normal children. In addition, more and more studies use engineering technology to extract relevant features from multimodal signals from various aspects for the evaluation and prediction of autism. For example, by watching the facial images corresponding to the pre-made videos of the autistic testers and the changes in facial temperature, heart rate and breathing when watching the images; or using visual cameras to assist in judging the subject's response to language; or collecting multi-channel audio and video multimodal data from multiple RGB-D camera perspectives of the subjects, evaluators and props during the laughter test; or extracting the EEG signals, myoelectric signals, eye contact signals, skin galvanic response signals, body temperature data, breathing rate, etc. of the subjects in different emotional states based on physiological signals, or even directly using the subject's facial expressions to extract facial features for auxiliary judgment. In short, a variety of multimodal signals and features that can be used have been tried for auxiliary diagnosis of children with autism.

[0005] In the prior art, the classification and auxiliary diagnosis method of autism using eye movement data of ASD usually focuses on watching static facial images or static images in visual tracking tasks, and uses an eye tracker or a glasses-type eye tracker to obtain the eye movement data or eye gaze data of the observed person. This is then used to perform complex manual feature extraction work, assisted by the training of traditional machine learning classification models such as support vector machines (SVM) or BP neural networks to obtain a classifier model with better performance, and the children are classified and identified based on the established classification model. In the schemes using eye tracking technology for auxiliary diagnosis, most of them focus on simple static face recognition as the main research direction, and there are very few reports on auxiliary diagnosis of ASD children based on facial emotion recognition. Face recognition and emotion perception disorders are the core problems of social disorders in ASD children, and the common facial emotion recognition defects in patients with autism spectrum disorders are the core cause of their social and communication disorders. There are significant differences in the eye movement patterns of patients with autism spectrum disorder and the typically developing population under social and facial stimulation conditions. Patients with autism spectrum disorder focus less on the eye area, and avoiding direct gaze may be the reason for the development of facial emotion recognition defects in patients with autism spectrum disorder. At present, the research on facial emotion recognition disorders in patients with autism spectrum disorder mainly uses eye tracking technology. Studies have shown that ASD children have eye movement patterns that are different from normal children in face recognition and emotion perception, and this feature exists in ASD children with symptoms ranging from mild to severe. The specific facial processing mode of patients with autism spectrum disorder is mainly to select and extract information from facial stimuli, which is suboptimal for the task of emotion recognition.

[0006] In summary, the existing technical solutions mainly have the following problems: 1) The stimulus materials of the eye movement-based solution are too typical and out of touch with real life scenes. In the current eye movement-based auxiliary diagnosis solution, the stimulus materials provided are all static pictures, which are out of touch with real life scenes and cannot truly evaluate the emotional recognition ability of ASD children in real social communication and interaction. 2) It is not suitable for auxiliary diagnosis of children aged 6-18 months. The current autism spectrum disorder scales mostly have age ranges. For autistic patients, there are often problems such as difficult questions or age not within the scale assessment range. Moreover, for younger children, it is impossible to conduct questionnaire surveys and interviews with them. They can only rely on the daily observations of their parents and elders, interviews with doctors, and behavioral verification to diagnose, which is easy to cause missed diagnosis and misdiagnosis. The current methods of obtaining eye movement data with glasses-type eye trackers, using multimodal data such as EEG for data collection, and using facial expression reactions are too difficult to implement for children aged 6-18 months. 3) It requires cumbersome manual feature extraction. The current engineering technology solutions for ASD auxiliary diagnosis, whether it is using eye tracking technology to obtain the subject's eye movement data, or obtaining the subject's facial features in face recognition related tasks, or even obtaining the subject's EEG signals, EMG signals and other multimodal signals, all require complex manual feature extraction and then training the classifier module for diagnosis and classification. The different forms of data obtained must be subjected to sophisticated and complex feature extraction to provide high-accuracy classification performance in the later stage. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an autism auxiliary assessment system and method based on deep learning, which combines eye movement technology and deep learning to predict and assess autism.

[0008] According to a first aspect of the present invention, a deep learning-based autism auxiliary assessment system is provided. The system includes a data acquisition and feature extraction unit, a first neural network, a second neural network, a third neural network and a result output unit, wherein the data acquisition and feature extraction unit and the result output unit are respectively connected to the first neural network, the second neural network and the third neural network in communication, wherein: the data acquisition and feature extraction unit is used to collect the eye movement data of the subject watching the video, and obtain the corresponding heat map, focus map and scanning path map, the heat map is used to characterize the dynamic changes of the time and position of the gaze point, the focus map is used to characterize the dynamic changes of the gaze position and time, and the path scanning map continuously displays the gaze point position and each gaze time information point by point; the first neural network is used to input the heat map to obtain the first classification result; the second neural network is used to input the focus map to obtain the second classification result; the third neural network is used to input the scanning path map to obtain the third classification result; the result output unit collects the first classification result, the second classification result and the third classification result to obtain the autism detection result of the subject.

[0009] In one embodiment, the first neural network, the second neural network and the third neural network have the same or different structures.

[0010] In one embodiment, the first neural network, the second neural network and the third neural network have the same structure, including an input layer, a first convolutional layer, a second pooling layer, a third convolutional layer, a fourth pooling layer, a fifth convolutional layer, a sixth fully connected layer, a seventh fully connected layer and an output layer.

[0011] In one embodiment, the activation functions of the first convolution layer, the second pooling layer, the third convolution layer, the fourth pooling layer, the fifth convolution layer, and the sixth fully connected layer of the first neural network, the second neural network, and the third neural network are ReLU nonlinear activation functions, the activation function of the seventh fully connected layer is Softmax activation function, and the number of neurons in the output layer is 4, corresponding to the four categories of health, mild autism symptoms, moderate autism symptoms, and severe autism symptoms, respectively.

[0012] In one embodiment, the result output unit uses a simple voting method to combine the first classification result, the second classification result and the third classification result to give a final prediction result.

[0013] In one embodiment, eye movement data from each subject is collected non-invasively using an eye tracker.

[0014] In one embodiment, the heat map displays the dynamic changes of the time and position of the gaze point with the warmth of the color, and the focus map displays the dynamic changes of the gaze position and time with the brightness.

[0015] According to a second aspect of the present invention, a method for assisting autism assessment based on deep learning is provided. The method comprises the following steps:

[0016] Collect the eye movement data of the subject watching the video, and obtain the corresponding heat map, focus map and scanning path map. The heat map is used to represent the dynamic changes of the time and position of the gaze point, the focus map is used to represent the dynamic changes of the gaze position and time, and the path scanning map continuously displays the gaze point position and each gaze time information point by point;

[0017] Inputting the heat map, the focus map and the scanning path map distribution into the trained first neural network, the second neural network and the third neural network to obtain a first classification result, a second classification result and a third classification result respectively;

[0018] The first classification result, the second classification result and the third classification result are combined to obtain an autism test result of the subject.

[0019] Compared with the prior art, the advantages of the present invention are: starting from the common facial emotion recognition defects of ASD patients, aiming at the research goal of ASD eye movement screening, compared with static picture stimulation, dynamic scenes with daily speech expression are adopted as stimulation materials, and through the stimulation design of dynamic videos, the emotional recognition reactions and eye movement data of the subjects in real social interactions are extracted, thereby improving the authenticity and reliability of auxiliary diagnosis; since the eye movement technology is non-invasive and the subjects do not need to wear any devices, and the stimulation materials can be appropriately adjusted according to the age group of the subjects, it can be suitable for autistic patients of different age groups and different development levels, especially for children aged 6-18 months, and has a wider scope of application; most of the existing methods for auxiliary diagnosis using eye movement technology solutions undergo tedious manual feature extraction and then perform traditional machine learning classification model training for result prediction, which is time-consuming and labor-intensive. On the basis of eye movement tracking technology, the present invention uses convolutional neural networks in deep learning to perform automatic feature extraction and perform neural network classifier training for result prediction, thereby improving prediction efficiency and prediction accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The following drawings are only used to illustrate and explain the present invention, and are not used to limit the scope of the present invention, wherein:

[0021] Figure 1 is a schematic diagram of an autism auxiliary assessment system based on deep learning according to an embodiment of the present invention;

[0022] Figure 2 is a diagram of a neural network structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution, design method and advantages of the present invention clearer, the present invention is further described in detail by specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0025] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0026] Patients with autism spectrum disorder generally have defects in facial emotion recognition, which is the core cause of their social and communication disorders. At present, the research on facial emotion recognition disorders in patients with autism spectrum disorder mainly uses eye tracking technology. There are significant differences in the eye movement patterns of patients with autism spectrum disorder under social and facial stimulation conditions compared with the typically developing population. Patients with autism spectrum disorder pay less attention to the eye area, and avoiding direct gaze is the reason for the development of facial emotion recognition defects in patients with autism spectrum disorder. There is evidence that the high level of arousal associated with direct gaze in patients with autism spectrum disorder is related to avoiding direct gaze and is related to more severe impairment of social skills. The specific facial processing mode of patients with autism spectrum disorder is mainly to select and extract information from facial stimuli, which is suboptimal for the task of emotion recognition.

[0027] According to the association between the features of eye movement data and patients with autism spectrum disorders, the present invention provides an efficient, convenient, non-invasive, and low-cost ASD auxiliary diagnosis method based on eye movement technology for face recognition and emotion perception tasks. In short, the embodiment of the present invention includes the following steps: guiding the subject to complete the video viewing task during the data collection process, requiring the subject to focus on what he hears and sees, and using an eye tracker to record the subject's eye movement data while the subject watches the video clip; completing data preprocessing for the obtained eye movement image data; then, combining the convolutional neural network (CNN) in the deep learning method to perform automatic feature extraction and obtain a neural network classifier through model training, and finally achieving auxiliary diagnosis of ASD. The present invention combines eye movement tracking data with a deep learning algorithm to effectively extract the specific facial processing patterns of patients with autism, thereby achieving auxiliary diagnosis of patients with mild to moderate autism.

[0028] Specifically, see Figure 1 As shown, the deep learning-based autism auxiliary assessment system provided by an embodiment of the present invention includes a data acquisition and feature extraction unit 110, a neural network for classification training and a result output unit 120 connected in sequence, wherein neural network 1, neural network 2 and neural network 3 are illustrated.

[0029] Combination Figure 1 ,The present invention mainly includes three processes, namely data collection and feature extraction, classifier training and ,result prediction, which will be introduced in detail below.

[0030] 1) Data collection and feature extraction

[0031] In order to achieve auxiliary diagnosis suitable for different age groups based on the different eye movement patterns of ASD children and normal children in the process of facial emotion recognition, and at the same time not limited to the influence of local medical level, it can also accurately evaluate patients with autism spectrum disorders when doctors lack professional ability and experience. During the data collection process, the present invention uses an eye tracker that is convenient for young children and does not require high professionalism of doctors to collect eye movement data with dynamic daily life scenes as stimulus materials for classification.

[0032] Aiming at the core issues of face recognition and emotion perception disorders in social disorders of ASD children, and focusing on the common facial emotion recognition defects in ASD patients, facial stimuli that can reflect their specific eye movement patterns are selected. Compared with the static facial expression stimuli commonly used in research, dynamic video emotion stimuli are selected in the data collection process in the present invention. Compared with static picture stimuli, dynamic video stimuli with daily verbal expressions are more in line with daily life scenes, and can more truly evaluate the emotion recognition ability of ASD children, because the communication that occurs in real social interactions is not always well perceived. At the same time, the stimulus materials can be dynamically adjusted according to the different age groups of the subjects in order to be suitable for children of different age groups.

[0033] In one embodiment, the stimulus material in the data collection process is composed of 20 movie clips provided by the Chinese Natural Emotional Audio-Visual Database (CHEAVD), which is intended to provide Chinese resources for studying multimodal and multimedia interactions. For example, six typical and relatively complete emotional stimulus videos are selected for analysis, which are composed of three positive emotion videos and three negative emotion videos. The duration of the video is between 3 seconds and 9 seconds. The data collection process uses the RED250 eye tracker of the German SMI company to collect the eye movement data of each subject in a non-invasive manner. The device has been integrated into a 22-inch widescreen display panel with a resolution of 1280*1024 pixels. The sampling frequency is 60Hz and the accuracy is 0.4 degrees. The degree of freedom of the subject's head movement is 40*20 cm at a distance of 70 cm. During the collection process, the experimental design software Experiment Center of the German SMI company is used to record the online eye movement data, and the SMI data analysis software BeGaze is used to perform offline data analysis. Before the formal experimental task, the subjects were first trained to enable them to understand the tasks in this experiment. After understanding the tasks, a pre-experiment was conducted using a Samsung tablet ST800 to ensure that the experimenter was familiar with the entire process of the experiment. The participants were then asked to sit about 60-80 cm away from the test screen until the eye tracker could stably detect the participant's pupil. During the experiment, the participants could not be intervened to avoid any attention bias. Specifically, a five-point calibration of the eye movement was first performed: the subjects were asked to look at the calibration points in the four corners and the middle of the screen in turn. The calibration part was passed only when the average error of all five calibration points did not exceed 1 degree of visual angle. During the formal experimental task phase, 20 test videos were played randomly, and participants were allowed to watch each video multiple times because they did not understand the content displayed in it.

[0034] After collecting the eye movement data, the matching data analysis software SMI BeGaze is used to process and output the heat map, focus map, and scan path map of each video clip that the subject watches. The heat map, for example, uses the warmth of the color to display the dynamic changes in the time and position of the gaze point, that is, the closer it is to the color on the right side of the color bar in the data analysis software, the longer the time of fixation on the area. The scan path map, for example, continuously displays the location of the gaze point and the time of each fixation. The focus map, for example, uses brightness to display the dynamic changes in the gaze position and time.

[0035] In summary, in the task design of the present invention, dynamic scenes with daily verbal expressions are used as stimulus materials around the core issues of face recognition and emotion perception disorders in social disorders of ASD children. The real life scenes can better reflect the natural reactions of the subjects in social interactions and the authenticity of the eye movement data. In addition, eye tracker tracking is non-invasive, and the subjects do not need to wear any devices when using the eye tracker for data collection, so it will not cause discomfort to the subjects, especially younger children. Moreover, the experimental task stimulation can be flexibly adjusted according to the age of the subjects, and it is also convenient to operate and diagnose for children aged 6-18 months.

[0036] 2) Classification training

[0037] After extracting the heat map, focus map and scanning path map reflecting the subject's eye movement characteristics, a neural network classifier for predicting and evaluating autism is obtained through neural network training and learning.

[0038] See also Figure 2 As shown, the embodiment of the present invention adopts a convolutional neural network, using a design similar to the LeNet structure, and the structure of the entire neural network includes an input layer (input), 3 convolutional layers (i.e., conv1, conv2, conv3), 2 layers of maximum pooling layers (i.e., max pooling1, max pooling2), 2 layers of fully connected layers (i.e., fc1 and fc2) and an output layer (output). Specifically, the first is a data input layer, and the input images are heat maps, focus maps, and scanning path maps obtained by the eye movement data through the analysis software, and the size of the input image is uniformly normalized to 1024*1024. The first layer is a convolutional layer, the second layer is a maximum pooling layer, the third layer is a convolutional layer, the fourth layer is a maximum pooling layer, the fifth layer is a convolutional layer, the sixth layer and the seventh layer are fully connected layers, and the eighth layer is an output layer. The activation function of the first six layers is a ReLU nonlinear activation function, and the activation function of the seventh layer is a Softmax activation function. The number of neurons in the output layer can be set to 4, corresponding to four categories of health, mild autism symptoms, moderate autism symptoms, and severe autism symptoms.

[0039] After designing the neural network structure, the training of the neural network for auxiliary diagnosis of ASD was started. Specifically, the number of nodes of the input layer, hidden layer and output layer neurons and the size of the convolution kernel were set, and the weight matrix was randomly initialized, including the weight matrix from the input layer to the hidden layer, between the hidden layer and the hidden layer, and between the hidden layer and the output layer. The heat map, focus map and scanning path map of the eye movement data of each video clip were input as the input of the neural network. Based on the initialized weight matrix, the weight matrix between each layer of the neural network was obtained by forward propagation and back propagation algorithms and gradient descent method. Finally, the loss function of the output layer is the cross entropy loss function.

[0040] When testing the effect of the present invention, a cross-validation method is used, and the classification result confusion matrix is ​​used to judge the effect of the neural network classifier according to the receiver operating characteristic (ROC) curve and the area under the ROC curve AUC (Area Under ROC Curve). A method similar to the bootstrap aggregating (Bagging) method in ensemble learning is used to combine the classification results of the three classifiers according to the three input images of the eye movement data to give the final classification result.

[0041] 3) Classification result prediction

[0042] ASD classification prediction is performed based on the trained neural network classifier model. First, three convolutional neural network classifiers are trained according to different input data. These three classifiers use the heat map, focus map and scanning path map of eye movement data as input, and output four categories corresponding to health, mild autism symptoms, moderate autism symptoms and severe autism symptoms. Then, based on the predicted output of the three classifiers, a simple voting method is used to combine them to give the final prediction result. The final four results are still one of the above four categories.

[0043] It should be noted that the neural network involved in the embodiments of the present invention may have the same or different structures, for example, more convolutional layers, fully connected layers, average pooling or maximum pooling methods may be used, and the classification results are not limited to the above four categories. In addition, the neural network and the result output unit may be implemented in software or hardware, such as a hardware processor or logic circuit.

[0044] In order to further verify the effect of the present invention, after a comparative experimental study of ten ASD children and nineteen normal children, it was found that the data analysis results of the present invention were consistent with the results shown in the research. The AOI (area of ​​interest) method is widely used in eye movement analysis. AOI aims to measure the facial areas of interest where the eyes are fixated, usually including the eyes, nose and mouth, and then count the frequency and time of the eyes fixating on these areas. In the preliminary experiments of the present invention, ASD children had relatively less fixation time and number of fixations in AOI compared with normal children. More specifically, ASD children looked at the body and objects much more than they looked at the eyes, and the heat map analysis and focus map analysis of their eye movement data illustrated the same problem. The present invention performs auxiliary diagnosis of ASD based on different eye movement patterns.

[0045] In summary, the present invention aims at the deficiencies in the current ASD auxiliary diagnosis technology and the main problems in ASD screening assessment and clinical application, and establishes a technical solution that is efficient, convenient, easy to implement and popularize, and suitable for the auxiliary diagnosis of ASD in young children. Focusing on the core issues of social disorders in ASD children: face recognition and emotion perception disorders, the eye movement patterns of ASD children in face recognition and emotion perception are different from those of normal children. With the help of infrared visual tracking technology (eye movement technology can be used for younger and younger ages, and the current eye movement technology can already be used to study infants as young as 3 months old), combined with deep learning methods, the complicated and time-consuming manual feature extraction work is eliminated, and auxiliary diagnosis of ASD is achieved, and the early diagnosis and discovery of ASD patients is improved, which opens an effective time window for treatment, realizes early intervention for related patients, and reduces the economic burden on society and families, thereby achieving true "early detection, early intervention, and early treatment."

[0046] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be executed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0047] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0048] Computer readable storage medium can be a tangible device that holds and stores instructions used by an instruction execution device. Computer readable storage medium can include, for example, but is not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a protruding structure in a groove on which instructions are stored, and any suitable combination thereof.

[0049] The embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A deep learning-based autism auxiliary assessment system, characterized in that: It includes a data acquisition and feature extraction unit, a first neural network, a second neural network and a third neural network, and a result output unit, wherein the data acquisition and feature extraction unit and the result output unit are respectively connected to the first neural network, the second neural network and the third neural network in communication, wherein: The data acquisition and feature extraction unit is used to collect eye movement data of the subject watching the dynamic video, and obtain the corresponding heat map, focus map and scanning path map. The heat map is used to characterize the dynamic changes of the time and position of the gaze point, the focus map is used to characterize the dynamic changes of the gaze position and time, and the scanning path map continuously displays the gaze point position and each gaze time information point by point; The first neural network is used to input the heat map to obtain a first classification result; The second neural network is used to input the focus image to obtain a second classification result; The third neural network is used to input the scanning path diagram to obtain a third classification result; The result output unit combines the first classification result, the second classification result and the third classification result to obtain an autism test result of the subject; The first neural network, the second neural network and the third neural network have the same structure, including an input layer, a first convolution layer, a second pooling layer, a third convolution layer, a fourth pooling layer, a fifth convolution layer, a sixth fully connected layer, a seventh fully connected layer and an output layer; Among them, the activation functions of the first convolution layer, the second pooling layer, the third convolution layer, the fourth pooling layer, the fifth convolution layer, and the sixth fully connected layer of the first neural network, the second neural network, and the third neural network are ReLU nonlinear activation functions, and the activation function of the seventh fully connected layer is Softmax activation function. The number of neurons in the output layer is 4, corresponding to the four categories of health, mild autism symptoms, moderate autism symptoms, and severe autism symptoms, respectively; The heat map uses color warmth to display the dynamic changes in time and position of the gaze point, and the focus map uses brightness to display the dynamic changes in gaze position and time.

2. The system according to claim 1, characterized in that The result output unit uses a simple voting method to combine the first classification result, the second classification result and the third classification result to give a final prediction result.

3. The system according to claim 1, characterized in that Eye movement data from each subject were collected non-invasively using an eye tracker.

4. A deep learning-based autism auxiliary assessment method, comprising the following steps: Collect the eye movement data of the subjects watching the dynamic video to obtain the corresponding heat map, focus map and scanning path map, the heat map is used to represent the dynamic changes of the time and position of the gaze point, the focus map is used to represent the dynamic changes of the gaze position and time, and the scanning path map continuously displays the gaze point position and each gaze time information point by point; Inputting the heat map, the focus map and the scanning path map into the trained first neural network, the second neural network and the third neural network respectively, to obtain a first classification result, a second classification result and a third classification result respectively; Collecting the first classification result, the second classification result and the third classification result to obtain an autism test result of the subject; The first neural network, the second neural network and the third neural network have the same structure, including an input layer, a first convolution layer, a second pooling layer, a third convolution layer, a fourth pooling layer, a fifth convolution layer, a sixth fully connected layer, a seventh fully connected layer and an output layer; Among them, the activation functions of the first convolution layer, the second pooling layer, the third convolution layer, the fourth pooling layer, the fifth convolution layer, and the sixth fully connected layer of the first neural network, the second neural network, and the third neural network are ReLU nonlinear activation functions, and the activation function of the seventh fully connected layer is Softmax activation function. The number of neurons in the output layer is 4, corresponding to the four categories of health, mild autism symptoms, moderate autism symptoms, and severe autism symptoms, respectively; The heat map uses color warmth to display the dynamic changes in time and position of the gaze point, and the focus map uses brightness to display the dynamic changes in gaze position and time.

Images