Asd child gaze detection method and apparatus, and hand-eye coordination ability testing device and method

Abstract

The application discloses an ASD child gaze detection method and device, and a hand-eye coordination ability test equipment and method. The gaze detection method comprises the following steps: constructing and training a gaze detection model; collecting video data containing the face of an ASD child and inputting the video data into the trained gaze detection model to determine whether the ASD child gazes at a target; the gaze detection model comprises a spatial feature extraction module, which extracts spatial features of the video data frame by frame and outputs low-level detail features and high-level semantic features; a time feature enhancement module, which enhances the time features of the low-level detail features and the high-level semantic features output by each frame; and a gaze detection result generation module, which uses an SSD detection framework to output an eye detection frame of a current frame, and then uses a memory enhancement process to gather key information from long-term frames and short-term frames to enhance the eye detection frame of the current frame, so as to obtain the position of the eye detection frame of the current frame and a gaze contact classification result. In the test of the application, the tester does not need to wear and participate in calibration of any equipment.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine-assisted medical treatment, and particularly relates to an ASD child gaze detection method and device and a hand-eye coordination ability testing equipment and method. BACKGROUND

[0002] Autism spectrum disorder (ASD) is one of the most common neurodevelopmental disorders in the world, characterized by highly repetitive behaviors, social interaction disorders, speech and non-verbal communication disorders, narrow interests and stereotyped impressions. Machine-assisted treatment simplifies and speeds up the diagnosis and treatment process by using artificial intelligence (AI) for information processing.

[0003] ASD patients show lower hand-eye coordination efficiency, which is particularly evident when pointing to a target of fixation. Hand-eye coordination, i.e., guiding both hands to perform actions according to the stimuli received by the eyes, is an important and complex cognitive ability. For example, M. D. Samad et al. (M. D. Samad, N. Diawara, J. L. Bobzien, J. W. Harrington, M. A. Witherow and K. M. Iftekharuddin, “A Feasibility Study of Autism Behavioral Markers in Spontaneous Facial, Visual, and Hand Movement Response Data,” in IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 26, no. 2, pp. 353-361, Feb. 2018, doi: 10.1109 / TNSRE.2017.2768482.) proposed a hand-eye coordination test scheme: using an eye tracking system composed of a camera and a Mirametrix S2 eye tracker, the hand-eye coordination skill was evaluated by analyzing the correlation between eye fixation data and the position of the cursor when the subject moved the mouse. The eye tracker used in this study required the subject to visually follow five targets in turn and needed to repeat multiple times until the calibration was successful. C. Casellato et al. (C. Casellato, M. Gandolla, A. Crippa and A. Pedrocchi, “Robotic set-up to quantify hand-eye behavior in motor execution and learning of children with autism spectrum disorder,” 2017 International Conference on Rehabilitation Robotics (ICORR), 2017, pp. 953-958, doi: 10.1109 / ICORR.2017.8009372.) proposed a robotic device composed of a T60 (Tobii) eye tracker, a haptic robotic manipulandum and a control PC. By capturing the gaze data and hand data generated by the subject when capturing and moving the target, the hand-eye behavior of ASD children in motor execution and learning was quantified. Among them, the T60 (Tobii) eye tracker required the subject to complete about 30 seconds of calibration by looking at 9 circles in the screen.

[0004] Eye contact can establish and recognize partnerships, express interest and attention in social processes. In the diagnostic criteria of DSM-5 ASD, less eye contact in social processes is usually the focus of early screening and treatment of ASD. Zhao, Z. et al. Characteristics of Visual Fixation in Chinese Children with Autism During Face-to-Face Conversations. J Autism Dev Disord (2021). doi: / 10.1007 / s10803-021-04985-y. in the study of visual attention in interpersonal interaction of Chinese ASD patients, the fixation behavior of the subjects during the structured conversation was recorded by using eye tracker Tobii Pro Glasses 2. Among them, Tobii Pro Glasses 2 is a pair of glasses-like eye tracker, which needs to be worn by the subjects.

[0005] At present, the fixation estimation instruments used in hand-eye coordination test and eye contact test of ASD children are almost eye tracking devices. Eye tracking devices have the following disadvantages: (1) eye tracking devices are too expensive; (2) the devices need to be worn and calibrated by the patients. This can be a huge challenge for subjects with poor compliance, attention dispersion or fatigue problems, and this can affect the yield and quality of data; (3) since the eye tracker only provides the fixation point in the captured video recording, manual region of interest annotation must be performed on the video in order to identify the fixation target; (4) if the test project only requires the patient's eye to fixate on a certain point, it is unnecessary to use eye tracking.

[0006] Chong, E. et al. Detection of eye contact with deep neural networks is as accurate as human experts. Nat Commun 11, 6386 (2020). doi:10.1038 / s41467-020-19712-x. pioneered a new eye contact test: a pair of glasses with a small camera embedded in the nose bridge is worn by the accompanying person to record the face of the ASD patient, and then a classifier is used to classify whether there is eye contact in each frame of the video record. The classifier uses ResNet-50 as the backbone network and uses the method of transfer learning to train. In addition, the output of the classifier needs to be set threshold and time smoothing processing to reduce the noise caused by face detection failure, motion blur and blinking. Among them, the time smoothing processing refers to replacing the single frame classification result with the average value of the classification results of all frames in the time window. The decision threshold of the classifier and the window size of the time smoothing processing need to be determined by maximizing the detection accuracy while minimizing the difference between the model and the artificial classification.

[0007] The eye contact test scheme proposed by Chong, E. et al. solves the above problems to some extent: 1) regarding the price, the scheme only needs to install a small camera on the test device or ordinary glasses; 2) regarding wearing, the device is worn by the accompanying person; 3) regarding manual processing, the scheme does not need to manually annotate the region of interest. However, this scheme has the following disadvantages: 1) this scheme uses a classification network to classify the video picture. The classification network potentially requires that only the eyes of a subject in the video appear in the picture, and the eye information of others will interfere with the classification result, and also limits the expansion of application. Alternatively, the cropped face region can be input into the classification network, but this requires additional processing of the input video; 2) the classification network does not take into account that the eye contact classification problem belongs to a typical small target problem, and the classification accuracy of eye contact still has room for improvement; 3) for video frames where face detection fails, motion blur and blinking occur, the post-processing step does not significantly improve the classification result, but it affects the classification result of normal video frames, so the overall classification level does not improve significantly. SUMMARY

[0008] To solve the above technical problems in the prior art, the present application provides a hand-eye coordination test method and device for ASD children, and a hand-eye coordination ability test equipment and method.

[0009] The technical scheme of the present application is as follows:

[0010] An ASD child gaze detection method, comprising: constructing and training a gaze detection model; collecting video data containing the face of an ASD child and inputting the video data into the trained gaze detection model to determine whether the ASD child gazes at a target through the gaze detection model;

[0011] The gaze detection model comprises:

[0012] A spatial feature extraction module, which extracts spatial features of the video data frame by frame, and outputs low-level detailed features and high-level semantic features in a shallow layer;

[0013] A temporal feature enhancement module, which uses a spatio-temporal long short-term memory network based on an attention mechanism to perform temporal feature enhancement on the low-level detailed features and the high-level semantic features output by each frame in the time domain;

[0014] A gaze detection result generation module, which uses an SSD detection framework to output an eye detection frame of the current frame based on the low-level detailed features and the high-level semantic features after temporal feature enhancement, and uses a memory enhancement process to gather key information from long-term frames and short-term frames to enhance the eye detection frame of the current frame, so as to obtain the eye detection frame position and the gaze contact classification result of the current frame.

[0015] The spatial feature extraction module extracts spatial features of the video data using a Darknet-53 network.

[0016] Further, in the Darknet-53 network, the video frame is first passed through a convolution module (Conv2D), and then sequentially passed through five residual module (Res-Block) groups.

[0017] In the residual module group, a feature layer one is obtained by performing a convolution once, a feature layer two is obtained by performing a convolution twice on the feature layer one, and the output result is obtained by adding the feature layer one and the feature layer two.

[0018] The output feature of the third residual module group is a low-level detailed feature, and the output feature of the fifth residual module group is a high-level semantic feature.

[0019] In the Darknet-53 network, batch normalization and LeakyReLU function activation are performed after each convolution.

[0020] The temporal feature enhancement module uses an AST-LSTM network to perform temporal feature enhancement on the low-level detailed features and the high-level semantic features, respectively.

[0021] Further, the AST-LSTM network comprises a feature preprocessing module, an attention mechanism module, and an ST-LSTM network.

[0022] The size of the to-be-input feature map is reduced by the feature preprocessing module to reduce the dimension of the input timing network, and then the reduced feature map is converted into an attention map using an attention mechanism module. The process is described by the following formula:

[0023]

[0024] In the formula, is the attention map, x t is the reduced feature map, h t-1 is the hidden state of the previous frame, w i is the weight matrix of the full connection layer, and sigma is the sigmoid function.

[0025] The converted attention map is used to replace the feature map in the inter-frame transmission of the ST-LSTM network. The process is described by the following formula:

[0026]

[0027]

[0028]

[0029]

[0030]

[0031]

[0032] In the formula, i t , g t , f t , o t are the input gate, input adjustment gate, forget gate and output gate, respectively. The attention map is used to replace the feature map in the calculation process; omega ** is the weight matrix of the full connection layer, b * is the bias vector of the full connection layer, w 1×1 is the weight matrix of the convolution layer, is the time memory state of the previous frame.

[0033] is the time memory state, is the hidden state, is the space-time memory state. and are generated by the above gates and are responsible for the information transmission between frames, thereby realizing the information transmission of the attention map between frames. are responsible for the information transmission between layers, and the corresponding gates are calculated by the feature map to maintain the degree of restoration of spatial information.

[0034] Since the proportion of eye information in the video frame is small, the attention map output by the above attention mechanism is used to replace the feature map for inter-frame transmission, which can effectively suppress the background information and improve the information mining ability of the AST-LSTM network.

[0035] In the gaze detection result generation module, based on the low-level detail features and high-level semantic features enhanced by time characteristics, the SSD detection framework is used to output the eye detection box of the current frame, including:

[0036] (i) The low-level detail features and high-level semantic features enhanced by time characteristics are used to obtain the position bias information and class confidence of the default box through convolution, and the candidate box is generated accordingly;

[0037] (ii) For the two categories of eye contact and non-eye contact, the candidate box with a class confidence higher than the threshold is selected, and then the position bias information and class confidence are used to perform a second round of screening on the candidate box through the non-maximum suppression operation (NMS) to obtain the eye detection box.

[0038] In order to fully utilize the long-term frame and short-term frame to improve the detection box information, preferably, a memory enhancement process is used to gather key information from the long-term frame and short-term frame to enhance the eye detection box of the current frame, including:

[0039] (I) The memory enhancement process (MEP) uses a multi-head attention algorithm to gradually enhance the eye detection box information of each frame to obtain the eye detection box position and eye contact classification result of the current frame;

[0040] (II) The memory storage module (MSM) is used to automatically update the enhanced frame information in real time, and the new frame information is stored in the MSM after enhancement, while the earliest frame information is removed. The frame information in the memory storage module includes a number of early frames and a number of short-term frames.

[0041] MSM is used to flexibly access frame information, and the main idea is to sequentially store the enhanced information of each frame and directly call them. MSM always exists in the frame sequence, and the new frame information is stored in MSM after enhancement, while the earliest frame information is removed. In order to balance the detection accuracy and speed, the storage capacity of MSM is determined to be 30 frames through experiments, of which 25 frames are early frames and 5 frames are short-term frames. In addition, the long-term frame is 15 frames, which is obtained by rearranging the early frames and taking the last 15 frames.

[0042] MEP mainly includes short-term frame enhancement and current frame enhancement. The multi-head attention mechanism is integrated in the enhancement process to realize deep mining of frame information.

[0043] Step (I) includes:

[0044] (I-1) Shuffle and randomly extract early frames from the memory storage module to obtain the long-term frame set S. l ;

[0045] (I-2) Using S l The long-term frames in the short-term frame set S s The detection boxes in the image are enhanced one by one to obtain an enhanced short frame set. The formula is described as follows:

[0046]

[0047] In the formula, This refers to the classification information of the detection boxes in the short frame set; M represents the classification information of the detection boxes in the long-term frame set; n It is a linear transformation matrix; for and The relevant weights; N is the number of attention heads in the multi-head attention mechanism; J is the number of detection boxes in the long-term frame set; It utilizes long frame B d The classification information in the set is used for short frame detection boxes The process of enhancement;

[0048] (I-3) use Enhanced short-term frames in the current frame B c detection box Each frame is augmented sequentially to obtain the eye detection bounding box position and gaze contact classification result for the current frame. The formula is described as follows:

[0049]

[0050]

[0051] In the formula, This provides the classification information for the detection bounding boxes in the current frame. To enhance the classification information of detection boxes in short-term frame sets; for and The relevant weights; I is the enhancement short frame The number of bounding boxes in the set; N is the number of attention heads in the multi-head attention mechanism; M n It is a linear transformation matrix; It utilizes enhanced short frames The classification information in the set is used to detect bounding boxes in the current frame. The process of enhancement;

[0052] concat is the Concat function; To enhance short frames The position information of the detection box in the collection. It is the position information of the detection box to be enhanced in the current frame; for and The relevant weights; It utilizes enhanced short frames The location information in the set is used to detect bounding boxes in the current frame. The process of enhancement.

[0053] Similar to short-term frame enhancement, classification information enhancement uses the Concat function to aggregate all mined location information.

[0054] Preferably, the gaze detection model is trained in an end-to-end manner, and the loss function used is:

[0055]

[0056] In the formula, L loca and L conf These are the localization loss function and confidence loss function built into the SSD detection framework, respectively; α, β, and γ are the weights of the loss; N is the batch size of the input data; λ is the L2 regularization term; θ is the parameter set, which contains all the weight matrices and bias vectors in the AST-LSTM calculation process;

[0057] L at This is the attention loss function, and its formula is described as follows:

[0058]

[0059] Among them, G i and P i These represent the actual attention map and the predicted attention map, respectively.

[0060] Furthermore, the 300Videos in the Wild (300-VW) dataset was used as the training dataset.

[0061] The present invention also discloses an ASD child hand-eye coordination testing device, including: an ASD child gaze detection device, a miniature camera, a timing device, a groundhog model, a wooden stick, a test table, a hand placement platform and a test stool, wherein the test stool is set on one side of the test table and the hand placement platform is set between the test table and the test stool;

[0062] The test desktop is provided with a plurality of holes, each of which is provided with a gopher model that can be popped out of the hole or retracted into the hole, and each hole is provided with a miniature camera facing the test bench, the miniature camera is turned on when the gopher model is popped out, and the miniature camera is turned off when the gopher model is retracted into the hole; one gopher model is randomly popped out of the hole at a time, and the popped-out gopher model is retracted into the hole after being knocked;

[0063] The test bench is used for carrying the tester, and the hand placing table is used for placing the tester's hands.

[0064] The miniature camera inputs the video data containing the tester's face collected to the ASD child gaze detection device, and whether the tester gazes at the popped-out gopher model is judged through the gaze detection model, so that the time from the gopher model being popped out to the tester gazing at the popped-out gopher model is obtained.

[0065] The timing device counts and records the time from the gopher model being popped out to being knocked by the tester.

[0066] The method for testing the hand-eye coordination ability of an ASD child by using the ASD child hand-eye coordination test device comprises the following steps:

[0067] (1) guiding the child to sit on the test bench, holding the wooden stick and placing the wooden stick on the hand placing table;

[0068] (2) turning on the test device;

[0069] (3) randomly popping out the gopher model from the hole, synchronously shooting through the corresponding miniature camera, and starting the timing device from zero;

[0070] (4) inputting the video data containing the child's face collected by the miniature camera to the gaze detection model, and performing real-time detection through the gaze detection model to determine the time when the child's eyes notice the popped-out gopher model;

[0071] (5) knocking the popped-out gopher model by the child, and obtaining the time when the child knocks the popped-out gopher model by the timing device;

[0072] (6) retracting the gopher model after being knocked, turning off the corresponding miniature camera, and turning off the timing system;

[0073] (7) judging whether the test device is turned off, if yes, performing step (8), and if no, returning to step (3);

[0074] (8) evaluating the hand-eye coordination ability of the tester according to the collected information and comprehensively considering the distance between each hole and the hand placing table.

[0075] Compared with the prior art, the ASD child hand-eye coordination test device has the following beneficial effects:

[0076] (1) Compared with the gaze contact classification network, only one subject's eye is allowed to appear in the input picture, but the gaze detection network of the present application is not disturbed by other eye information.

[0077] (2) Compared with the gaze contact classification network, the subject's face needs to be cropped and then classified separately, but the gaze detection network of the present application can directly classify the gaze of multiple subjects at the same time and provide the position information of the eyes. Therefore, the application scenario of the network is more flexible, such as applied in the comparative experiment of ASD children and normal children.

[0078] (3) Compared with the gaze contact classification network, the gaze detection network of the present application can better cope with face detection failure, motion blur and blinking, etc., and realize accurate detection without affecting the detection of normal frames.

[0079] (4) Compared with the gaze contact classification network, the overall gaze detection accuracy of the gaze detection network of the present application is obviously improved.

[0080] (5) Compared with other hand-eye coordination tests, the device required for the test of the present application is not expensive.

[0081] (6) Compared with other hand-eye coordination tests, the children in the test of the present application do not need to wear and participate in calibration of any equipment.

[0082] (7) Compared with other hand-eye coordination tests, the eye information in the test of the present application does not need to be obtained by manual annotation. BRIEF DESCRIPTION OF DRAWINGS

[0083] Figure 1 is a structural schematic diagram of the gaze detection network;

[0084] Figure 2 is a schematic diagram of the memory enhancement method of the gaze detection network;

[0085] Figure 3 is a top view of the auxiliary device used in the hand-eye coordination interesting test of ASD children;

[0086] Figure 4 is a schematic diagram of the process of the hand-eye coordination interesting test of ASD children. DETAILED DESCRIPTION

[0087] The shortcomings of the eye contact test scheme proposed by Chong, E. et al. mainly focus on the gaze contact classification. To improve it, the gaze detection network is designed, and the technical problems it faces are:

[0088] 1) For shortcoming one. This network intends to use a detection network instead of a classification network, but the amount of calculation and the required calculation time are common problems of the target detection model for videos. How to maintain the balance between accuracy and speed is a technical problem faced by this gaze detection network.

[0089] 2) For shortcoming two. Because there is too little RGB information of the eyes of the subjects, there is too little discriminative feature contained in the video frame. Although deep network helps to extract semantic information, the subject's eye detail information is seriously lost after layer-by-layer convolution down-sampling. How to balance semantic information and detail information is a technical problem faced by this gaze detection network.

[0090] 3) For shortcoming three. When fusing the information of other frames, the length of time distance, the relevance of the detection frame and the utilization value of different information are factors that need to be considered. How to comprehensively and specifically use the time domain information to improve the detection effect of the problem frame while reducing the influence on the normal frame is a technical problem faced by this gaze detection network.

[0091] In the existing hand-eye coordination test of ASD patients, the eye information is obtained by an eye tracking device. A set of interesting hand-eye coordination test designed by the present application needs to solve the following technical problems in view of the shortcomings of the eye tracking device:

[0092] 1) For shortcoming one. This gaze detection network replaces the expensive eye tracking device, but how to select and use an economic and practical hardware device to complete the hand-eye coordination test is a technical problem.

[0093] 2) For shortcoming two. This gaze detection network does not require the patient to wear and calibrate, but the installation position of the eye information input camera is a technical problem.

[0094] 3) For shortcoming three and four. This gaze detection network does not require manual processing and can realize end-to-end detection, but the applicable scenario of the network is to detect the gaze of the patient's eyes on a certain point. How to design a hand-eye coordination test using only the above visual information to provide accurate test evaluation is a technical problem.

[0095] 4) The subjects of this hand-eye coordination test include but are not limited to ASD children, and the subjects usually have cognitive impairment, poor compliance, scattered attention, limited frustration tolerance and other characteristics. How to deal with the above situations is also a technical problem in designing a hand-eye coordination test.

[0096] The present application includes two parts: 1) gaze detection network, 2) ASD children's interesting hand-eye coordination test. Next, the above two parts will be described in the form of chapters.

[0097] I. Gaze detection network

[0098] The structure of the gaze detection network is shown in FIG. 3, where the horizontal solid arrows represent the information flow within the t-th frame, and the vertical dashed arrows represent the information flow between adjacent frames; the ellipses above the image and the detection box enhancement information represent the frames before the t-th frame, and the ellipses below represent the frames after the t-th frame. The network mainly consists of three parts: Figure 1

[0099] 1) Spatial feature extraction. The video frame is input into the Darknet-53 network to perform frame-by-frame spatial feature extraction, and output low-level detailed features and high-level semantic features in the shallow and deep layers. The shallow network of the backbone contains more rich eye detail features, which can be used to make up for the loss of detail features in the deep network.

[0100] 2) Temporal feature enhancement. The output features of each level are enhanced in the time domain using an attention mechanism-based spatio-temporal long short-term memory (AST-LSTM). The feature preprocessing module in the AST-LSTM is used to reduce the subsequent computational load. The attention mechanism module is used to suppress a large amount of useless background information, so that the spatio-temporal long short-term memory (ST-LSTM) module can fully mine the spatio-temporal context information.

[0101] 3) Generation of gaze detection results. The detection box is output using the SSD detection mechanism, and then the memory enhancement process (MEP) is used to gather key information from long-term frames and short-term frames to improve the detection box of the current frame. The memory storage method (MSM) is used to speed up the operation in this process.

[0102] The Darknet-53 network used in the spatial feature extraction part is derived from the existing target detection model YoloV3. This network is one of the most advanced feature extraction networks, which can well balance the accuracy and speed of feature extraction. In the Darknet-53 network, the video frame is first passed through a convolution module (Conv2D), and then through five residual module (Res-Block) groups. In the residual module, first perform a convolution to get feature layer 1, then perform two convolutions on feature layer 1 to get feature layer 2, and finally add feature layer 1 and feature layer 2 to get the output result. In this network, batch normalization and LeakyReLU function activation are performed after each convolution. The output feature maps of the third residual module group and the fifth residual module group in this gaze detection network are defined as low-level detail feature maps and high-level semantic feature maps, respectively.

[0103] The temporal feature enhancement and gaze detection result generation parts are the main innovative parts of the gaze detection network, which will be described in detail in the following subsections. The last subsection of this part also gives the training process of the gaze detection network.

[0104] 1.1 Temporal feature enhancement​

[0105] The low-level detail feature map and the high-level semantic feature map output by the spatial feature extraction part are respectively enhanced by an AST-LSTM, and the two AST-LSTMs constitute a group. The AST-LSTM is improved and realized on the basis of the ST-LSTM. The ST-LSTM is derived from the PredRNN model, which memorizes spatial and temporal features in a unified memory unit and transmits the memory in the vertical and horizontal layers.

[0106] In the AST-LSTM, the size of the feature map to be input is first reduced by the feature preprocessing module to reduce the dimension of the input time sequence network and thus reduce the calculation amount. Then, according to the attention mechanism, the reduced feature map is converted into an attention map, and the description formula of this process is as follows:

[0107]

[0108] In the formula, is the attention map, x t is the reduced feature map, h t-1 is the hidden state of the previous frame, w i is the weight matrix of the full connection layer, and σ is the sigmoid function, which can convert the numerical range to 0-1. The conversion process allocates the weight of the feature points according to the importance, and the feature points multiplied by the weight are

[0109] Since the proportion of eye information in the video frame is very small, the attention map output by the above attention mechanism is used to replace the feature map in inter-frame transmission, which can effectively suppress the background information and improve the information mining ability of the ST-LSTM. The description formula of this process is as follows:

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116] In the formula, i t , g t , f t , o t are the input gate, input adjustment gate, forgetting gate and output gate respectively, and the attention map is used to replace the feature map in the calculation process. is the time memory state, is the hidden state, is the space-time memory state. is the time memory state, is generated by the above gate, responsible for the information transmission between frames, so as to realize the information transmission of the attention map between frames. is responsible for the information transmission between layers, and the corresponding gate is still obtained by the feature map calculation, so as to maintain the degree of restoration of spatial information. Wherein, w ** is the weight matrix of the full connection layer, b * is the bias vector of the full connection layer, w 1×1 is the weight matrix of the convolution layer, is the time memory state of the last frame.

[0117] 1.2 Generate gaze detection results

[0118] The network adopts the target detection framework SSD instead of the classification framework to avoid the interference of other personnel eye information and can obtain the detection information of the gaze of multiple persons at the same time. This part first generates the preliminary detection results according to the target detection mechanism of SSD: 1) the feature map after the time feature enhancement, the position bias information of the default frame, the class confidence are obtained through 1x1 convolution, and the candidate frame is generated according to this; 2) for the two classes of eye contact and non-eye contact, the candidate frame whose class confidence is higher than the threshold (set to 0.5) is selected, and then the NMS (the IoU threshold is set to 0.7) is executed according to the position information and the confidence to perform the second round of screening on the candidate frame to obtain the detection frame.

[0119] Next, in order to fully utilize the long-term frame and the short-term frame to improve the detection frame information, the patent proposes a novel memory enhancement method, which mainly includes: 1) memory enhancement process MEP, which uses multi-head attention algorithm to realize step-by-step enhancement of the detection frame information; 2) memory storage module MSM, which realizes real-time automatic update of the enhanced frame information for direct use of MEP. The specific implementation process of the memory enhancement method is shown in the accompanying Figure 2 , wherein the solid circle is the frame after memory enhancement, and the hollow circle is the frame without memory enhancement. The details are as follows.

[0120] MSM is used to flexibly access frame information, and the main idea is to sequentially store the enhanced information of each frame and directly call them. MSM always exists in the frame sequence, and the new frame information is stored in MSM after being enhanced, and the earliest frame information in MSM is removed. In order to balance the detection accuracy and speed, the storage capacity of MSM is determined to be 30 frames through experiments, of which 25 frames are early frames and 5 frames are short-term frames. In addition, the long-term frame is 15 frames, which is obtained by rearranging the early frames and taking the last 15 frames.

[0121] MEP (Mean Enhancement Process) is mainly divided into two parts: short-term frame enhancement and current frame enhancement. A multi-head attention mechanism is integrated into the enhancement process to achieve in-depth mining of frame information. To more clearly explain the technical details of MEP, the main objects involved in this section are listed below:

[0122]

[0123] Before performing short-term frame enhancement, the MEP first scrambles and truncates the early frames in the MSM to obtain the long-term frame set S. l When performing short-term frame enhancement, since the positional information of long-term frames has little reference value for the current frame, enhancement is only applied to classification information. First, S... l Long-term frame B in d For S s Detection box in The enhancement is performed one by one, and the formula is described as follows:

[0124]

[0125] In the formula, It is using B d The classification information in the set is used to check the bounding box. The process of enhancement; It represents the classification information of the detection boxes to be enhanced in short-term frames; N is the number of attention heads in the multi-head attention mechanism, and M is the number of attention heads in the multi-head attention mechanism. n It is a linear transformation matrix; J is the B of the long frame. d The number of detection boxes in the set, This provides the classification information for the bounding boxes in the set. for and The relevant weights.

[0126] This process enhances the category information of the detection boxes to be enhanced based on the category correlation between each detection box in the long-term frame and the detection box to be enhanced. Repeating this process completes the long-term frame set S. l For the short-term frame set S s Information enhancement of all detection boxes in the dataset is used to obtain the enhanced short frame set S. s .

[0127] Next, the MEP performs current frame enhancement. Since the location and classification information from enhanced short-term frames are highly relevant to the current frame, this part utilizes both of these pieces of information. Enhanced short frames in For the current frame B c detection box The enhancement is performed one by one, and the formula is described as follows:

[0128]

[0129]

[0130] In the formula, the classification information enhancement is the same as the short-term frame enhancement, and the position information enhancement uses the Concat function to aggregate all the mined position information. Among them, is the process of enhancing the detection box using the enhanced classification information in the set ; is the classification information of the detection box to be enhanced in the current frame; N is the number of attention heads of the multi-head attention mechanism, and M n is the linear transformation matrix; I is the number of detection boxes in the set of short-term frames, is the enhanced classification information of the detection box in the set; is the correlation weight of and . is the process of enhancing the detection box using the position information in the set ; is the position information of the detection box to be enhanced in the current frame; N is the number of attention heads of the multi-head attention mechanism, and M n is the linear transformation matrix; J is the number of detection boxes in the set of long-term frames, is the position information of the detection box in the set; is the correlation weight of and .

[0131] After the above process is completed, the enhanced information of the current frame can be obtained, that is, the eye detection box position and gaze contact classification result of the gaze detection network.

[0132] 1.3 Training scheme

[0133] The model designs an end-to-end training scheme without using pre-trained components. This scheme is a correlation training scheme, and the formula is described as follows:

[0134]

[0135] The positioning loss function L loca , the confidence loss function L conf is the built-in loss function of the target detection framework SSD. The attention loss L at is designed for the attention mechanism introduced in the time sequence feature enhancement part, and the formula is described as follows:

[0136]

[0137] In Equation (11), a, b, g are the weights of loss, N is the batch size of incoming data; l is the L2 regularization term, and q is the parameter set, containing all weight matrices and bias vectors in the AST-LSTM computation process. In Equation (12), G i and P i represent the real attention map and the predicted attention map, respectively.

[0138] Our data source is the 300 Videos in the Wild (300-VW) dataset. This dataset contains 114 videos, a total of 218,595 frames, of which 64 videos are used for testing and 50 videos are used for training, each frame contains more than one face. We use INTERACT video annotation software of Mangold International to mark the gaze contact and offset at the frame level. The annotation process is carried out by 4 independent raters to establish reliability.

[0139] The implementation process of training is as follows: using the annotated dataset for training and verification, the number of training iterations is 120K, and the learning rate is 0.001 (first 80K) and 0.0001 (last 40K) respectively. In order to obtain more helpful candidate frames, the IoU threshold of NMS is set to 0.7. Among them, the training method of memory enhancement part is: 1) randomly select some frames and store them in MSM; 2) use these frame information to complete the memory enhancement process; 3) the storage capacity, the number of early frames and short-term frames randomly change to observe and select the best number combination. Through experiments, it is determined that the long-term features contained in MSM, the short-term features contained in MSM and the size of MSM are set to 15, 5 and 30 respectively. In addition, the network based on SSD is trained by SGD optimizer, and the network based on ST-LSTM is trained by ADAM optimizer. The deep model is designed under the Pytorch framework, and three RTX 2080ti GPUs provide computing resources for the training process.

[0140] II. Hand-eye coordination fun test for ASD children

[0141] This part aims to apply the gaze test for ASD children proposed by Chong, E. et al. to the hand-eye coordination test, that is, to replace the positioning of each fixation point of the child with the classification of whether the child's gaze contacts. This test uses the gaze detection network designed in the first part instead of the eye tracking device to solve many problems brought by eye tracking for testing.

[0142] The eye-hand coordination defect of ASD patients is most obvious when pointing to the target of fixation. Accordingly, the present application is inspired by the game of Whack-a-mole to design a set of interesting tests of hand-eye coordination. The ordinary Whack-a-mole game is described as follows: there are several holes, and each hole has a doll. A doll is randomly popped out of a hole and retracted into the hole within a short period of time (only one hole pops out a doll at the same time). If the player hits the head of the doll before it is retracted, he or she will get points, otherwise not. After the doll is retracted, another random hole pops out the doll. This cycle continues until the game is closed. The experimental apparatus involved in the test is shown in FIG. 1, wherein the game device is improved as follows: Figure 3

[0143] 1) Each hole is equipped with a camera and faces the center point of the stool. When the doll in the hole is popped out, the corresponding camera is started; when the doll is retracted into the hole, the corresponding camera is turned off.

[0144] 2) The doll does not automatically retract into the hole after being popped out, but only retracts when it is hit by the stick, to prevent ASD children from being stimulated because they cannot hit the doll.

[0145] 3) According to the video captured by the camera, the gaze detection network detects the fixation of the child on the corresponding hole to determine the time used from the doll being popped out to the child's eyes noticing it.

[0146] 4) The device is equipped with a timing system to record the time used from the doll being popped out to being hit.

[0147] 5) For the accuracy of the hand-eye coordination test, the device is equipped with a fixed child hand placement table.

[0148] 6) The hole and the doll are small in size and far apart to increase the hand movement time and improve the test effect.

[0149] The specific process of the test is shown in FIG. 2, and the implementation details of each step will be introduced below in conjunction with the drawings. Figure 4

[0150] Step 1: Guide the child to sit on the stool, hold the stick and place it on the hand placement table;

[0151] Step 2: Turn on the test device;

[0152] Step 3: A doll is randomly popped out of a hole, the corresponding camera is simultaneously captured, and the timing system starts timing from zero;

[0153] Step 4: The gaze detection network detects in real time to determine the time when the eyes notice;

[0154] Step 5: The child hits the doll, and the timing system obtains the time when the hand movement arrives; ​​

[0155] Step 6: the doll is retracted, the corresponding camera is closed, and the timing system is closed;

[0156] Step 7: determine whether to close the test device, if yes, proceed to step 8; if no, return to step 3;

[0157] Step 8: the expert evaluates the hand-eye coordination ability of the ASD patient according to the collected information and comprehensively considers the distance between each hole and the hand placement table.

[0158] The above embodiments have described the technical solutions and beneficial effects of the present application in detail. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the present application. Any modification, supplement and equivalent replacement within the principle range of the present application should be included in the protection scope of the present application.

Claims

1. An ASD child gaze detection method, characterized in that, The application relates to a gaze detection method and a gaze detection model. The gaze detection model comprises: a spatial feature extraction module which extracts spatial features of video data frame by frame and outputs low-level detailed features and high-level semantic features in a shallow layer; a time feature enhancement module which uses an AST-LSTM network to enhance the time features of the low-level detailed features and the high-level semantic features output by each frame in a time domain range; the AST-LSTM network comprises a feature preprocessing module, an attention mechanism module and an ST-LSTM network; the feature preprocessing module is used to reduce the size of a feature map to be input, thereby reducing the dimension of the input time sequence network, and then the attention mechanism module is used to convert the reduced feature map into an attention map, and the process can be described by the following formula: the converted attention map is used to replace the feature map in the inter-frame transmission of the ST-LSTM network, and the process can be described by the following formula: ； In the formula, is an attention map, is a reduced feature map, is a hidden state of a previous frame, is a weight matrix of a full connection layer, is a sigmoid function; a gaze detection result generation module which uses an SSD detection framework to output an eye detection frame of a current frame based on the time-enhanced low-level detailed features and high-level semantic features, and uses a memory enhancement process to gather key information from long-term frames and short-term frames to enhance the eye detection frame of the current frame, so as to obtain the position of the eye detection frame of the current frame and the gaze contact classification result. ； ； ； ； ； ； In the formula, are input gate, input adjustment gate, forget gate and output gate respectively, and the attention map is used to replace the feature map in the calculation process; is a weight matrix of the full connection layer, is a bias vector of the full connection layer, is a weight matrix of the convolution layer, is a time memory state of the previous frame; is a time memory state, is a hidden state, is a space-time memory state; In the gaze detection result generation module, the SSD detection framework is used to output the eye detection frame of the current frame based on the time-enhanced low-level detailed features and high-level semantic features, and the process comprises:

2. The ASD child gaze detection method of claim 1, wherein, (i) the time-enhanced low-level detailed features and high-level semantic features are used to obtain the position bias information and the class confidence of a default frame through convolution, and candidate frames are generated based on the position bias information and the class confidence; (ii) for the two classes of gaze contact and non-gaze contact, candidate frames with a class confidence higher than a threshold value are selected, and a second round of screening is performed on the candidate frames through a non-maximum suppression operation based on the position bias information and the class confidence, so as to obtain the eye detection frame. The memory enhancement process is used to gather key information from long-term frames and short-term frames to enhance the eye detection frame of the current frame, and the process comprises:

3. The ASD child gaze detection method of claim 1, wherein, (I) the memory enhancement process uses a multi-head attention algorithm to enhance the eye detection frame information of each frame step by step, so as to obtain the position of the eye detection frame of the current frame and the gaze contact classification result; (II) the enhanced frame information is automatically updated in real time through a memory storage module, the new frame information is stored in the MSM after being enhanced, and the earliest frame information is removed; the frame information in the memory storage module comprises a plurality of early frames and a plurality of short-term frames. Step (I) comprises:

4. The ASD child gaze detection method of claim 3, wherein, concat (I-1) Shuffling and randomly intercepting the early frames in the memory storage module to obtain a long-term frame set ; (I-2) using long-term frames in the set of short-term frames each of the bounding boxes in the set of short-term frames is enhanced to obtain a set of enhanced short-term frames The formula is described as follows: ； In the formula, is the classification information of the detection frame in the short-term frame set; is the classification information of the detection frame in the long-term frame set; M n is a linear transformation matrix; is is the correlation weight of ; N is the number of attention heads of the multi-head attention mechanism; J is the number of detection frames in the long-term frame set; is a process of enhancing the short-term frame detection frame using the classification information of the long-term frame set; (I-3) using the enhanced short-term frame in (I-2) to enhance the detection frame of the current frame (I-4) using the enhanced detection frame in (I-3) to obtain the eye detection frame position and the gaze contact classification result of the current frame, which is described by the following formula: ； ； In the formula, This provides the classification information for the detection bounding boxes in the current frame. To enhance the classification information of detection boxes in short-term frame sets; for and The relevant weights; I It is to enhance short frames The number of bounding boxes in the set; N is the number of attention heads in the multi-head attention mechanism; M n It is a linear transformation matrix; It utilizes enhanced short frames The classification information in the set is used to detect bounding boxes in the current frame. The process of enhancement; The gaze detection model is trained in an end-to-end mode, and a loss function used in the training is as follows: is a Concat function; is an enhanced short-term frame is position information of a bounding box in the set, is position information of a bounding box to be enhanced in the current frame; is a is a correlation weight of is a process of enhancing a bounding box in the current frame using position information in the set of enhanced short-term frames .​ 5. The ASD child gaze detection method of claim 1, wherein, lambda is an L2 regularization term; ； wherein, and are the localization loss function and the confidence loss function built-in the SSD detection framework, respectively; a, b, g are the weights of the losses; N is the batch size of the incoming data; theta is a parameter set, which comprises all weight matrices and bias vectors in the AST-LSTM calculation process; The application further relates to a gaze detection system which comprises the gaze detection model. is the attention loss function, whose formula is described as follows: ； wherein, and respectively represent the real attention map and the predicted attention map.

6. An ASD child gaze detection apparatus characterized by, The application relates to a gaze detection method and a gaze detection model.

7. An ASD child hand-eye coordination test device, characterized in that, The gaze detection model comprises: a spatial feature extraction module which extracts spatial features of video data frame by frame and outputs low-level detailed features and high-level semantic features in a shallow layer; a time feature enhancement module which uses an AST-LSTM network to enhance the time features of the low-level detailed features and the high-level semantic features output by each frame in a time domain range; the AST-LSTM network comprises a feature preprocessing module, an attention mechanism module and an ST-LSTM network; the feature preprocessing module is used to reduce the size of a feature map to be input, thereby reducing the dimension of the input time sequence network, and then the attention mechanism module is used to convert the reduced feature map into an attention map, and the process can be described by the following formula: the converted attention map is used to replace the feature map in the inter-frame transmission of the ST-LSTM network, and the process can be described by the following formula: a gaze detection result generation module which uses an SSD detection framework to output an eye detection frame of a current frame based on the time-enhanced low-level detailed features and high-level semantic features, and uses a memory enhancement process to gather key information from long-term frames and short-term frames to enhance the eye detection frame of the current frame, so as to obtain the position of the eye detection frame of the current frame and the gaze contact classification result. In the gaze detection result generation module, the SSD detection framework is used to output the eye detection frame of the current frame based on the time-enhanced low-level detailed features and high-level semantic features, and the process comprises: (i) the time-enhanced low-level detailed features and high-level semantic features are used to obtain the position bias information and the class confidence of a default frame through convolution, and candidate frames are generated based on the position bias information and the class confidence; (ii) for the two classes of gaze contact and non-gaze contact, candidate frames with a class confidence higher than a threshold value are selected, and a second round of screening is performed on the candidate frames through a non-maximum suppression operation based on the position bias information and the class confidence, so as to obtain the eye detection frame. The memory enhancement process is used to gather key information from long-term frames and short-term frames to enhance the eye detection frame of the current frame, and the process comprises: (I) the memory enhancement process uses a multi-head attention algorithm to enhance the eye detection frame information of each frame step by step, so as to obtain the position of the eye detection frame of the current frame and the gaze contact classification result; (II) the enhanced frame information is automatically updated in real time through a memory storage module, the new frame information is stored in the MSM after being enhanced, and the earliest frame information is removed; the frame information in the memory storage module comprises a plurality of early frames and a plurality of short-term frames. Step (I) comprises: concat The ASD children gaze detection device, miniature camera, timing device, gopher model, wooden stick, test table, hand placement table and test bench of claim 6, wherein the test bench is arranged on one side of the test table, and the hand placement table is arranged between the test table and the test bench. The test table is provided with a plurality of holes, and each hole is provided with a gopher model that can be ejected or retracted into the hole. A miniature camera is arranged at the hole opening and faces the test bench. The miniature camera is turned on when the gopher model is ejected and turned off when the gopher model is retracted into the hole. One gopher model is randomly ejected from the hole at a time, and the ejected gopher model is retracted into the hole after being hit. The test bench is used to carry the tester, and the hand placement table is used to place the tester's hands. The miniature camera inputs the collected video data containing the tester's face into the ASD children gaze detection device, and determines whether the tester gazes at the ejected gopher model through the gaze detection model to obtain the time from the ejection of the gopher model to the tester's gaze at the ejected gopher model. The timing device counts and records the time from the ejection of the gopher model to the tester's hit.

8. A method for testing the hand-eye coordination of ASD children using the device according to claim 7, characterized in that, The method comprises the following steps: (1) guiding the child to sit on the test bench, hold the wooden stick and place it on the hand placement table; (2) turning on the test device; (3) randomly ejecting the gopher model from the hole, and synchronously shooting the corresponding miniature camera and starting the timing device; (4) inputting the video data containing the child's face collected by the miniature camera into the gaze detection model, and performing real-time detection by the gaze detection model to determine the time when the child's eyes notice the ejected gopher model; (5) hitting the ejected gopher model by the child, and obtaining the time when the child hits the ejected gopher model by the timing device; (6) retracting the gopher model after being hit, turning off the corresponding miniature camera and the timing system; (7) determining whether to turn off the test device. If yes, proceed to step (8); if no, return to step (3); (8) evaluating the hand-eye coordination ability of the tester according to the collected information and considering the distance between each hole and the hand placement table.

Images