An intelligent screening system for children with autism

Abstract

The invention relates to an intelligent screening system for children with autism, comprising a data acquisition module for collecting facial videos of children completing tasks in various paradigms; a data preprocessing module for acquiring basic data related to the sight point, facial expression and head posture of the children according to the facial videos; a feature extraction module for extracting sight point features, facial expression features and head posture features from the basic data acquired in the data preprocessing module using a deep learning technology; a feature fusion module for fusing the sight point features, facial expression features and head posture features; a classification module for comprising an LSTM model for integrating the fused features into time series data and inputting the fused features into the LSTM model for classification analysis; in this way, the explicit behavior features under various typical paradigms are integrated through the feature extraction module and the feature fusion module, and the features are subjected to time series analysis through the LSTM model, so as to realize accurate classification of children with ASD.

Description

Technical Field

[0001] The present invention relates to the technical field of machine screening, and in particular to an intelligent screening system for children with autism. Background Art

[0002] Autism is a neurodevelopmental disorder with social disorders, narrow interests and stereotyped behaviors as its main clinical manifestations. Given that the earlier the intervention, the better the effect, early identification and diagnostic evaluation are crucial. At present, many diagnostic standards for childhood autism have been formulated at home and abroad, such as the Diagnostic and Statistical Manual of Mental Disorders, the relevant diagnostic standards of the World Health Organization and the standards of the Chinese Psychiatric Association, and are equipped with corresponding diagnostic tools. The Autism Diagnostic Interview Scale (ADI-R) and the Child Autism Rating Scale (CARS) are commonly used abroad, and CARS and the Child Autism Behavior Scale (ABC) are commonly used in China. However, these scale assessments require the cooperation of multiple professionals and the participation of parents. The process is complicated and time-consuming, and multiple conversations and observations are indispensable. In addition, due to the diverse manifestations of autism, it is difficult for doctors to find a universal diagnostic model, and misdiagnosis and missed diagnosis occur from time to time.

[0003] With the booming development of deep learning and computer vision technology, video analysis based on behavioral characteristics is gradually becoming a trend for ASD assisted screening. By capturing children's facial expressions, gaze, head posture, etc., the behavioral differences between them and typically developing children (TD) are analyzed to achieve intelligent screening. However, existing studies mostly focus on a single feature or task paradigm, and fail to make good use of the amplification effect of different situations on abnormal behaviors of ASD children, resulting in poor accuracy and sensitivity of the screening method, making it difficult to meet the needs of various clinical scenarios. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide an intelligent screening system for children with autism, comprising a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module and a classification module;

[0005] The data collection module is used to collect facial videos of children completing various paradigm tasks;

[0006] The data preprocessing module is used to obtain basic data related to the child's gaze point, facial expression and head posture based on the facial video;

[0007] The feature extraction module is used to extract the sight point feature, facial expression feature and head posture feature from the basic data obtained in the data preprocessing module by using deep learning technology;

[0008] The feature fusion module is used to fuse the gaze point features, facial expression features and head posture features;

[0009] The classification module includes an LSTM model, which is used to integrate the fused features into time series data and input them into the LSTM model for classification analysis;

[0010] Among them, the multiple paradigm tasks include blank-overlap paradigm, figure gaze paradigm, spatial cue paradigm and name-calling response paradigm; the blank-overlap paradigm is used to examine children's attention shifting ability based on their gaze shifting, head micro-movements and facial expression changes; the figure gaze paradigm is used to evaluate children's gaze behavior in social situations based on their gaze concentration, facial expression reactions and head posture in social gaze situations; the spatial cue paradigm is used to test children's sensitivity to external visual cues by observing children's gaze tracking patterns, facial expression sensitivity and head turning tendencies; the name-calling response paradigm is used to examine children's sensitivity to familiar sounds based on the guardian's call.

[0011] Preferably, the deep learning technology includes:

[0012] The visual Transformer model is used to calculate basic data related to where the child’s gaze falls;

[0013] The MobileNetV3 model is used to calculate basic data related to children's facial expressions, including neutral, angry, scared, happy, and sad;

[0014] The FSA-Net model is used to analyze basic data related to children's head posture angles, including pitch angle, horizontal angle and roll angle.

[0015] Preferably, the calculating of basic data related to the child's sight point includes calculating the cumulative number of sight points, and the calculating of the cumulative number of sight points is used to reflect the sight point characteristics;

[0016] The paradigm task distributes different elements at different positions on the screen. The calculation of the cumulative number of sight points is to calculate the cumulative number of sight points in different screen areas. The formula is as follows:

[0017] ;

[0018] in, It is expressed as the cumulative number of nine areas from the 1st second to the nth second during the time when the paradigm appears and the cumulative number of unpredictable gaze points. - Indicates the number of sight points in 9 areas in 30 frames at the i-th second, It indicates the number of unpredictable gaze points in the 30 frames of the i-th second due to excessive head deflection.

[0019] Preferably, the calculation of basic data related to the child's facial expression includes calculating the cumulative number of different emotions, and the calculation of the cumulative number of different emotions is used to reflect the facial expression characteristics; the formula is as follows:

[0020] ;

[0021] in, It is expressed as the cumulative number of seven expressions from the 1st second to the nth second during the time when the paradigm appears, as well as the cumulative number of unpredictable expression labels. - represents the number of seven expression labels in the 30 frames of the i-th second, It indicates the number of labels that cannot be predicted in the 30 frames of the i-th second due to excessive head deflection.

[0022] Preferably, the basic data related to the child's head posture angle includes a calculated frequency of occurrence of the head posture angle, and the calculated frequency of occurrence of the head posture angle is used to reflect the head posture characteristics; the formula is as follows:

[0023]

[0024] Among them, the head posture angle range of -15° to 15° is "stable", the head posture angle range of 15° to 30° or -15° to -30° is "slightly deviated", and the head posture angle range of >30° or <-30° is "severely deviated". , , Indicates the cumulative number of yaw, pitch, and roll angles from the 1st second to the nth second during the pattern appearance time. arrive Indicates the number of Yaw angles in three intervals in 30 frames of the i-th second, arrive Indicates the number of Pitch angles in three intervals in the 30th frame of the i-th second. arrive Indicates the number of Roll angles in three intervals in 30 frames of the i-th second.

[0025] Preferably, the formula for the feature fusion module to fuse the sight point feature, the facial expression feature and the head posture feature is:

[0026] .

[0027] Preferably, the child's sensitivity to familiar sounds is specifically reflected by the active response time of the head gesture and the latency time of the head gesture response;

[0028] The formula for calculating the active response time of the head posture is:

[0029] ;

[0030] in, Indicates whether the yaw angle of the mth frame is greater than 20 degrees; It indicates the cumulative number of frames in which the yaw angle of the child's head posture is greater than 20 degrees from the call frame when the child's name is called to the end frame 3 seconds after the name is called;

[0031] The formula for calculating the latency of head gesture response is:

[0032] ;

[0033] in, Indicates whether the yaw angle of the mth frame is greater than 20 degrees. It indicates the cumulative number of frames in which the yaw angle of the child's head posture is greater than 20 degrees, from the call frame when the child's name is called to the pos frame in which the child's head has a positive response.

[0034] Preferably, the child's sensitivity to the familiar sound can also be reflected by the active response time of the gaze and the latency time of the active response of the gaze;

[0035] The formula for calculating the active response time of the line of sight is:

[0036] ;

[0037] in, Indicates whether the gaze point in the mth frame falls outside the screen. It represents the cumulative number of frames in which the child's gaze point falls outside the screen from the call frame when the name is called to the end frame 3 seconds after the name is called;

[0038] The formula for calculating the latency of the gaze positive response is:

[0039] ;

[0040] in, Indicates whether the gaze point in the mth frame falls outside the screen. It indicates the cumulative number of frames in which the child's gaze point falls outside the screen, from the call frame when the name is called to the pos frame in which the child's gaze produces a positive response.

[0041] Preferably, the blank overlapping task includes a blank task and an overlapping task, wherein the blank task first displays the central visual stimulus, then displays a blank screen, and finally displays the target visual stimulus at another location; the overlapping task first displays the central visual stimulus, then the central visual stimulus does not disappear, and displays the target visual stimulus at another location;

[0042] The character gaze paradigm first displays a “+” locator to calibrate the child’s gaze; then displays a character’s face image;

[0043] The spatial cue paradigm first displays a “+” sign as a gaze calibration point, and then a screen containing a cue image and a distractor appears;

[0044] The name-calling response paradigm first plays a video to attract the child's attention, and then during the video playing, two Bluetooth speakers are placed on the left and right sides behind the child to play the guardian's call.

[0045] Preferably, the feature extraction module can also be used to extract target gaze features based on facial videos of multiple paradigm tasks; the target gaze features are used to respond according to the total gaze time, first gaze time and first gaze latency time of the child's gaze on the area of ​​interest;

[0046] In the blank-overlap paradigm, the region of interest is the central and surrounding stimuli; in the figure-gaze paradigm, the region of interest is the facial image of the central figure; in the spatial cue paradigm, the region of interest is the target image;

[0047] The total fixation time is the cumulative amount of time that the child fixates on the point in the region of interest during the paradigm presentation time, and is used to reflect the child's interest in the region of interest. The calculation formula for the total fixation time is:

[0048] ;

[0049] in, Indicates whether the gaze point of the mth frame falls within the area of ​​interest. Represents the cumulative number of fixations in the area of ​​interest from the 1st frame to the nth frame;

[0050] The first fixation time is the cumulative amount of time that the child first fixates on the area of ​​interest, and is used to reflect the child's initial impression and reaction to the area of ​​interest. The calculation formula for the first fixation time is:

[0051] ;

[0052] in, Indicates whether the gaze point of the mth frame falls within the area of ​​interest. It represents the cumulative number of fixations in the interest area from the first frame when the child first starts to fixate on the interest area to the end frame when the fixation leaves the interest area;

[0053] The first fixation latency is the cumulative fixation time from the start of the paradigm to the first fixation of the child on the area of ​​interest, which is used to reflect the child's reaction speed after the appearance of the area of ​​interest. The calculation formula is as follows:

[0054] ;

[0055] in, Indicates whether the gaze point of the mth frame falls within the area of ​​interest. Represents the cumulative number of fixations in the area of ​​interest from the first frame to the first frame when the child first starts to look at the area of ​​interest.

[0056] Beneficial effects of the present invention: The multi-task paradigm design is completed through the data acquisition module and the data preprocessing module, which effectively eliminates the influence of environmental factors on the screening accuracy. The multimodal behavioral features under various typical paradigms are integrated through the feature extraction module and the feature fusion module, and these features are analyzed through the LSTM model of the classification module to achieve accurate classification of ASD children. It helps to improve the accuracy and applicability of early screening for ASD, especially in scenarios where there is a lack of professional diagnostic resources, and can significantly reduce misdiagnosis and missed diagnosis, providing timely support for clinical intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, without paying creative labor, other drawings obtained based on these drawings still belong to the scope of the present invention.

[0058] Figure 1 is a screening flow chart of the present invention;

[0059] Figure 2 It is a blank-overlap paradigm diagram of the present invention;

[0060] Figure 3 It is a figure gaze paradigm diagram of the present invention;

[0061] Figure 4 It is a spatial clue paradigm diagram of the present invention;

[0062] Figure 5 A screen area division diagram of the present invention;

[0063] Figure 6 It is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0064] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention will be further described in detail below with reference to the accompanying drawings.

[0065] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are for distinguishing two non-identical entities with the same name or non-identical parameters. It can be seen that "first" and "second" are only for the convenience of expression and should not be understood as limitations on the embodiments of the present invention. The subsequent embodiments will not explain this one by one.

[0066] The directions and positions mentioned in the present invention, such as "upper", "lower", "front", "back", "left", "right", "inside", "outside", "top", "bottom", "side", etc., are only for reference to the directions or positions of the drawings. Therefore, the directions and positions used are for explaining and understanding the present invention, but not for limiting the scope of protection of the present invention.

[0067] like Figures 1 to 6 As shown in the embodiment of the present invention, an intelligent screening system for children with autism includes a data acquisition module, a data preprocessing module, a feature extraction module, a feature fusion module and a classification module;

[0068] The data collection module is used to collect facial videos of children completing various paradigm tasks;

[0069] The data preprocessing module is used to obtain basic data related to the child's gaze point, facial expression and head posture based on the facial video;

[0070] The feature extraction module is used to extract the sight point feature, facial expression feature and head posture feature from the basic data obtained in the data preprocessing module by using deep learning technology;

[0071] The feature fusion module is used to fuse the gaze point features, facial expression features and head posture features;

[0072] The classification module includes an LSTM model, which is used to integrate the fused features into time series data and input them into the LSTM model for classification analysis;

[0073] Among them, the multiple paradigm tasks include blank-overlap paradigm, figure gaze paradigm, spatial cue paradigm and name-calling response paradigm; the blank-overlap paradigm is used to examine children's attention shifting ability based on their gaze shifting, head micro-movements and facial expression changes; the figure gaze paradigm is used to evaluate children's gaze behavior in social situations based on their gaze concentration, facial expression reactions and head posture in social gaze situations; the spatial cue paradigm is used to test children's sensitivity to external visual cues by observing children's gaze tracking patterns, facial expression sensitivity and head turning tendencies; the name-calling response paradigm is used to examine children's sensitivity to familiar sounds based on the guardian's call.

[0074] It is understandable that during the collection process, in order to ensure the accuracy and consistency of the data, the children were arranged to be tested in a quiet, undisturbed environment, sitting in front of the camera to collect their facial data from the best perspective. In this process, the system reads the child's facial video data in real time through the camera, and the collection tasks include gaze point, facial expression and head posture characteristics. These data will be recorded in specific paradigm tasks to capture the behavioral characteristics of ASD children in different situations.

[0075] In the classification stage, since the features extracted by the accumulation method contain a large amount of time information, the present invention chooses to send the fused features to LSTM for classification. As a deep learning network suitable for time series data, the LSTM model can effectively extract and analyze the feature information of the previous and next dependencies in the time series through its internal memory unit and gating mechanism.

[0076] The deep learning techniques include:

[0077] The Visual Transformer (ViT) model is used to calculate basic data related to where children’s gaze falls;

[0078] The MobileNetV3 model is used to calculate basic data related to children's facial expressions, including neutral, angry, scared, happy, and sad;

[0079] The FSA-Net model is used to analyze basic data related to children's head posture angles, including pitch angle, horizontal angle and roll angle.

[0080] The calculating of basic data related to the child's sight point includes calculating the cumulative number of sight points, and the calculating of the cumulative number of sight points is used to reflect the sight point characteristics;

[0081] The paradigm task distributes different elements at different positions on the screen. The calculation of the cumulative number of sight points is to calculate the cumulative number of sight points in different screen areas. The formula is as follows:

[0082] ;

[0083] in, It is expressed as the cumulative number of nine areas from the 1st second to the nth second during the time when the paradigm appears and the cumulative number of unpredictable gaze points. - Indicates the number of sight points in 9 areas in 30 frames at the i-th second, It indicates the number of unpredictable gaze points in the 30 frames of the i-th second due to excessive head deflection.

[0084] It is understandable that, since our paradigm task distributes different elements at different positions on the screen during design, the present invention chooses to divide the screen with a resolution of 1920*1080 into 9 rectangular areas, such as Figure 5 As shown, the coordinates of the upper left corner are (0, 0) and the lower right corner is (1920, 1080). The screen can also be divided into other numbers of rectangular areas.

[0085] The calculation of basic data related to children's facial expressions includes calculating the cumulative number of different emotions, and the calculation of the cumulative number of different emotions is used to reflect the facial expression characteristics; the formula is as follows:

[0086] ;

[0087] It is expressed as the cumulative number of seven expressions from the 1st second to the nth second during the time when the paradigm appears, as well as the cumulative number of unpredictable expression labels. - represents the number of seven expression labels in the 30 frames of the i-th second, It indicates the number of labels that cannot be predicted in the 30 frames of the i-th second due to excessive head deflection.

[0088] The basic data related to the child's head posture angle includes the calculated frequency of occurrence of the head posture angle, and the calculated frequency of occurrence of the head posture angle is used to reflect the head posture characteristics; the formula is as follows:

[0089] ;

[0090] Among them, the intervals of "stable" (-15° to 15°), "slight deviation" (15° to 30° or -15° to -30°) and "severe deviation" (>30° or <-30°) are abbreviated as 0, 15, and 30. , , Indicates the cumulative number of yaw, pitch, and roll angles from the 1st second to the nth second during the pattern appearance time. arrive Indicates the number of Yaw angles in three intervals in 30 frames of the i-th second, arrive Indicates the number of Pitch angles in three intervals in the 30th frame of the i-th second. arrive Indicates the number of Roll angles in three intervals in 30 frames of the i-th second.

[0091] The formula for the feature fusion module to fuse the gaze point feature, facial expression feature and head posture feature is:

[0092] .

[0093] It is understandable that since the extracted features come from multiple experimental paradigms, when performing feature fusion, it is also necessary to fuse the features of different paradigms. The three experimental paradigms of blank-overlap, person gaze, and spatial clue are referred to as blank, person, and clue. It can be expressed as:

[0094] ;

[0095] Different experimental paradigms (blank, person, clue, call) observe and extract features of the target object's behavior from different angles. For example, features such as sight, facial expression, and head posture may have different emphases in different paradigms. Through feature fusion, these features from multiple experimental paradigms can be integrated together, making full use of the complementary information provided by different paradigms, so as to more comprehensively and accurately describe the state or behavior pattern of the target object. Provide a richer and more representative data foundation for subsequent classification tasks.

[0096] By integrating features from different paradigms, we can reduce data redundancy and noise, improve the signal-to-noise ratio of data, and enable the model to focus more on key features, thereby improving the generalization and robustness of the model, so that it can maintain good performance in different scenarios and data distributions.

[0097] The children's sensitivity to familiar sounds is specifically reflected by the active response time of head gesture and the latency time of head gesture response;

[0098] The formula for calculating the active response time of the head posture is:

[0099] ;

[0100] Indicates whether the yaw angle of the mth frame is greater than 20 degrees; It indicates the cumulative number of frames in which the yaw angle of the child's head posture is greater than 20 degrees from the call frame when the child's name is called to the end frame 3 seconds after the name is called;

[0101] The formula for calculating the latency of head gesture response is:

[0102] ;

[0103] Indicates whether the yaw angle of the mth frame is greater than 20 degrees. It indicates the cumulative number of frames in which the yaw angle of the child's head posture is greater than 20 degrees, from the call frame when the child's name is called to the pos frame in which the child's head has a positive response.

[0104] The children's sensitivity to familiar sounds can also be reflected by the positive response time of gaze and the latency time of positive response of gaze;

[0105] The formula for calculating the active response time of the line of sight is:

[0106] ;

[0107] Indicates whether the gaze point in the mth frame falls outside the screen. It represents the cumulative number of frames in which the child's gaze point falls outside the screen from the call frame when the name is called to the end frame 3 seconds after the name is called;

[0108] The formula for calculating the latency of the gaze positive response is:

[0109] ;

[0110] Indicates whether the gaze point in the mth frame falls outside the screen. It indicates the cumulative number of frames in which the child's gaze point falls outside the screen, from the call frame when the name is called to the pos frame in which the child's gaze produces a positive response.

[0111] The blank overlap task includes a blank task and an overlap task. The blank task first displays the central visual stimulus, then displays a blank screen, and finally displays the target visual stimulus at another location. The overlap task first displays the central visual stimulus, then the central visual stimulus does not disappear, and displays the target visual stimulus at another location.

[0112] The character gaze paradigm first displays a “+” locator to calibrate the child’s gaze; then displays a character’s face image;

[0113] The spatial cue paradigm first displays a “+” sign as a gaze calibration point, and then a screen containing a cue image and a distractor appears;

[0114] The name-calling response paradigm first plays a video to attract the child's attention, and then during the video playing, two Bluetooth speakers are placed on the left and right sides behind the child to play the guardian's call.

[0115] It is understandable that the blank-overlap paradigm mainly examines children's attention shifting ability. Figure 2As shown in (a), at the beginning of the experiment, a fixed visual stimulus was presented in the center of the screen for 500-2000 milliseconds, followed by a blank screen for 200-250 milliseconds to help children shift their attention away from the central stimulus. The target stimulus then appeared above or below the screen for 3000-8000 milliseconds. Figure 2 As shown in (b), at the beginning of the experiment, a fixed visual stimulus was presented in the center of the screen for 500-2000 milliseconds, followed by the simultaneous appearance of the target stimulus and the central stimulus for 3000-8000 milliseconds. The camera system recorded the children's gaze changes, head movements, and facial expressions during this process, thus reflecting their behavioral response characteristics in the attention switching task.

[0116] Figure-gaze paradigm: This paradigm is used to assess children’s gaze behavior in social situations. Figure 3 As shown in the figure, at the beginning of the experiment, a "+" locator appears on the screen to calibrate the child's line of sight; then the facial image of a person (such as a parent, teacher, etc.) is displayed for 20 seconds. The system records the child's gaze at the person's face, changes in emotional expression, and fine-tuning of the head direction. ASD children may show less facial gaze, or show abnormal emotional reactions and head postures during the gaze process.

[0117] Spatial cue paradigm: This paradigm is used to test children's sensitivity to external visual cues. The specific process is as follows: Figure 4 As shown in the figure, a "+" sign first appears in the center of the screen (lasting 1000 milliseconds) as a gaze calibration point, and then a screen containing a cue image and a distractor appears for 5000 milliseconds. The system records whether the child's gaze is focused on the target cue, slight changes in expression, and head turning to detect the characteristics of their response to exogenous cues under multiple target conditions. ASD children may show weaker responses in this situation.

[0118] Name-calling response paradigm: This paradigm is used to test the child's sensitivity to the guardian's calling of his or her name. First, a cartoon video (lasting 20 seconds) is played on the screen to attract the child's attention. Then, during the cartoon, the guardian's call is played twice through two Bluetooth speakers (placed 45 degrees to the left and right behind the child). The system records whether the child's eyes move away from the screen, the changes in his or her facial expressions when hearing the familiar call, and the movement of his or her head. ASD children may show a slower reaction in this situation.

[0119] The feature extraction module can also be used to extract target gaze features based on facial videos of various paradigm tasks; the target gaze features are used to respond to the total gaze time, first gaze time and first gaze latency time of the region of interest set according to the child's gaze;

[0120] In the blank-overlap paradigm, the region of interest is the central and surrounding stimuli; in the figure-gaze paradigm, the region of interest is the facial image of the central figure; in the spatial cue paradigm, the region of interest is the target image;

[0121] The total fixation time is the cumulative amount of time that the child fixates on the point in the region of interest during the paradigm presentation time, and is used to reflect the child's interest in the region of interest. The calculation formula for the total fixation time is:

[0122] ;

[0123] Indicates whether the gaze point of the mth frame falls within the area of ​​interest. Represents the cumulative number of fixations in the area of ​​interest from the 1st frame to the nth frame;

[0124] The first fixation time is the cumulative amount of time that the child first fixates on the area of ​​interest, and is used to reflect the child's initial impression and reaction to the area of ​​interest. The calculation formula for the first fixation time is:

[0125] ;

[0126] Indicates whether the gaze point of the mth frame falls within the area of ​​interest. It represents the cumulative number of fixations in the interest area from the first frame when the child first starts to fixate on the interest area to the end frame when the fixation leaves the interest area;

[0127] The first fixation latency is the cumulative fixation time from the start of the paradigm to the first fixation of the child on the area of ​​interest, which is used to reflect the child's reaction speed after the appearance of the area of ​​interest. The calculation formula is as follows:

[0128] ;

[0129] Indicates whether the gaze point of the mth frame falls within the area of ​​interest. Represents the cumulative number of fixations in the area of ​​interest from the first frame to the first frame when the child first starts to look at the area of ​​interest.

[0130] It is understandable that in the present invention, multiple paradigm tasks can be repeated multiple times, and the positions of the region of interest and other regions in the video can be changed at the same time. By observing the target gaze characteristics of children on the region of interest, it can be understood whether the children have dependence or stereotypes on objects (for example, as long as the region of interest is changed in position or direction, the child becomes emotionally unstable). When normal children watch the same video repeatedly, the children's sight may be converted from the region of interest to other regions, or the attention may not be directly on the paradigm video. However, when facing a favorite object, ASD children are very single-minded and no longer willing to contact other things. In daily life, many parents of children with ASD often regard children with ASD as developmental delays in the early stages, so they do not intervene in time for treatment; the target gaze characteristics can also be used to distinguish whether the child is developmentally delayed or ASD, as follows:

[0131] Total fixation time:

[0132] When normal children watch a video repeatedly, their total fixation time on the area of ​​interest will decrease as the number of times they watch increases because their attention is easily distracted. They will gradually lose interest in the video and shift their gaze to other areas or simply stop paying attention to the video. Children with developmental delays are similar to normal children in this regard, except that their development process may be slightly slower. For example, normal children may have a significant decrease in the total fixation time on the area of ​​interest (such as the face of a person in the person fixation paradigm) when they watch the video for the third time, while children with developmental delays may need to watch it several times before this happens.

[0133] It is different for children with ASD. When they become interested in a certain area of ​​interest (such as the target image in the spatial cue paradigm), the total fixation time may remain relatively stable and long over multiple viewings. This is because they tend to have a strong focus on specific things. Once attracted, it is difficult for them to shift their attention to other things. They may keep staring at the things they like, reflecting their dependence on specific things.

[0134] First fixation time:

[0135] When normal children watch a video for the first time, the time they first fixate on the area of ​​interest may be longer due to novelty, but as familiarity increases, the time they first fixate on the area of ​​interest will be shorter when they watch it again. The trend of the first fixation time of children with developmental delay is roughly the same, but they may be slightly slower in reaction speed. For example, when normal children watch the character fixation paradigm video for the second time, the time they first fixate on the central character's face may be much shorter than the first time, while the reduction in children with developmental delay may be smaller.

[0136] When children with ASD are faced with familiar and favorite things, their first fixation time may not change significantly because they have a stereotyped way of perceiving things. Even if they watch the same video multiple times, their first impression and reaction pattern of the area of ​​interest is relatively fixed, and they may first fixate on the area of ​​interest in a similar way and for a similar duration each time.

[0137] First fixation latency:

[0138] When normal children watch the video repeatedly, their first fixation latency will gradually increase because their attention will gradually be dispersed and they will no longer be so eager to focus on the area of ​​interest. Children with developmental delays will also have a similar trend, but the process may be slower. For example, when normal children watch the blank-overlap paradigm video for the first time, they may quickly make their first fixations to the central and peripheral stimuli, but after multiple viewings, they may have a long delay before making their first fixations to these stimuli.

[0139] Children with ASD may have a relatively stable first fixation latency for the areas of interest they are interested in. Their response patterns are relatively fixed. Once they see familiar objects, they will first fixate on the areas of interest at a relatively fixed speed, without a significant delay as normal children do with repeated viewing, which reflects the stereotyped nature of their behavior.

[0140] Differences in response to position changes:

[0141] Normal children and children with developmental delays can adapt relatively well when the positions of the area of ​​interest and other areas in the video change. They may become curious about the area of ​​interest in the new position, and their eyes will shift to it, and their emotions are usually stable. For example, in the spatial cue paradigm, if the position of the target image changes, normal children and children with developmental delays will look for and look at the target image in the new position without strong emotional fluctuations.

[0142] Children with ASD may have strong emotional reactions to this change in position. Because they have strong stereotypes about the position and order of things, they may become emotionally unstable, such as crying and getting irritable, when the position of their area of ​​interest changes. They show a strong dependence on the characteristics of the target gaze and find it difficult to accept changes in the position of the area of ​​interest. This difference in reaction pattern helps distinguish ASD children from other children.

[0143] The specific implementation process is as follows:

[0144] Step 1: Collect facial videos of children completing paradigm tasks.

[0145] Step 2: In the feature extraction stage, the system first uses a deep learning model to obtain basic data on gaze point, facial expression, and head posture from the collected video data, and then designs a corresponding feature extraction scheme based on the characteristics of different behavioral data.

[0146] Step 3: In the feature fusion stage, the present invention fuses the sight line, facial expression and head posture features extracted from the above three paradigm tasks.

[0147] In the classification stage, since the features extracted by the accumulation method contain a large amount of time information, the present invention chooses to send the fused features to LSTM for classification. As a deep learning network suitable for time series data, the LSTM model can effectively extract and analyze the feature information of the previous and next dependencies in the time series through its internal memory unit and gating mechanism.

[0148] For the blank-overlap, person-gaze, and spatial-cue paradigms, the present invention designs some features based on the region of interest. Among them, for the blank-overlap paradigm, the region of interest is mainly the central and surrounding stimuli. For the person-gaze paradigm, the region of interest is mainly the face area of ​​the central person. For the spatial-cue paradigm, the region of interest is mainly the target object.

[0149] Total fixation time in the area of ​​interest: Calculate the cumulative amount of time the child fixates on the point in the area of ​​interest during the presentation of the paradigm. This feature reflects the degree of interest of the child in the area of ​​interest. The calculation formula is as follows:

[0150] ;

[0151] Indicates whether the gaze point of the mth frame falls within the area of ​​interest. Represents the cumulative number of fixations in the area of ​​interest from the 1st frame to the nth frame.

[0152] First fixation time of the interest area: Calculate the cumulative fixation time of the child's first fixation on the interest area. This feature reflects the child's initial impression and reaction to the interest area. The calculation formula is as follows:

[0153] ;

[0154] Indicates whether the gaze point of the mth frame falls within the area of ​​interest. It represents the cumulative number of fixations in the area of ​​interest from the first frame when the child first starts to fixate on the area of ​​interest to the end frame when the fixation leaves the area of ​​interest.

[0155] The first fixation latency of the area of ​​interest was calculated from the beginning of the paradigm presentation to the time when the child first fixed his gaze on the area of ​​interest. This feature reflects the child's reaction speed after the appearance of the area of ​​interest. The calculation formula is as follows:

[0156] ;

[0157] Indicates whether the gaze point of the mth frame falls within the area of ​​interest. Represents the cumulative number of fixations in the area of ​​interest from the first frame to the first frame when the child first starts to look at the area of ​​interest.

[0158] In view of the name-calling response paradigm, the present invention designs some special behavioral features, which are specifically designed as follows.

[0159] Active response time of head posture: Calculate the number of frames in which the yaw angle of the child's head posture is greater than 20 degrees within 3 seconds after being called. This feature reflects the degree of active head response of the child to familiar voices. The calculation formula is as follows:

[0160] ;

[0161] Indicates whether the yaw angle of the mth frame is greater than 20 degrees. It indicates the cumulative number of frames in which the yaw angle of the child's head posture is greater than 20 degrees, from the call frame when the child's name is called to the end frame 3 seconds after the name is called.

[0162] Latency of positive head gesture response: Calculate the time from when the child's name is called to when the child's head gesture yaw angle is greater than 20 degrees (the number of frames required is greater than 10, because a duration of more than 0.3 is considered a stable head deflection). This feature mainly extracts the preparation time required before the child makes a positive head response. The calculation formula is as follows:

[0163] ;

[0164] Indicates whether the yaw angle of the mth frame is greater than 20 degrees. It indicates the cumulative number of frames in which the yaw angle of the child's head posture is greater than 20 degrees, from the call frame when the child's name is called to the pos frame in which the child's head has a positive response.

[0165] Active response time of sight: Calculate the cumulative amount of time that the child's gaze point is outside the screen within 3 seconds after being called. This feature mainly reflects the degree of children's active response to familiar voices. The calculation formula is as follows:

[0166] ;

[0167] Indicates whether the gaze point in the mth frame falls outside the screen. It indicates the cumulative number of frames in which the child's gaze point falls outside the screen, from the call frame when the name is called to the end frame 3 seconds after the name is called.

[0168] Latency of positive eye gaze response: Calculate the time from when the name is called to when the child's gaze falls outside the screen (the duration must be greater than 10 frames, because a duration greater than 0.3 is considered a stable eye gaze deflection). This feature mainly extracts the preparation time required before the child makes a positive eye gaze response. The calculation formula is as follows:

[0169] ;

[0170] Indicates whether the gaze point in the mth frame falls outside the screen. It indicates the cumulative number of frames in which the child's gaze point falls outside the screen, from the call frame when the name is called to the pos frame in which the child's gaze produces a positive response.

[0171] It is understandable that the present invention proposes an intelligent screening system for ASD children based on multimodal behavioral characteristics, which can be used to assist doctors in screening for autism spectrum disorder (ASD). This method extracts multiple behavioral characteristics such as gaze point, facial expression and head posture by collecting and analyzing children's facial videos, and combines task paradigm design to effectively eliminate the influence of environmental factors on screening accuracy. At the same time, the method uses the LSTM model for classification based on the time series changes of features under different task scenarios to ensure the efficiency and accuracy of the screening results.

[0172] In order to achieve high-precision screening, the present invention has designed a number of representative task paradigms: blank-overlap, person gaze, spatial clues, and name-calling response. Each paradigm improves the sensitivity of screening by inducing differential behaviors of ASD children in attention shifting, social gaze, and spatial clue response. At the same time, corresponding feature extraction methods are designed according to the characteristics of different behaviors, and different feature extraction schemes are also designed for different paradigms. The sight point feature divides the gaze area and analyzes the gaze concentration in different situations; the facial expression feature counts the frequency of occurrence of multiple emotions to reflect emotional abnormalities; the head posture feature captures the posture deviation angle and reveals the body language differences of ASD children. The feature extraction scheme comprehensively captures the behavioral details of ASD children in different situations, providing rich data support for the subsequent classification process.

[0173] Experiments were conducted on a self-built ASD screening dataset. The multimodal features designed by the present invention were combined with the LSTM model to achieve a classification accuracy of 93.18% and a sensitivity of 94.67%. The experimental results show that the multimodal fusion of the three major features of gaze point, facial expression and head posture, as well as the multi-paradigm feature fusion of blank-overlap, character gaze, spatial clues, and name-calling response significantly improved the effectiveness of screening. Compared with traditional methods, the feature design and classification model of the present invention are more sensitive in identifying behavioral abnormalities, and the screening results are stable, ensuring the high accuracy and wide applicability of ASD screening.

[0174] A person skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, CD-ROM, etc.

[0175] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

[0176] It should be noted that the embodiments of the present invention can be implemented by hardware, software or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in processor control codes, such as such codes provided on a programmable memory or a data carrier such as an optical or electronic signal carrier.

[0177] In addition, although the operation of the method of the present invention is described in a particular order in the accompanying drawings, this does not require or imply that these operations must be performed in this particular order, or that all the operations shown must be performed to achieve the desired result. On the contrary, the steps described in the flow chart can change the order of execution. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step for execution, and / or one step can be decomposed into multiple steps for execution. It should also be noted that the features and functions of two or more devices according to the present invention can be embodied in one device. On the contrary, the features and functions of a device described above can be further divided into being embodied by multiple devices.

[0178] Although the invention has been described with reference to several specific embodiments, it should be understood that the invention is not limited to the specific embodiments disclosed. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. An intelligent screening system for children with autism, comprising a data acquisition module, a data preprocessing module and a feature extraction module, characterized in that: It also includes feature fusion module and classification module; The data collection module is used to collect facial videos of children completing various paradigm tasks; The data preprocessing module is used to obtain basic data related to the child's gaze point, facial expression and head posture based on the facial video; The feature extraction module is used to extract the sight point feature, facial expression feature and head posture feature from the basic data obtained in the data preprocessing module by using deep learning technology; The feature fusion module is used to fuse the gaze point features, facial expression features and head posture features; The classification module includes an LSTM model, which is used to integrate the fused features into time series data and input them into the LSTM model for classification analysis; Among them, the various paradigm tasks include the blank-overlap paradigm, the person-gaze paradigm, the spatial cue paradigm and the name-calling response paradigm; the blank-overlap paradigm is used to examine children's attention shifting ability based on their gaze shifting, head micro-movements and facial expression changes; the person-gaze paradigm is used to evaluate children's gaze behavior in social situations based on their gaze concentration, facial expression reactions and head posture in social gaze situations; the spatial cue paradigm is used to test children's sensitivity to external visual cues by observing children's gaze tracking patterns, facial expression sensitivity and head turning tendencies; the name-calling response paradigm is used to examine children's sensitivity to familiar sounds based on the guardian's call.

2. The intelligent screening system for children with autism according to claim 1, characterized in that: The deep learning techniques include: The visual Transformer model is used to calculate basic data related to where the child’s gaze falls; The MobileNetV3 model is used to calculate basic data related to children's facial expressions, including neutral, angry, scared, happy, and sad; The FSA-Net model is used to analyze basic data related to children's head posture angles, including pitch angle, horizontal angle and roll angle.

3. The intelligent screening system for children with autism according to claim 2, characterized in that: The calculating of basic data related to the child's sight point includes calculating the cumulative number of sight points, and the calculating of the cumulative number of sight points is used to reflect the sight point characteristics; The various paradigm tasks distribute different elements at different positions on the screen, and the calculation of the cumulative number of sight points is to calculate the cumulative number of sight points in different screen areas, and the formula is as follows: ； in, It is expressed as the cumulative number of nine areas from the 1st second to the nth second during the paradigm appearance time and the cumulative number of unpredictable sight points. - Indicates the number of sight points in 9 areas in 30 frames at the i-th second, It indicates the number of unpredictable gaze points in the 30 frames of the i-th second due to excessive head deflection.

4. The intelligent screening system for children with autism according to claim 2, characterized in that: The calculation of basic data related to children's facial expressions includes calculating the cumulative number of different emotions, and the calculation of the cumulative number of different emotions is used to reflect the facial expression characteristics; the formula is as follows: ； in, It is expressed as the cumulative number of seven expressions from the 1st second to the nth second during the time when the paradigm appears, and the cumulative number of unpredictable expression labels. - represents the number of seven expression labels in the 30 frames of the i-th second, It indicates the number of labels that cannot be predicted in the 30 frames of the i-th second due to excessive head deflection.

5. The intelligent screening system for children with autism according to claim 2, characterized in that: The basic data related to the child's head posture angle includes the calculated frequency of occurrence of the head posture angle, and the calculated frequency of occurrence of the head posture angle is used to reflect the head posture characteristics; the formula is as follows: Among them, the head posture angle range of -15° to 15° is "stable", the head posture angle range of 15° to 30° or -15° to -30° is "slightly deviated", and the head posture angle range of >30° or <-30° is "severely deviated". , , Indicates the cumulative number of yaw, pitch, and roll angles from the 1st second to the nth second during the pattern appearance time. arrive Indicates the number of Yaw angles in three intervals in the 30th frame of the i-th second, arrive Indicates the number of Pitch angles in three intervals in the 30th frame of the i-th second. arrive Indicates the number of Roll angles in three intervals in 30 frames of the i-th second.

6. The intelligent screening system for children with autism according to claim 1, characterized in that: The formula for the feature fusion module to fuse the gaze point feature, facial expression feature and head posture feature is: 。 7. The intelligent screening system for children with autism according to claim 1, characterized in that: The children's sensitivity to familiar sounds is specifically reflected by the active response time of head gesture and the latency time of head gesture response; The formula for calculating the active response time of the head posture is: ； in, Indicates whether the yaw angle of the mth frame is greater than 20 degrees; It indicates the cumulative number of frames in which the yaw angle of the child's head posture is greater than 20 degrees from the call frame when the child's name is called to the end frame 3 seconds after the name is called; The formula for calculating the latency of head gesture response is: ； in, Indicates whether the yaw angle of the mth frame is greater than 20 degrees. It indicates the cumulative number of frames in which the yaw angle of the child's head posture is greater than 20 degrees, from the call frame when the child's name is called to the pos frame in which the child's head has a positive response.

8. The intelligent screening system for children with autism according to claim 7, characterized in that: The children's sensitivity to familiar sounds can also be reflected by the positive response time of gaze and the latency time of positive response of gaze; The formula for calculating the active response time of the line of sight is: ； in, Indicates whether the gaze point in the mth frame falls outside the screen. It represents the cumulative number of frames in which the child's gaze point falls outside the screen from the call frame when the name is called to the end frame 3 seconds after the name is called; The formula for calculating the latency of the gaze positive response is: ； in, Indicates whether the gaze point in the mth frame falls outside the screen. It indicates the cumulative number of frames in which the child's gaze point falls outside the screen, from the call frame when the name is called to the pos frame in which the child's gaze produces a positive response.

9. The intelligent screening system for children with autism according to claim 1, characterized in that: The blank-overlap paradigm includes a blank task and an overlapping task. The blank task first displays a central visual stimulus, then displays a blank screen, and finally displays a target visual stimulus at another location. The overlapping task first displays a central visual stimulus, then the central visual stimulus does not disappear, and displays a target visual stimulus at another location. The character gaze paradigm first displays a "+" locator to calibrate the child's gaze; then displays the character's facial image; The spatial cue paradigm first displays a "+" sign as a gaze calibration point, and then appears a screen containing a cue image and a distractor; The name-calling response paradigm first plays a video to attract the child's attention, and then during the video playing, two Bluetooth speakers are placed on the left and right sides behind the child to play the guardian's call.

10. The intelligent screening system for children with autism according to claim 9, characterized in that: The feature extraction module can also be used to extract target gaze features based on facial videos of multiple paradigm tasks; the target gaze features are used to respond to the total gaze time, first gaze time and first gaze latency time of the child's gaze on the area of ​​interest; In the blank-overlap paradigm, the region of interest is the central and surrounding stimuli; in the figure-gaze paradigm, the region of interest is the facial image of the central figure; in the spatial cue paradigm, the region of interest is the target image; The total fixation time is the cumulative amount of time that the child fixates on the point in the region of interest during the paradigm presentation time, and is used to reflect the child's interest in the region of interest. The calculation formula for the total fixation time is: ； in, Indicates whether the gaze point of the mth frame falls within the area of ​​interest. Represents the cumulative number of fixations in the area of ​​interest from the 1st frame to the nth frame; The first fixation time is the cumulative amount of time that the child first fixates on the area of ​​interest, and is used to reflect the child's initial impression and reaction to the area of ​​interest. The calculation formula for the first fixation time is: ； in, Indicates whether the gaze point of the mth frame falls within the area of ​​interest. It represents the cumulative number of fixations in the interest area from the first frame when the child first starts to fixate on the interest area to the end frame when the fixation leaves the interest area; The first fixation latency is the cumulative fixation time from the start of the paradigm to the first fixation of the child on the area of ​​interest, which is used to reflect the child's reaction speed after the appearance of the area of ​​interest. The calculation formula is as follows: ； in, Indicates whether the gaze point of the mth frame falls within the area of ​​interest. Represents the cumulative number of fixations in the area of ​​interest from the first frame to the first frame when the child first starts to look at the area of ​​interest.

Images