Low age children autism screening method based on browser camera eye movement data and multi-modal behaviors

Abstract

The application discloses a low-age child autism screening method based on browser camera eye movement data and multi-modal behavior, and belongs to the field of computer vision. The method comprises the following steps: collecting image streams and audio signals; obtaining screen gaze drop points of a formal task section based on the image streams; taking a task stimulus event as a time anchor point, aligning the screen gaze drop points, head posture, blink state, upper body and acoustic summary, and constructing a task-level multi-modal response unit; and generating a risk assessment result based on the task-level multi-modal response unit through a multi-modal hierarchical time sequence model. The application converts calibration section samples into conversation-specific screen drop point projections, realizes stable screen gaze drop point estimation under uncontrolled conditions, takes a task stimulus event as a common anchor point, organizes eye movement, head, upper body and sound recording into a task-level multi-modal response unit within the same test, makes each mode have consistent task meaning and time relationship, and completes quality control before scoring, and finally outputs an evaluation result.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and digital medical technology, specifically relating to a method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior quantification. Background Technology

[0002] Autism Spectrum Disorder (ASD)-related risks typically manifest in infancy and preschool years as differences in social orientation, joint attention, social interaction, disengagement, interests, perceptual responses, and repetitive behaviors. The period between 12 and 36 months is a crucial window for early screening and referral; however, children at this stage have limited language expression, limited instruction comprehension, and weaker stable gaze and proactive cooperation abilities. Traditional assessment processes relying on questionnaires, manual observation, in-person interviews, or laboratory equipment are difficult to implement cost-effectively and frequently in families, primary care facilities, pediatric clinics, or research cohorts.

[0003] Existing eye-tracking research and products often rely on specialized eye trackers, fixed head positions, or require strong cooperation. While ordinary camera-based eye-tracking solutions lower the hardware barrier, they still have significant limitations in scenarios involving children aged 12-36 months, as they only output the point where the subject's gaze falls on the screen. Children may frequently turn their heads away from the screen, move closer to the screen, shake their bodies, briefly close their eyes, blink, make sounds, or have their vision briefly obscured by a caregiver. These phenomena may lead to a decrease in data acquisition quality, but they may also contain behavioral information related to developmental status. Simply deleting all non-ideal frames can easily result in the loss of behavioral evidence; however, feeding them into the model indiscriminately may misclassify failed data acquisition as risk characteristics.

[0004] Furthermore, ASD-related risk screening should not simply replace the memory encoding, delayed recognition, and search efficiency indicators in adult cognitive or dementia screening tasks with the ASD name. Programs for children aged 12-36 months should be constructed around direct social gaze, eye contact, head movements, gestures and pointing cues, joint attention, object-social preferences, attention disengagement, and non-social controls. In addition to collecting data on eye movement focus, head movements, blinking, upper body movements, and speech / acoustic features should also be assessed.

[0005] Therefore, there is an urgent need for a browser-based multimodal behavior assessment technology solution for non-cooperative scenarios of young children, which can complete low-friction data collection using ordinary web browsers, ordinary front-facing cameras, and microphones; and can complete conversation-adaptive screen gaze point estimation, multimodal behavior feature extraction, quality control classification, and subject-level ASD-related risk probability output without relying on a dedicated eye tracker or requiring children to actively respond. Summary of the Invention

[0006] To address the aforementioned problems in existing technologies, this invention provides a method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior. This method targets children aged 12-36 months, using the ASD social attention task chain as a stimulus carrier. Multi-point attraction calibration and conversation-adaptive screen-based projection ensure the availability of screen gaze points. A unified timeline aligns eye movements, head movements, blinks, upper body movements, and speech / acoustic signals. Quality control gating distinguishes between unreadable, retestable, partially modal missing, and readable samples. A multimodal hierarchical temporal risk inference model outputs the probability of ASD-related risks. This invention achieves low-friction multimodal quantitative screening of ASD-related risks in children aged 12-36 months without relying on a dedicated eye tracker or requiring children to actively press buttons or provide verbal responses. It maintains the stability and interpretability of risk output despite fluctuations in data acquisition due to head position drift, blinking, screen absence, occlusion, and environmental noise.

[0007] To achieve the above objectives, the technical solution of the present invention is as follows:

[0008] A method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior includes:

[0009] S1: Collect image streams and audio signals of the face, periorbital area, head, and upper body of the children during the viewing of the preset task stimulus sequence; extract key points of the face and periorbital area, head posture, blinking state, and upper body key points frame by frame from the image stream; extract acoustic summaries from the audio signals; and construct original multimodal samples frame by frame; the task stimulus sequence includes various task stimulus segments of the calibration segment and the formal task segment executed sequentially;

[0010] S2: Construct a calibration support set based on the original multimodal samples of the calibration segment, solve the final session projection matrix, and calculate the screen gaze point of the subject children in the formal task segment based on the final session projection matrix.

[0011] S3: Using the task stimulus events in the preset task stimulus sequence as time anchors, align the screen gaze point, head posture, blinking state, upper body movement and acoustic summary to construct a task-level multimodal response unit for each task stimulus segment, and distinguish between the behavior-not-occurred state and the modality-undeterminable state in the task-level multimodal response unit.

[0012] S4: Calculate quality control indicators and determine whether to enter the risk reasoning stage;

[0013] S5: When the quality control indicator result is passed, based on the task-level multimodal response unit and the effective state of each modality, an autism risk assessment result is generated through a multimodal hierarchical temporal risk reasoning model.

[0014] Furthermore, in step S2, the screen gaze point of the children in the formal task segment is calculated, including:

[0015] S2.1: Extract the original multimodal samples of the calibration segment based on the task identifier of the task stimulus segment to which each original multimodal sample belongs, and construct the calibration support set based on the original multimodal samples of the calibration segment;

[0016] S2.2: Solve the initial session projection matrix based on the calibration support set, and iteratively refine the initial session projection matrix to obtain the final session projection matrix;

[0017] S2.3: For any valid facial and periorbital key point sample within the formal task segment, perform mapping calculations using the final session projection matrix to obtain the corresponding screen gaze point.

[0018] Furthermore, in step S2.1, constructing the calibration support set based on the original multimodal samples of the calibration segment includes:

[0019] Identify the time window of each calibration anchor point in the calibration segment, that is, the time period presented by each calibration anchor point;

[0020] The original multimodal samples of the calibration segment are divided into the corresponding anchor time windows based on the timestamp;

[0021] Within each anchor point time window, invalid frame removal, stable head pose screening, and blink frame filtering are performed to obtain valid samples;

[0022] The valid samples within each anchor time window are aggregated into a calibration support set, which is represented as follows:

[0023]

[0024] in, To support the sample size; Indicates the calibration support set within the first A set of coordinates of key facial and periorbital points for each supporting sample; Indicates the first The anchor point screen reference coordinates corresponding to the anchor point time window of each supporting sample; Indicates the first Confidence weights of each supporting sample; Indicates the first The anchor point number corresponding to the anchor point time window of each supporting sample; Indicates the first The timestamps of the supporting samples.

[0025] Furthermore, in step S2.2, the final session projection matrix is ​​obtained based on the following method:

[0026] Based on the calibrated support set, the initial session projection matrix is ​​solved using regularized least squares with projection prior constraints:

[0027]

[0028] in, This is the initial session projection matrix. The projective prior obtained from offline training; The regularization coefficient is used. Represents the square of the Frobenius norm; Indicates the calibration support set within the first Extended features of each supporting sample ; The output of the shared geometric representation network represents the first... Geometric descriptive features of each supporting sample , Represents a shared geometric representation network. Indicates the first A set of normalized key points obtained by performing translation and scale normalization on the coordinates of all facial and periorbital key points within a support sample; This is a local eye geometry vector constructed based on normalized keypoints; Represents the head pose vector;

[0029] Using the initial session projection matrix as the initial value, perform multiple rounds of recursive least squares refinement on the calibration support set. After each round of refinement, calculate the anchor point residual. When the decrease in the anchor point residual between two adjacent rounds is less than a preset threshold or reaches the preset maximum number of iterations, stop the refinement and use the matrix obtained from the current round update as the final session projection matrix.

[0030] Furthermore, the multi-round recursive least squares refinement of the calibration support set is specifically as follows:

[0031] Let the first Round The inverse covariance matrix of each supporting sample before update is: Then the recursive update formula is:

[0032]

[0033]

[0034]

[0035]

[0036] in, For the first Round absorption Gain vector for each support sample; and The first Round Each session projection matrix supports both the pre- and post-update samples; For the first Round The two-dimensional reprojection residuals of each supporting sample, i.e., the real screen reference coordinates. Compared with current forecasts The difference between them; and The first Round A support sample inverse covariance matrix before and after the update.

[0037] Furthermore, in step S3, the construction of task-level multimodal response units for each task stimulus fragment includes:

[0038] During the response period after each task stimulus segment is played, eye-tracking events, head events, upper body movement events, eyelid behavior events, and sound response events are constructed based on the screen gaze point, head posture, upper body key points, blinking state, and acoustic summary, respectively, to obtain a multimodal behavioral event sequence composed of the eye-tracking events, head events, upper body movement events, eyelid behavior events, and sound response events; each modal behavioral event records at least the event type, the start and end times of the event type, the task stimulus segment and trial number to which it belongs, and the valid state;

[0039] Based on the multimodal behavioral event sequence of any task stimulus segment, construct the response feature vector of the corresponding event type according to the screening purpose of the task stimulus segment, including whether the corresponding event type has occurred, the time of first occurrence, the end time, the number of occurrences, and the effective status;

[0040] The response feature vectors of different event types under the same modal behavioral event are concatenated in a preset order to form response features of each modal behavioral event, namely eye movement event response features, head event response features, upper body movement event response features, eyelid behavior event response features, and sound response event response features;

[0041] The modal behavioral event response features of the same task stimulus fragment within the same trial are combined into a task-level multimodal response unit.

[0042] Furthermore, the task-level multimodal response unit records the task stimulus fragments, behavioral event response characteristics of each modality, effective states of each modality, and cross-modal response relationships formed by multiple modalities. When all the basic modalities constituting the cross-modal response relationship are determinable, the cross-modal response relationship is determined to have occurred. When any basic modality is not determinable, the cross-modal response relationship is recorded as an undeterminable state to distinguish it from the state where the behavior has not occurred.

[0043] Furthermore, the quality control indicators include: calibration anchor point coverage, task completion rate, gaze efficiency, AOI attribution rate, and modal efficiency.

[0044] If the coverage of the calibration anchor points does not meet the preset conditions, the system will trigger recalibration. If the calibration pass conditions are still not met after multiple calibrations, the system will output that the current data collection is unreadable.

[0045] When the calibration anchor coverage meets the preset conditions, but the task completion rate or gaze effectiveness does not meet the preset conditions, the output needs to be retested.

[0046] When the task completion rate and gaze effectiveness meet the preset conditions, but the modality effectiveness in the head, upper body, or audio does not meet the conditions, the corresponding modality is recorded as missing. Inference is only allowed to continue if the remaining valid records are supported in the state of modality missing; otherwise, the output needs to be retested.

[0047] Furthermore, the multimodal hierarchical temporal risk reasoning model includes a modality encoding layer, a task-level fusion layer, a task sequence encoding layer, and a risk output layer;

[0048] Let the first In the first task-level multimodal response unit The modal behavioral event response characteristics are: ,in Representing eye movements, eyelid movements, head movements, upper body movements, or vocal modalities; constructing a modality missing mask based on the valid states of each modality, where the corresponding element in the modality missing mask is... The modality is set to 1 when it is available and 0 when it is not available.

[0049] The modality coding layer encodes the response features of each modality behavioral event respectively:

[0050]

[0051] in, Indicates the first Feature encoding network corresponding to each modality Indicates the first Missing modality features used when a modality is unavailable ;

[0052] The task-level fusion layer concatenates the modality coding features and modality missing masks from the same task-level multimodal response unit:

[0053]

[0054] in, Indicates a fully connected converged network. Indicates the first The fusion characteristics of multiple task-level multimodal response units;

[0055] The task sequence coding layer employs a bidirectional gated recurrent unit (BiGRU) network to process task-level fusion features according to the task chain and trial order.

[0056]

[0057] in, Indicates the number of task-level multimodal response units. Indicates the first [number] after task sequence encoding Each task representation is used; the system averages and summarizes the representations from each task to obtain the subject-level representation. :

[0058]

[0059] The risk output layer represents the subject level. The data is concatenated with the age-based encoding and then output using the Sigmoid function to determine the probability of ASD-related risk. :

[0060]

[0061] in, Represents the month-age encoding function, For the Sigmoid function, , These are the weight vector and the bias, respectively. The age of the children being tested is in months.

[0062] When the risk probability If the result is greater than or equal to the preset screening threshold, a high-risk screening result will be output; otherwise, a low-risk screening result will be output.

[0063] Furthermore, the preset task stimulus sequence includes the following task stimulus segments executed sequentially:

[0064] A multi-point attraction calibration segment is used to present attention-grabbing stimuli at multiple known anchor points on the screen to acquire the calibration support set;

[0065] The calibration quality verification section is used to verify the usability of the final session-level projection matrix;

[0066] The direct social segment is used to present stimuli that include human faces or social interactions in order to quantify the spontaneous orientation of the children in the study to social stimuli.

[0067] Gestures and joint attention segments were used to present stimuli containing cueing cues and targets to quantify the joint attention abilities of the children in the study.

[0068] The object-social preference section is used to present social and non-social elements simultaneously or alternately to quantify social preferences.

[0069] The non-social control segment is used to present control stimuli without people or social interaction in order to differentiate the children's responses to physical stimuli.

[0070] The beneficial effects of this invention are:

[0071] (1) This invention utilizes a regular browser, front-facing camera, and microphone to complete the natural viewing task, without requiring children aged 12-36 months to press buttons, answer questions, or maintain a fixed head position.

[0072] (2) The present invention uses task stimulus events as common anchor points to organize eye movement, eyelid, head, upper body and sound recording into multimodal behavioral events and task-level multimodal response units within the same trial, so that each modality has consistent task meaning and time relationship.

[0073] (3) This invention forms complementary multi-task multimodal evidence through direct social interaction, gestures and joint attention, object-social preferences and non-social control tasks, and uses the common response relationship under the same stimulus for ASD-related risk inference.

[0074] (4) Before risk reasoning, this invention distinguishes between behaviors that have not occurred, modalities that are undeterminable, and conversations that are of substandard quality, thus avoiding misinterpreting collection failures or partial modal absences as low-risk behaviors.

[0075] (5) This invention outputs a single ASD-related risk probability, risk stratification and corresponding evidence summary, which facilitates primary screening, family retesting, waiting room triage and further professional assessment. Attached Figure Description

[0076] Figure 1 This is a flowchart of a method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior, as described in an embodiment of the present invention. Detailed Implementation

[0077] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0078] like Figure 1 As shown, this invention provides a method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior. This method uses task-related stimuli as common anchor points, sequentially organizing multimodal behavior records into multimodal behavior events, task-related response features, and task-level multimodal response units. ASD-related risk results are obtained from the quality-controlled multi-task multimodal response sequence. Specifically, the method includes the following steps:

[0079] S1: Collect image streams and audio signals of the face, periorbital area, head, and upper body of the children during the viewing of a pre-set task stimulus sequence; extract facial and periorbital key points, head posture, blinking status, and upper body key points frame by frame from the image stream; extract acoustic summaries from the audio signals; and construct original multimodal samples frame by frame. Specifically, this includes:

[0080] S1.1: Launch the screening page in the user's standard web browser, complete camera permission requests, microphone permission requests, volume prompts, light checks, face alignment guidance, viewing distance estimation, and device orientation detection. The system can prompt guardians to place children within a safe distance in front of the screen and requires minimizing prompts, obstructions, and human intervention during the task.

[0081] S1.2: The system loads a preset task stimulus sequence with a unified timeline—the ASD social attention task chain. This task chain uses short videos, animations, sounds, or interactive segments to be presented in a natural viewing manner, without requiring children to press buttons, touch screens to select, verbally answer, or complete complex rules. The system calls the front-facing camera to collect image streams of the subject's face, eye area, head, and upper body during the viewing process, and calls the microphone to collect speech or acoustic signals.

[0082] Each stimulus segment in the task chain is assigned a unique task identifier (taskId), media index (mediaIndex), trial identifier (trialId), start time (startTime), end time (endTime), and AOI (area of ​​interest) metadata. The AOI metadata is used to identify screen areas where gaze attribution can be determined, such as people, faces, eyes, target objects, and non-social motion areas. The task chain includes a multi-point attraction calibration segment, a calibration quality verification segment, and a formal task segment, which are executed sequentially. The formal task segments are, respectively, a direct social segment, a gesture and joint attention segment, an object-social preference segment, and a non-social control segment.

[0083] The multi-point attraction calibration segment sequentially presents attention-grabbing stimuli such as animation, sound, or human face appearance at at least nine known anchor points, including the center, four corners, and midpoints of the four sides of the screen, to collect session-specific facial key point to screen coordinate mapping support samples.

[0084] The calibration quality verification section presents verification points located at the center, four corners, or edges, or briefly echoes the gaze point calculated from the current mapping, to verify whether the current projection is usable before the formal task begins.

[0085] The direct social segment presents real faces, close-up smiles, peek-a-boo, greetings, looking at the camera, or calling gestures, used to quantify the spontaneous orientation and sustained engagement of the children in the study with people, faces, eyes, and mouths.

[0086] The gestures and common attention segments present stimuli such as head turning, eye cues, head cues, hand gestures, pointing to a target on or off the screen, to quantify whether the children in the subjects look at the social cues first, then turn to the target along the direction of the cues, and form a triadic look-back between the person and the target.

[0087] The object-social preference segment simultaneously or alternately presents human interaction, toys, or non-social elements in the same frame or adjacent segments. The non-social elements refer to objects, toys, or moving images that do not contain people, faces, or social interactions. This segment is used to quantify the dwell ratio, first selection, number of switching, and excessive dwell on dynamic objects or local features between social AOI and object AOI.

[0088] The non-social control segment presents a control stimulus without people or social interaction. Its duration, brightness, movement amplitude, or sound intensity are roughly similar to the social stimulus, so as to distinguish the differences in the subjects' responses to social meaning and the differences in their responses to simple changes in movement, brightness, or sound.

[0089] S1.3: During the playback of each stimulus segment in the task chain, the browser performs keypoint detection frame by frame on the image stream captured by the front-facing camera, and extracts facial and peri-eye keypoints, head posture, blinking state, and upper body keypoints. Among them, facial and peri-eye keypoints include the inner and outer corners of the eyes, the center of the iris, the upper and lower eyelids, the eyelid contours, the bridge of the nose, the tip of the nose, the space between the eyebrows, the forehead, and the chin; head posture is the head yaw angle, pitch angle, and roll angle calculated from the facial and peri-eye keypoints, used to characterize the head orientation; blinking state is the process of opening, closing, or blinking determined by the distance between the upper and lower eyelids or the aspect ratio of the eyes; upper body keypoints include the neck, shoulders, center of the torso, elbows, wrists, and hands.

[0090] The browser side synchronously extracts acoustic summaries from speech or acoustic signals, including vocalization events, sound energy, fundamental frequency, pitch, rhythm, non-word vocalization categories (such as crying, laughing, babbling), background noise signal-to-noise ratio, and sound responses after people call out or social events.

[0091] For frame t, construct the original multimodal samples. :

[0092]

[0093]

[0094] in, Indicates the session identifier; Indicates the subject's identification; Indicates the subject's age in months; The timestamp of frame t after it has been aligned to the unified timeline; This represents the set of facial and periorbital key points in frame t. This represents the set of key points for the upper body in frame t. This represents the acoustic summary of frame t. This represents the head pose vector in frame t. This indicates the blinking state in frame t. This represents a summary of upper body motion obtained from the inter-frame displacement of upper body keypoints. The mask represents the validity of the original modalities such as face, eyes, head, blinking, upper body, and audio. When a missing face, insufficient number of key points, strong occlusion, excessive facial deflection, off-screen, invisible eyes, invisible upper body key points, low audio signal-to-noise ratio, or occlusion by a guardian is detected, the corresponding modality is marked as invalid or low confidence. It represents metadata such as screen width and height, browser type, estimated camera frame rate, device orientation, acquisition mode, guardian assistance status, lighting and noise.

[0095] S1.4: The browser buffers the original multimodal samples locally and uploads them asynchronously to the processing side in a compressed block manner, thus decoupling stimulus playback from network transmission.

[0096] S2: Construct a calibration support set based on the original multimodal samples from the calibration segment, solve for the final session-level projection matrix, and calculate the screen gaze points of the children in the formal task segment based on the final session-level projection matrix. Specifically, this includes:

[0097] S2.1: Extract the original multimodal samples of the calibration segment based on the task identifier of the task stimulus segment to which each original multimodal sample belongs, and construct the calibration support set based on the original multimodal samples of the calibration segment; as follows:

[0098] After receiving the raw multimodal samples and reconstructing the complete sample sequence, the processing side first sorts them according to timestamps, and then segments the samples into corresponding task stimulus segments according to taskId, mediaIndex, and trialId. Raw multimodal samples from the multi-point attraction calibration segment are extracted. The processing side identifies each anchor point time window based on a pre-recorded anchor point presentation timeline, and divides the raw multimodal samples of the multi-point attraction calibration segment into the corresponding anchor point time windows according to timestamps. Within each anchor point time window, invalid frame removal, stable head pose screening, and blink frame filtering are performed to obtain valid frame samples. The remaining valid frame samples within all anchor point time windows are then aggregated into a calibration support set. :

[0099]

[0100] in, To support the sample size; Indicates the calibration support set within the first A set of coordinates of key facial and periorbital points for each supporting sample; Indicates the first The anchor point screen reference coordinates corresponding to the anchor point time window of each supporting sample, that is, the calibration anchor point coordinates that the subject child should look at at that moment. Indicates the first The confidence weight of each supporting sample can be obtained by weighting the overall confidence of facial and periorbital key points, the visibility of periorbital key points, and head posture stability. If continuous confidence is not currently output, the confidence weight of the selected valid supporting samples will be uniformly set to 1. ; Indicates the first The anchor point number corresponding to the anchor point time window of each supporting sample; Indicates the first A timestamp for each supporting sample is used to bind anchor time windows and a unified timeline.

[0101] S2.2: Solve for the initial session projection matrix based on the calibrated support set, and iteratively refine the initial session projection matrix to obtain the final session projection matrix; specifically as follows:

[0102] S2.2.1: Construct normalized keypoints, local eye geometry vectors, and head pose vectors based on the calibration support set;

[0103] S2.2.2: Encode the normalized keypoints, local eye geometry vectors, and head pose vectors to obtain geometric description features, and further construct extended features;

[0104] S2.2.3: Introducing projection priors, based on the calibration support set, extended features, and projection priors, the sum of reprojection error and regularization term is minimized to obtain the initial session projection matrix;

[0105] S2.2.4: Using the initial session projection matrix as the initial value, perform multiple rounds of recursive least squares refinement on the calibration support set to absorb the residuals and obtain the final session projection matrix.

[0106] In one embodiment, the normalized keypoints are constructed as follows:

[0107] For any support sample within the calibrated support set Inner Key points on the face and around the eyes: coordinates Perform translation and scale normalization to obtain normalized key points. :

[0108]

[0109] in, A facial reference center; Scale factor; To prevent stable terms with a denominator of zero; Take the midpoint between the outer corners of the left and right eyes. Take the Euclidean distance between the outer corners of the eyes; or Take the midpoint between the centers of the left and right eyes. Take the Euclidean distance between the centers of the left and right eyes.

[0110] In one embodiment, the local eye geometry vector is constructed based on normalized keypoints, specifically as follows:

[0111] Let the centers of the left and right irises be respectively and The coordinates of the inner and outer corners of the left eye are respectively , The coordinates of the inner and outer corners of the right eye are respectively , The key points of the upper and lower eyelids of the left eye are respectively , The key points of the upper and lower eyelids of the right eye are respectively , Common anchor point set Includes key points such as the bridge of the nose, the tip of the nose, and the area between the eyebrows; local eye geometry vectors. Concatenate the following subvectors in a fixed order: Relative to each , Two-dimensional offset, Relative to each , Two-dimensional offset; to , Normalized distance, to , The normalized distance; the opening and closing of the left and right eyelids, which are the ratios of the distance between the upper and lower eyelids to the distance between the inner and outer corners of the eyes; , to The normalized distances of all common anchor points; and the differences and ratios of the corresponding quantities for the left and right eyes (i.e., the same parameters corresponding to the left and right eyes).

[0112] In one embodiment, step S2.2.2, which involves encoding the normalized keypoints, local eye geometry vectors, and head pose vectors to obtain geometric description features, and further constructing extended features, specifically includes:

[0113]

[0114] in, Indicates the first The normalized keypoint set of each supporting sample; Represents the head pose vector; , and These represent the nose yaw angle, pitch angle, and roll angle, respectively. The relative depth surrogate quantity derived from interocular distance or facial scale; This represents a shared geometric representation network (which may employ an equivariant graphical neural network EGNN, a lightweight multilayer perceptron, or other equivalent structures) or a parameterized feature projector; Indicates by The output of the first Geometric description features of each supporting sample;

[0115] Based on the geometric description features Build extended features .

[0116] In one embodiment, in step S2.2.3, by introducing a projection prior, the objective function can be expressed as:

[0117]

[0118] in, The projective prior obtained from offline training; The regularization coefficient is used. denoted as the squared Frobenius norm. The objective function consists of two terms: the first is the reprojection error, which measures how well the projection matrix fits the current session data; the second is the regularization term, which constrains the projection matrix not to deviate too far from the general prior.

[0119] In one embodiment, to improve the conformity of the projection results to the calibration anchor points, the shared geometric representation network is trained during the offline training phase. Introducing calibration anchor point fidelity loss :

[0120]

[0121] in, The session projection matrix is ​​obtained from the current support set; For the first Extended features of each supporting sample; For the first The anchor point screen reference coordinates corresponding to the anchor point time window of each supporting sample. The calibration anchor point fidelity loss and the query sample prediction loss are jointly optimized, so that the encoder not only pursues the reduction of overall error, but also explicitly reduces the reprojection error on the calibration anchor point.

[0122] In one embodiment, in step S2.2.4, using the initial session projection matrix as the initial value, multiple rounds of recursive least squares refinement are performed on the calibration support set to absorb residuals, resulting in the final session projection matrix; specifically:

[0123] Let the first Round The inverse covariance matrix of each supporting sample before update is: Then the recursive update formula is:

[0124]

[0125]

[0126]

[0127]

[0128] in, For the first Round absorption Gain vector for each support sample; and The first Round Each session projection matrix supports both the pre- and post-update samples; For the first Round The two-dimensional reprojection residuals of each supporting sample, i.e., the real screen reference coordinates. Compared with current forecasts The difference between them; and The first Round Each support sample inverse covariance matrix before and after update; For the first Extended features of each supporting sample; during the first round of initialization , , To initialize the regularization coefficients, It is an identity matrix; subsequent rounds use the last value of the previous round as the initial value.

[0129] Using the recursive formula described above, each iteration fine-tunes the projection matrix based on the residuals of the current supporting samples. The multi-round iteration process essentially allows the projection matrix to continuously absorb residuals, gradually approaching the optimal solution. This iterative refinement mechanism significantly improves the observability of the projection results to the calibration anchor points, ensuring that the projected screen gaze point coordinates are as close as possible to the actual calibration anchor point coordinates. This solves the problem of static calibration failure and improves the accuracy of screen gaze point prediction.

[0130] In one embodiment, the forgetting factor of the recursive least squares algorithm is set to 1 to maintain accurate recursive absorption of the current calibrated support set; the anchor point residual is calculated after each round:

[0131]

[0132] in, The length of the screen diagonal; Indicates the first All rounds completed A session projection matrix supporting sample absorption;

[0133] When the residual decrease rate of two adjacent rounds meets the requirement If the update is stopped early, the final session projection matrix will be obtained. This matrix is ​​used to calculate the screen gaze landing point for all subsequent formal task segments; otherwise, continue to the next round of refinement until the stopping condition is met or the maximum number of rounds is reached; the number of iteration rounds is set according to actual needs, for example, 3 to 5 rounds.

[0134] S2.3: For any valid facial and periorbital keypoint sample within the formal task segment, perform mapping calculations using the final session projection matrix to obtain the corresponding screen gaze point. :

[0135]

[0136] in, and These are the x and y coordinates in the screen coordinate system, respectively; This represents the extended features obtained from the effective multimodal samples of frame t in the formal task segment using the same method as in step S2.2, i.e. , , , The first The normalized keypoint set, local eye geometry vector, and head pose vector of the effective multimodal samples in the frame.

[0137] S3: Using the task stimulus events in the preset task stimulus sequence as time anchors, align the screen gaze point, head posture, blinking state, upper body movement, and acoustic summary to construct a task-level multimodal response unit for each task stimulus segment, and distinguish between the behavior-not-occurred state and the modality-undeterminable state in the task-level multimodal response unit. Specifically, this includes:

[0138] S3.1: When the facial and eye data obtained in step S1 is valid (i.e. If the calibration result of step S3 is qualified (for it to be effective), then the screen gaze point obtained in step S2 is... Record it as valid with high confidence; otherwise, record it as invalid or with low confidence. Then, according to the taskId, mediaIndex, trialId, and time information recorded in step S1, mark the screen gaze point. The head posture, blinking state, upper body movement, and acoustic summary collected in step S1 are mapped to the same task stimulus fragment.

[0139] S3.2: During the response period after each task stimulus segment is played, based on the screen gaze point, head posture, upper body key points, blinking state, and acoustic summary, eye movement events, head events, upper body movement events, eyelid behavior events, and sound response events are constructed respectively, resulting in a multimodal behavioral event sequence composed of the eye movement events, head events, upper body movement events, eyelid behavior events, and sound response events; each modal behavioral event records at least the event type, the start and end times of the event type, the task stimulus segment and trial number to which it belongs, and the valid state. When the data of a certain modality is invalid ( When the modality is invalid, the system records the modality as undecidable and does not record the behavior corresponding to the modality as not having occurred.

[0140] Specifically, eye-tracking events such as gaze entry, gaze persistence, gaze exit, and shift between AOIs are generated based on the relationship between the screen gaze point and the AOI; head events such as person orientation, target orientation, and off-screen or back-screen are generated based on head posture vectors; upper body movement events such as approaching, leaving, swinging, or repetitive movements are generated based on upper body key points and upper body movement summaries; sound response events after children's vocalization and task events are generated based on acoustic summaries; short-term natural blinks are retained as eyelid behavior events; and periods of continuous eye closure or when the eyes are not visible are recorded as undeterminable to avoid misinterpreting them as a lack of gaze response.

[0141] In one embodiment, eye-tracking events are constructed as follows: during the task response period, when the screen gaze point enters a certain AOI and remains there for a preset time, a gaze entry and sustained stay event is formed; when the screen gaze point leaves the AOI and remains there for a preset time, a gaze exit event is formed; when the screen gaze point moves from one AOI to another AOI, an AOI transfer event is formed.

[0142] The head event is constructed as follows: the system first uses the median effective head posture before the occurrence of the corresponding task stimulus as the baseline posture, and then calculates the head posture changes during the response period:

[0143]

[0144] in, , As the baseline attitude;

[0145] The task configuration records the corresponding horizontal and vertical directions based on the positions of the character's AOI and the target's AOI relative to the center of the screen. When When the direction of change aligns with the direction of a certain AOI, the magnitude of the change reaches a preset head movement threshold, and continues for a preset time, a head orientation event towards that AOI is formed; if the AOI is a person or face, it is recorded as person orientation; if the AOI is a target object, it is recorded as target orientation. or When a device moves beyond a preset screen orientation range for a preset time, an off-screen event is generated; when it subsequently re-enters the screen orientation range for a preset time, a return-to-screen event is generated.

[0146] The upper body movement event is constructed as follows: the upper body posture before the task stimulus occurs is used as a reference, and the body movement is judged based on the positional changes of the key points of the upper body; when the body moves continuously towards the screen, it forms an approach event; when it moves continuously away from the screen, it forms a departure event; when the body moves repeatedly left and right or forward and backward within a certain period of time, it forms a swinging event; when the same type of movement is repeated a preset number of times, it forms a repetitive movement event.

[0147] The eyelid behavioral events are constructed as follows: the eye opening and closing state is determined based on the distance between the upper and lower eyelids; a blinking event is formed when the eyelids reopen within a preset blinking time after closing; a continuous eye-closing event is formed when the closing time exceeds the preset duration; when the eyes are obscured or key points of the eyelids cannot be identified, it is recorded as undeterminable.

[0148] The sound response event is constructed as follows: within a preset response period after the task stimulus occurs, when a child continuously vocalizes for a preset time, a sound response event is formed, and the start and end times of the vocalization are recorded; if the audio is invalid or it cannot be confirmed whether the sound comes from the child, it is recorded as undeterminable.

[0149] S3.3: Based on the multimodal behavioral event sequence of any task stimulus segment, construct the response feature vector of the corresponding event type according to the screening purpose of the task stimulus segment, including whether the corresponding event type has occurred, the time of first occurrence, the end time, the number of occurrences, and the effective status.

[0150] In one embodiment, to convert a variable number of multimodal behavioral events into a fixed-length input that the model can use, for the first... Any event type in the task stimulus fragment The system constructs a response feature vector within the corresponding response time period:

[0151]

[0152] in, Indicates event type Whether it happened; This indicates the process from the occurrence of the task stimulus to the event type. The time of the first start; This indicates the progression from the occurrence of the task stimulus to the event type. Final end time; Indicates event type The cumulative duration; Indicates the number of times it appears; This indicates the proportion of the cumulative duration to the effective observation time; Indicates whether the corresponding mode is valid; if the corresponding mode cannot be determined, Set to 0, and record it using the modality missing mask; the corresponding modality data is valid but the event type is not. When it does not occur, Except Set all parameters except 1 to 0.

[0153] S3.4: Concatenate the response feature vectors of different event types under the same modal behavioral event in a preset order to form response features for each modal behavioral event, namely, eye movement event response features, head event response features, upper body movement event response features, eyelid behavior event response features, and sound response event response features; specifically:

[0154] No. The first task stimulus fragment Modal behavioral event response characteristics :

[0155]

[0156] in, This indicates eye movements, eyelid movements, head and upper body movements, or vocal modalities. to This indicates the event types included in the corresponding modality. For example, the behavioral event response features of the head modality are sequentially concatenated with the response features of character orientation, target orientation, off-screen, and on-screen events.

[0157] In one embodiment, during the direct social segment, the system extracts whether the child looks at a person or face, the latency and duration of looking at a person or face, and combines this with the head orientation, upper body approaching or moving away after the person appears or makes a sound, as well as the child's vocal response, to describe the child's orientation and sustained participation in direct social stimuli.

[0158] During the gesture and joint attention segment, the system extracts whether the child turns from the person or face toward the cue and the target, whether they reach the target in the direction of the cue, the time required to reach the target, and whether they look back at the person. At the same time, it retains whether the head is facing the same target and whether there is a sound or physical response.

[0159] In the object-social preference segment, the system extracts the child's initial selection, dwell time allocation, and transfer between social AOIs and object AOIs, and retains head, upper body, and voice responses during the same task as auxiliary information.

[0160] In the non-social control segment, the system uses the same response definitions as the corresponding social task to form comparable orientation, dwell, and transfer characteristics between social and non-social stimuli, which are used to distinguish children’s responses to social information from their responses to changes in motion, brightness, or sound.

[0161] Each task retains a summary of eyelid behaviors such as short-term natural blinking within the effective observation period as auxiliary information in the corresponding task-level multimodal response unit; undeterminable states caused by the invisibility of the eye are not used as blinking behavior input.

[0162] S3.5: Combine the modal behavioral event response features of the same task stimulus segment within the same trial into a task-level multimodal response unit. The task-level multimodal response unit records the task stimulus segment it belongs to, the behavioral event response features of each modality, the effective state of each modality, and the cross-modal response relationship formed by multiple modalities.

[0163] The cross-modal response relationship is used to indicate whether different modalities provide corresponding behavioral evidence for the same stimulus. For example, in a joint attention task, when the screen gaze falls into the target AOI and the head is simultaneously facing the target, a cross-modal joint response relationship is formed where the gaze and head direction are consistent; in a direct social task, looking at a person or face accompanied by head orientation, vocal response, or upper body approach forms a combined response towards the same social stimulus.

[0164] The system determines that a common cross-modal response relationship has occurred only when all the basic modes that make up a common cross-modal response relationship are decidable; when any basic mode is undecidable, the common cross-modal response relationship is recorded as an undecidable state, rather than an inactive state.

[0165] For multiple trials of the same task stimulus segment, the system aggregates the multimodal response units at each task level within the valid trials to form a multimodal response representation for that task stimulus segment. Subsequently, following the task chain sequence specified in step S1, the response representations of the direct social segment, the gesture and joint attention segment, the object-social preference segment, and the non-social control segment are combined to obtain a multi-task multimodal response sequence; wherein, the non-social control segment task is used to illustrate the performance of the same type of orientation, dwell, or transfer response in the absence of social meaning.

[0166] S4: Calculate quality control indicators and determine whether to proceed to the risk reasoning stage; details are as follows:

[0167] The quality control indicators include the calibration anchor point coverage rate. Task completion rate Fixation efficiency AOI Attribution Rate Modal efficiency :

[0168]

[0169]

[0170]

[0171]

[0172] For any modality in head, upper body movement, and audio, the modal efficiency is... for:

[0173]

[0174] Among them, superscript Indicates the number of valid records, superscript Indicates the total number of records; and These represent the actual number of trials completed and the total number of trials required, respectively. Indicates the number of sampled frames within a task segment; This indicates the number of valid gaze points that can be classified into the task's AOI.

[0175] When the anchor point coverage is calibrated If the preset conditions are not met, the system will trigger recalibration. If the calibration conditions are still not met after multiple calibrations, the system will output that the current data acquisition is unreadable.

[0176] When the anchor point coverage is calibrated The preset conditions are met, and the task completion rate is... Or gaze efficiency If the preset conditions are not met, the output needs to be retested;

[0177] When the task completion rate And gaze efficiency Meets preset conditions, and modal effectiveness in the head, upper body, or audio. If the conditions are not met, the corresponding missing modality is recorded. When it is supported to use other valid records under the condition that the modality is missing, the system uses the modality missing mask to exclude unusable modalities and allows inference to continue. Otherwise, the output needs to be retested.

[0178] S5: When the quality control indicator result is "passed", based on the task-level multimodal response unit and the effective state of each modality, an autism risk assessment result is generated through a multimodal hierarchical temporal risk inference model. Specifically, this includes:

[0179] The multimodal hierarchical temporal risk reasoning model is a hierarchical temporal model, including a modality encoding layer, a task-level fusion layer, a task sequence encoding layer, and a risk output layer;

[0180] Let the first In the first task-level multimodal response unit The modal behavioral event response characteristics are: ,in Representing eye movements, eyelid movements, head movements, upper body movements, or vocal modalities; constructing a modality missing mask based on the valid states of each modality, where the corresponding element in the modality missing mask is... The modality is set to 1 when it is available and 0 when it is not available.

[0181] The modality coding layer encodes the response features of each modality behavioral event respectively:

[0182]

[0183] in, Indicates the first Feature encoding network corresponding to each modality Indicates the first Missing modality features used when a modality is unavailable ;

[0184] The task-level fusion layer concatenates the modality coding features and modality missing masks from the same task-level multimodal response unit:

[0185]

[0186] in, Indicates a fully connected converged network. Indicates the first The fusion characteristics of multiple task-level multimodal response units;

[0187] The task sequence coding layer employs a bidirectional gated recurrent unit (BiGRU) network to process task-level fusion features according to the task chain and trial order.

[0188]

[0189] in, Indicates the number of task-level multimodal response units. Indicates the first [number] after task sequence encoding Each task representation is used; the system averages and summarizes the representations from each task to obtain the subject-level representation. :

[0190]

[0191] The risk output layer represents the subject level. The data is concatenated with the age-based encoding and then output using the Sigmoid function to determine the probability of ASD-related risk. :

[0192]

[0193] in, Represents the month-age encoding function, For the Sigmoid function, , These are the weight vector and the bias, respectively. The age of the children being tested is in months.

[0194] When the risk probability If the result is greater than or equal to the preset screening threshold, a high-risk screening result will be output; otherwise, a low-risk screening result will be output.

[0195] In one embodiment, during the model training phase, training samples with ASD-related risk labels are constructed following the same processing flow as steps S1 to S3, and the model is trained using binary cross-entropy loss:

[0196]

[0197] in, Indicates the number of training samples. Indicates the first Risk labels for each training sample This represents the risk probability output by the model.

[0198] In one specific embodiment, each modality coding layer uses a two-layer fully connected network, the task-level fusion layer uses a fully connected network, and the task sequence coding layer uses a BiGRU with a hidden dimension of 64; the screening threshold is predetermined based on an independent validation set.

[0199] Finally, it should be noted that the above embodiments are intended to illustrate the technical solutions of the present invention and do not constitute any limitation on the present invention. Those skilled in the art should fully understand that modifications to the technical solutions described in the foregoing embodiments or equivalent substitutions for any part or all of the technical features are entirely feasible. Such modifications or substitutions, as long as they do not depart from the scope of protection defined by the claims of the present invention, should be considered reasonable extensions of the present invention.

Claims

1. A method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior, characterized in that, include: S1: Collect image streams and audio signals of the face, periorbital area, head, and upper body of the children during the process of viewing the pre-set task stimulus sequence; Key facial and peri-eye points, head posture, blinking state, and upper body key points are extracted frame by frame from the image stream. Acoustic summaries are extracted from the audio signals and constructed frame by frame as original multimodal samples. The task stimulus sequence includes various task stimulus segments of the calibration segment and the formal task segment executed sequentially. S2: Construct a calibration support set based on the original multimodal samples of the calibration segment, solve the final session projection matrix, and calculate the screen gaze point of the subject children in the formal task segment based on the final session projection matrix. S3: Using the task stimulus events in the preset task stimulus sequence as time anchors, align the screen gaze point, head posture, blinking state, upper body movement and acoustic summary to construct a task-level multimodal response unit for each task stimulus segment, and distinguish between the behavior-not-occurred state and the modality-undeterminable state in the task-level multimodal response unit. S4: Calculate quality control indicators and determine whether to enter the risk reasoning stage; S5: When the quality control indicator result is passed, based on the task-level multimodal response unit and the effective state of each modality, an autism risk assessment result is generated through a multimodal hierarchical temporal risk reasoning model.

2. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 1, characterized in that, In step S2, the screen gaze point of the children in the formal task segment is calculated, including: S2.1: Extract the original multimodal samples of the calibration segment based on the task identifier of the task stimulus segment to which each original multimodal sample belongs, and construct the calibration support set based on the original multimodal samples of the calibration segment; S2.2: Solve the initial session projection matrix based on the calibration support set, and iteratively refine the initial session projection matrix to obtain the final session projection matrix; S2.3: For any valid facial and periorbital key point sample within the formal task segment, perform mapping calculations using the final session projection matrix to obtain the corresponding screen gaze point.

3. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 2, characterized in that, In step S2.1, constructing the calibration support set based on the original multimodal samples of the calibration segment includes: Identify the time window of each calibration anchor point in the calibration segment, that is, the time period presented by each calibration anchor point; The original multimodal samples of the calibration segment are divided into the corresponding anchor time windows based on the timestamp; Within each anchor point time window, invalid frame removal, stable head pose screening, and blink frame filtering are performed to obtain valid samples; The valid samples within each anchor time window are aggregated into a calibration support set, which is represented as follows: in, To support the sample size; Indicates the calibration support set within the first A set of coordinates of key facial and periorbital points for each supporting sample; Indicates the first The anchor point screen reference coordinates corresponding to the anchor point time window of each supporting sample; Indicates the first Confidence weights of each supporting sample; Indicates the first The anchor point number corresponding to the anchor point time window of each supporting sample; Indicates the first The timestamps of the supporting samples.

4. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 3, characterized in that, In step S2.2, the final session projection matrix is ​​obtained based on the following method: Based on the calibrated support set, the initial session projection matrix is ​​solved using regularized least squares with projection prior constraints: in, This is the initial session projection matrix. The projective prior obtained from offline training; The regularization coefficient is used. Represents the square of the Frobenius norm; Indicates the calibration support set within the first Extended features of each supporting sample ; The output of the shared geometric representation network represents the first... Geometric descriptive features of each supporting sample , Represents a shared geometric representation network. Indicates the first A set of normalized key points obtained by performing translation and scale normalization on the coordinates of all facial and periorbital key points within a support sample; This is a local eye geometry vector constructed based on normalized keypoints; Represents the head pose vector; Using the initial session projection matrix as the initial value, perform multiple rounds of recursive least squares refinement on the calibration support set. After each round of refinement, calculate the anchor point residual. When the decrease in the anchor point residual between two adjacent rounds is less than a preset threshold or reaches the preset maximum number of iterations, stop the refinement and use the matrix obtained from the current round update as the final session projection matrix.

5. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 4, characterized in that, The process of performing multi-round recursive least squares refinement on the calibration support set is as follows: Let the first Round The inverse covariance matrix of each supporting sample before update is: Then the recursive update formula is: in, For the first Round absorption Gain vector for each support sample; and The first Round Each session projection matrix supports both the pre- and post-update samples; For the first Round The two-dimensional reprojection residuals of each supporting sample, i.e., the real screen reference coordinates. Compared with current forecasts The difference between them; and The first Round A support sample inverse covariance matrix before and after the update.

6. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 5, characterized in that, In step S3, the construction of task-level multimodal response units for each task stimulus fragment includes: During the response period after each task stimulus segment is played, eye-tracking events, head events, upper body movement events, eyelid behavior events, and sound response events are constructed based on the screen gaze point, head posture, upper body key points, blinking state, and acoustic summary, respectively, to obtain a multimodal behavioral event sequence composed of the eye-tracking events, head events, upper body movement events, eyelid behavior events, and sound response events; each modal behavioral event records at least the event type, the start and end times of the event type, the task stimulus segment and trial number to which it belongs, and the valid state; Based on the multimodal behavioral event sequence of any task stimulus segment, construct the response feature vector of the corresponding event type according to the screening purpose of the task stimulus segment, including whether the corresponding event type has occurred, the time of first occurrence, the end time, the number of occurrences, and the effective status; The response feature vectors of different event types under the same modal behavioral event are concatenated in a preset order to form response features of each modal behavioral event, namely eye movement event response features, head event response features, upper body movement event response features, eyelid behavior event response features, and sound response event response features; The modal behavioral event response features of the same task stimulus fragment within the same trial are combined into a task-level multimodal response unit.

7. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 6, characterized in that, The task-level multimodal response unit records the task stimulus fragments, behavioral event response characteristics of each modality, effective states of each modality, and cross-modal response relationships formed by multiple modalities. When all the basic modalities constituting the cross-modal response relationship are determinable, the cross-modal response relationship is determined to have occurred. When any basic modality is not determinable, the cross-modal response relationship is recorded as an undeterminable state to distinguish it from the state where no behavior has occurred.

8. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 7, characterized in that, The quality control indicators include: calibration anchor point coverage, task completion rate, gaze efficiency, AOI attribution rate, and modal efficiency. If the coverage of the calibration anchor points does not meet the preset conditions, the system will trigger recalibration. If the calibration pass conditions are still not met after multiple calibrations, the system will output that the current data collection is unreadable. When the calibration anchor coverage meets the preset conditions, but the task completion rate or gaze effectiveness does not meet the preset conditions, the output needs to be retested. When the task completion rate and gaze effectiveness meet the preset conditions, but the modality effectiveness in the head, upper body, or audio does not meet the conditions, the corresponding modality is recorded as missing. Inference is only allowed to continue if the remaining valid records are supported in the state of modality missing; otherwise, the output needs to be retested.

9. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 8, characterized in that, The multimodal hierarchical temporal risk reasoning model includes a modality encoding layer, a task-level fusion layer, a task sequence encoding layer, and a risk output layer; Let the first In the first task-level multimodal response unit The modal behavioral event response characteristics are: ,in Representing eye movements, eyelid movements, head movements, upper body movements, or vocal modalities; constructing a modality missing mask based on the valid states of each modality, where the corresponding element in the modality missing mask is... The modality is set to 1 when it is available and 0 when it is not available. The modality coding layer encodes the response features of each modality behavioral event respectively: in, Indicates the first Feature encoding network corresponding to each modality Indicates the first Missing modality features used when a modality is unavailable ; The task-level fusion layer concatenates the modality coding features and modality missing masks from the same task-level multimodal response unit: in, Indicates a fully connected converged network. Indicates the first The fusion characteristics of multiple task-level multimodal response units; The task sequence coding layer employs a bidirectional gated recurrent unit (BiGRU) network to process task-level fusion features according to the task chain and trial order. in, Indicates the number of task-level multimodal response units. Indicates the first [number] after task sequence encoding Each task representation is used; the system averages and summarizes the representations from each task to obtain the subject-level representation. : The risk output layer represents the subject level. The data is concatenated with the age-based encoding and then output using the Sigmoid function to determine the probability of ASD-related risk. : in, Represents the month-age encoding function, For the Sigmoid function, , These are the weight vector and the bias, respectively. The age of the children being tested is in months. When the risk probability If the result is greater than or equal to the preset screening threshold, a high-risk screening result will be output; otherwise, a low-risk screening result will be output.

10. The method for screening autism in young children based on browser camera eye-tracking data and multimodal behavior according to claim 1, characterized in that, The preset task stimulus sequence includes the following task stimulus segments executed sequentially: A multi-point attraction calibration segment is used to present attention-grabbing stimuli at multiple known anchor points on the screen to acquire the calibration support set; The calibration quality verification section is used to verify the usability of the final session-level projection matrix; The direct social segment is used to present stimuli that include human faces or social interactions in order to quantify the spontaneous orientation of the children in the study to social stimuli. Gestures and joint attention segments were used to present stimuli containing cueing cues and targets to quantify the joint attention abilities of the children in the study. The object-social preference section is used to present social and non-social elements simultaneously or alternately to quantify social preferences. The non-social control segment is used to present control stimuli without people or social interaction in order to differentiate the children's responses to physical stimuli.

Images