An autism early screening method and system based on fNIRS and action feature double modal fusion

Abstract

This invention discloses a method and system for early autism screening based on fNIRS and action feature bimodal fusion, comprising a neural signal acquisition module, a behavioral video acquisition module, a brain network analysis unit, an action analysis unit, and a multimodal fusion decision unit. The brain network analysis unit constructs a dynamic brain functional connectivity map and introduces a graph attention network to automatically learn the importance of key brain regions and their connections. The action analysis unit uses the AlphaPose algorithm to extract key point sequences of the human body and combines bidirectional long short-term memory networks to model action temporal dependencies. The multimodal fusion decision unit employs an attention-based decision-level fusion mechanism, adaptively weighting the classification probabilities of the two branches. This invention achieves multidimensional information complementarity and intelligent fusion for autism by fusing brain signals and behavioral action features, exhibiting strong screening objectivity, high interpretability, and excellent classification accuracy, providing an efficient and automated solution for early autism screening in primary healthcare settings.

Description

Technical Field

[0001] This invention relates to the field of medical auxiliary diagnostic technology, specifically to an early screening method and system for autism based on the fusion of fNIRS and motor features. Background Technology

[0002] Autism spectrum disorder (ASD) is a neurodevelopmental disorder, and early identification and intervention are crucial for improving the prognosis of affected children. Currently, existing technologies have the following limitations: 1. Existing functional near-infrared spectroscopy (fNIRS) research largely focuses on using statistical features and traditional machine learning or simple deep learning models for classification. This fails to fully explore the complex spatial topological relationships within the brain network and is sensitive to noise and individual differences. 2. Current motor behavior analysis primarily relies on scales or manual observation, resulting in high subjectivity and low standardization. Video-based automated analysis often uses single motor features or simple temporal models, limiting its ability to capture subtle and atypical motor patterns in children with ASD. 3. Existing research rarely integrates neural and behavioral data. Multimodal fusion often employs early or late fusion, failing to dynamically assess the contribution of different modalities in specific samples, thus limiting model performance improvement.

[0003] Therefore, developing a screening system that can integrate objective neurophysiological signals, accurately quantify behavioral characteristics, and intelligently integrate bimodal information is of great significance for achieving efficient and objective early auxiliary screening for ASD. Summary of the Invention

[0004] The purpose of this invention is to provide an early screening method and system for autism based on the fusion of fNIRS and motor features, in order to solve the problems of strong subjectivity in assessment, incomplete information in a single modality, and lack of dynamic weighting mechanism in multimodal fusion in the prior art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: A method and system for early autism screening based on fNIRS and action feature bimodal fusion includes: a neural signal acquisition module for acquiring near-infrared spectral signals of brain function of subjects under standardized social interaction tasks; a behavioral video acquisition module for simultaneously recording behavioral videos of subjects under the same task; a brain network analysis unit connected to the neural signal acquisition module for constructing a dynamic brain functional connectivity map based on the near-infrared spectral signals of brain function, analyzing brain network features through a graph attention network, and outputting neural modality classification probabilities; an action analysis unit connected to the action video acquisition module for extracting human key point sequences from the action videos, modeling action temporal dependencies through a bidirectional long short-term memory network, and outputting behavioral modality classification probabilities; and a multimodal fusion decision unit connected to both the brain network analysis unit and the action analysis unit for dynamically weighting and fusing the neural modality classification probabilities and the behavioral modality classification probabilities based on an attention mechanism, and outputting a final classification decision.

[0006] Preferably, the brain network analysis unit includes: a functional connectivity graph construction module, used to segment the preprocessed brain functional near-infrared spectral signals according to time windows, calculate the wavelet coherence coefficient between any two channel signals within each time window, and construct a functional connectivity graph, wherein nodes are fNIRS channels and edge weights are the connection strength between channels; and a graph attention network module, used to receive the functional connectivity graph, assign attention weights to the connection between each node and its neighboring nodes through a self-attention mechanism, and output the neural modality classification probability after multi-layer graph attention network aggregation and graph pooling.

[0007] Preferably, in the graph attention network module, each layer of the graph attention network calculates the attention coefficient between each node in the graph and all its neighboring nodes through a self-attention mechanism, and dynamically assigns different importance weights to different neighbor connections in order to focus on abnormal functional connection patterns related to autism.

[0008] Preferably, the action analysis unit includes: a pose estimation module, used to input the behavior video frame by frame into the AlphaPose pose estimation model, detect and output the two-dimensional coordinate sequence of human key points frame by frame; a temporal modeling module, used to input the key point coordinate sequence into a bidirectional long short-term memory network, learn the contextual dependency of the action from both forward and backward directions, and output the behavior modality classification probability after passing through a fully connected layer and Softmax. Preferably, the multimodal fusion decision unit includes: an attention weight generation module, used to concatenate the graph global feature vector extracted by the brain network analysis unit with the last hidden state of the bidirectional long short-term memory network extracted by the action analysis unit, input it into the attention network, and generate neural modality weight α and behavior modality weight 1-α; a weighted fusion module, used to perform fusion according to the formula:

[0009] Calculate the classification probability after fusion, and output the final classification decision based on the classification probability after fusion.

[0010] Preferably, the neural signal acquisition module is a portable near-infrared spectroscopy imaging device, which focuses on acquiring blood oxygen dynamics signals of the social brain network in the prefrontal cortex and temporoparietal junction; the behavioral video acquisition module includes at least one frontal camera and one side camera for synchronously recording the subject's behavioral videos from multiple angles.

[0011] In the above technical solution, the present invention provides an early autism screening method and system based on fNIRS and motor feature bimodal fusion, which has the following beneficial effects: 1. This invention achieves multi-dimensional information complementarity for autism screening by integrating fNIRS and video dual-modal data, which can provide a more complete biological behavioral profile and effectively overcome the problem of incomplete information from a single modality.

[0012] 2. This invention uses graph attention network to model the spatial topology of brain networks and automatically focuses on abnormal functional connection patterns related to autism through self-attention mechanism, thereby improving the accuracy and interpretability of neural feature extraction.

[0013] 3. This invention combines AlphaPose pose estimation with a bidirectional long short-term memory network to achieve automatic quantitative analysis of children's fine motor patterns, effectively capturing typical autistic motor characteristics such as stereotyped behaviors and abnormal social gestures.

[0014] 4. This invention innovatively introduces a decision-level fusion method based on attention mechanism, which can dynamically adjust the fusion weights of neural modalities and behavioral modalities according to specific samples, adaptively evaluate the credibility of different modalities, and improve classification accuracy and model robustness.

[0015] 5. This invention achieves fully automated analysis, effectively reducing the subjective bias of manual assessment, and provides an efficient technical tool for standardized and scalable early screening of autism in primary healthcare settings. Attached Figure Description

[0016] Figure 1 This is a complete system flowchart. Detailed Implementation

[0017] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0018] like Figure 1 As shown, the present invention provides an early screening method and system for autism based on fNIRS and action feature dual-modal fusion, including a neural signal acquisition module, a behavioral video acquisition module, a brain network analysis unit, an action analysis unit, and a multimodal fusion decision unit.

[0019] Example 1: Data Acquisition and Preprocessing Stage In practical applications, children are placed in a quiet, well-lit testing room to perform standardized social interaction tasks. These standardized social interaction tasks include two paradigms: joint attention tasks and free play tasks.

[0020] Taking the joint attention task as an example: the tester sits opposite the child and guides the child to focus on a specific target object through pointing and gazing actions. The entire process lasts about 3-5 minutes. During this process, the neural signal acquisition module and the behavioral video acquisition module work synchronously.

[0021] The neural signal acquisition module employs a portable near-infrared spectroscopy imaging device, specifically the NIRSport2 or a similar model, with a sampling frequency of 11 Hz. The device is equipped with 12 light source probes and 8 detection probes, forming 20 effective channels, focusing on key areas of the social brain network such as the prefrontal cortex (Brodmann areas 10 and 46) and the temporoparietal junction (Brodmann areas 39 and 40). The acquired blood oxygen dynamics signals include changes in oxyhemoglobin concentration (HbO) and deoxyhemoglobin concentration (HbR), with HbO signals being the primary focus of analysis due to their high signal-to-noise ratio.

[0022] The acquired raw fNIRS signals underwent the following preprocessing steps: First, the raw light intensity signal was converted into light density change data. Then, an abnormal abrupt change point in the signal was identified using a motion artifact detection algorithm, and the identified artifacts were corrected using cubic spline interpolation. Next, a bandpass filter of 0.01-0.1 Hz was applied to the corrected data to eliminate physiological noise such as respiration (approximately 0.2-0.3 Hz), heart rate (approximately 1-1.5 Hz), and low-frequency signal drift. After filtering, according to the modified Beer-Lambert law, the light density data was converted into a time series of oxyhemoglobin (HbO) and deoxyhemoglobin (HbR) concentration changes.

[0023] Meanwhile, the behavioral video acquisition module uses two high-definition cameras to simultaneously record the behavior of the children. One camera is positioned approximately 1.5 meters in front of the child, and the other is positioned approximately 2 meters to the side. This dual-angle recording ensures complete capture of the child's body movements, facial expressions, and gaze direction. The video acquisition frame rate is 30 frames per second, with a resolution of 1920×1080 pixels.

[0024] Example 2: Neural Modal Analysis Phase The brain network analysis unit receives the preprocessed fNIRS signal and contains a functional connectivity graph construction module and a graph attention network module.

[0025] 1. Functional connection diagram construction The functional connectivity graph construction module uses a sliding time window method to segment the continuous signal. Specifically, the time window length is set to 10 seconds (containing 100 sampling points), and the sliding step size is 2 seconds (20 sampling points), dividing the entire task duration (e.g., 240 seconds) into approximately 115 time windows. For each time window, the wavelet coherence coefficients between each pair of the 20 fNIRS channels are calculated as the functional connectivity strength.

[0026] The wavelet coherence coefficient value is between 0 and 1. The closer the value is to 1, the higher the phase synchronization of the channel pair in a specific frequency band.

[0027] Therefore, a 20×20 symmetric functional connectivity matrix is ​​generated for each time window, and the elements in the matrix are the edge weights. Combining the spatial location information of the 20 channels, a functional connectivity graph sequence G = {G1, G2, ..., G...} is constructed. T}, where T is the number of time windows (T=115 in this example). Each node in the graph is an fNIRS channel, and the initial feature of the node is the average HbO signal intensity of that channel within that time window.

[0028] 2. Graph Attention Network Classification The functional connection graph sequence G is input into the graph attention network module. The graph attention network module adopts a three-layer graph attention network (GAT) structure, and the specific parameter settings are as follows: The first layer of GAT has an input feature dimension of 32 (number of nodes) × 1 (initial features of each node) and an output feature dimension of 64. It uses 4 attention heads, each of which independently calculates its attention coefficients before concatenating the output. The activation function used is ELU.

[0029] The second layer of GAT has an input feature dimension of 32×64 and an output feature dimension of 128. It uses 4 attention heads and adopts an average aggregation method.

[0030] The third layer GAT has an input feature dimension of 32×128 and an output feature dimension of 256. It uses one attention head for the final node feature aggregation.

[0031] In each layer of the graph attention network, the attention coefficient e ij The calculation formula is:

[0032] Among them, h i and h j Let be the feature vectors of nodes i and j, respectively; W be the learnable weight matrix; a be the learnable parameter vector of the attention mechanism; and || denotes the vector concatenation operation. The attention coefficients are normalized using the Softmax function to obtain α. ij That is, the importance weight of node j to node i.

[0033] After three layers of GAT processing, each node aggregates information from its neighboring nodes, resulting in a dynamically weighted feature representation. Subsequently, Global Average Pooling is used to aggregate the features of the 32 nodes into a global graph feature vector with a dimension of 256.

[0034] The global feature vector is sequentially fed into a fully connected layer (256→128) and a Softmax classification layer, outputting the preliminary classification probability P of the neural modality. neuro The classification probability includes two categories: "High risk of ASD" (denoted as P). neuro_ASD ) and “typical development” (denoted as P) neuro_TD The sum of the two is 1.

[0035] Example 3: Behavioral Modality Analysis Phase The motion analysis unit receives video data recorded by the behavior video acquisition module and includes a pose estimation module and a temporal modeling module.

[0036] 1. Pose estimation and feature extraction The pose estimation module processes the synchronously recorded dual-angle video frame by frame. Taking the front-facing camera video as an example, the video is first extracted frame by frame, resulting in approximately 7200 frames (240 seconds × 30 frames / second). Each frame is then input into the pre-trained AlphaPose pose estimation model.

[0037] The AlphaPose model uses YOLOv3 as the human detector, and the region multi-pose estimator employs a symmetric spatial transformation network, enabling high-precision human keypoint detection. The model outputs the two-dimensional coordinates of 17 human keypoints, including: 0-nose, 1-left shoulder, 2-right shoulder, 3-left elbow, 4-right elbow, 5-left wrist, 6-right wrist, 7-left hip, 8-right hip, 9-left knee, 10-right knee, 11-left ankle, 12-right ankle, 13-left eye, 14-right eye, 15-left ear, and 16-right ear.

[0038] For each frame of the image, the (x, y) coordinates of all 17 key points are extracted to form an original feature vector with a dimension of 34 (17 points × 2 coordinates). To ensure the robustness of the features to individual body shape and position, the coordinates are normalized: the origin is taken as the center point of the line connecting the left and right shoulders, and the scale is normalized based on the shoulder width.

[0039] After the above processing, the entire video segment is converted into a feature sequence X = {x1, x2, ..., x...} N}, where N is the total number of frames (N=7200 in this example), and each x t This is a normalized keypoint coordinate vector with dimension 34.

[0040] 2. Time Series Modeling The feature sequence X is input into the temporal modeling module. The temporal modeling module uses a bidirectional long short-term memory network (BiLSTM), and the specific network structure is as follows: Input layer: Accepts feature sequences with a dimension of 34 and a sequence length of 7200.

[0041] BiLSTM layer: The number of hidden units is set to 128, including two directions: forward LSTM and backward LSTM. The forward LSTM processes the sequence from t=1 to t=N, capturing past dependencies of actions; the backward LSTM processes the sequence from t=N to t=1, capturing future dependencies of actions. The outputs of the two directions are concatenated to obtain a 256-dimensional hidden state sequence.

[0042] Dropout layer: A dropout layer with a dropout rate of 0.5 is added after the BiLSTM layer to prevent overfitting.

[0043] Fully connected layer: The hidden state of the last time step of BiLSTM (the concatenated 256-dimensional vector) is connected to the fully connected layer and mapped to the 2-dimensional output space.

[0044] Softmax layer: Outputs the initial classification probability P of the behavioral modality. action This includes "high-risk ASD" (P action_ASD ) and "typical development" (P action_TD ).

[0045] The core advantage of BiLSTM lies in its ability to capture the temporal integrity of actions. Taking the social action of "reaching out, pointing, and looking back" as an example, the action sequence of a typical child with developmental delays presents a complete temporal pattern, while children with ASD may exhibit interrupted actions or disordered sequences. BiLSTM can effectively identify such anomalies through bidirectional contextual modeling.

[0046] Example 4: Multimodal Fusion Decision Stage The multimodal fusion decision unit is connected to the brain network analysis unit and the action analysis unit, respectively, and contains an attention weight generation module and a weighted fusion module.

[0047] 1. Attention Weight Generation The attention weight generation module first extracts the high-level feature vectors of the two branches: Neural modality characteristics f neuro : The graph global feature vector output from the graph attention network module in the brain network analysis unit, with a dimension of 256.

[0048] Behavioral modal features f action : The last hidden state of BiLSTM output from the temporal modeling module in the action analysis unit, with a dimension of 256.

[0049] Concatenate the two feature vectors to obtain a joint feature vector F = [f] with dimension 512. neuro , f action ].

[0050] The joint feature vector is input into a small attention network. The structure of the attention network is as follows: The first layer is fully connected: 512 → 128, with ReLU as the activation function.

[0051] The second fully connected layer is 128 → 2, with the activation function being Softmax.

[0052] The attention network outputs two weight values: α (neural modality weight) and 1-α (behavioral modality weight), satisfying α + (1-α) = 1, and α ∈ [0, 1]. These two weight values ​​are dynamically generated, calculated based on the joint feature vector of the current test sample, reflecting the relative reliability of neural and behavioral information for the final diagnosis under the current sample.

[0053] 2. Weighted fusion and final decision The weighted fusion module 502 receives the neural modality classification probability P. neuro Behavioral modality classification probability P actionThe dynamic weights α generated by the attention network are used to calculate the fused classification probability according to the following formula:

[0054] Specifically, the probabilities for the two categories are calculated separately:

[0055] The final classification decision is based on P final_ASD The value is determined by comparing it with a preset threshold. In this embodiment, the threshold is set to 0.5, that is: If P final_ASD If the value is ≥ 0.5, then output "High risk of ASD"; If P final_ASD If the value is less than 0.5, the output will be "typical development".

[0056] Example 5: System Workflow Example To more clearly illustrate the workflow of this invention, a specific test case will be used as an example below.

[0057] A 3-year-old boy underwent screening using this system. The testing process is as follows: Data Acquisition Phase: Children performed a joint attention task for 3 minutes under the guidance of testers. The fNIRS device acquired HbO signals from 20 channels in the prefrontal and temporoparietal junction areas at a sampling rate of 11 Hz; a dual-angle camera simultaneously recorded their behavioral video at 30 frames per second.

[0058] Preprocessing stage: The fNIRS signal is denoised, filtered and converted to obtain a clean time sequence signal; the video is divided into 5400 frames.

[0059] Neuromodal analysis: The functional connectivity graph construction module divides the fNIRS signal into 90 time windows (10-second window length, 2-second step size), calculates the PLV values ​​between 20 channels within each time window, and constructs 90 functional connectivity graphs. The graph attention network module processes these graph sequences and automatically identifies, through a self-attention mechanism, that the functional connectivity strength between the left prefrontal cortex and the right temporoparietal junction is significantly lower than the normal reference range, which is one of the typical characteristics of children with ASD. The final output is the neuromodal probability P. neuro_ASD = 0.72, P neuro_TD = 0.28.

[0060] Behavioral modality analysis: The pose estimation module performs AlphaPose keypoint detection frame-by-frame on 5400 frames of images, extracting the coordinate sequences of 17 keypoints. The temporal modeling module analyzes the action sequences using BiLSTM, detecting a lack of eye following and discontinuous gesture trajectories in children when making pointing movements, exhibiting typical abnormal action patterns. The final output is the behavioral modality probability P. action_ASD = 0.68, P action_TD = 0.32.

[0061] Multimodal fusion decision: The attention weight generation module concatenates neural modality features (256 dimensions) and behavioral modality features (256 dimensions) and inputs them into the attention network, calculating α = 0.55 and 1-α = 0.45. The weighted fusion module calculates: P final_ASD = 0.55 × 0.72 + 0.45 × 0.68 = 0.702 P final_TD = 0.55 × 0.28 + 0.45 × 0.32 = 0.298 Because of P final_ASD = 0.702>0.5, the system outputs the final classification decision as "ASD high risk".

[0062] In this case, the neural modality weight α is slightly higher than the behavioral modality weight, indicating that the brain network features of this sample are more typical than the action features. The attention mechanism automatically assigns higher credibility, demonstrating the advantages of adaptive weighted fusion.

[0063] As can be seen from the above description of the specific embodiments, the present invention has the following technical advantages: 1. By processing neural signals and behavioral videos in parallel, complementary fusion of dual-modal information is achieved, providing a more comprehensive diagnostic basis; 2. The introduction of graph attention networks enables brain network analysis to focus on abnormal functional connectivity patterns related to ASD, which has strong interpretability; 3. The combination of AlphaPose and BiLSTM enables precise capture of subtle motion patterns, solving the problem of strong subjectivity in traditional manual observation; 4. The attention-based dynamic weighted fusion method can adaptively adjust modality weights according to sample features, thereby improving the accuracy and robustness of classification; 5. Full-process automated analysis reduces human intervention and provides technical support for standardized screening in primary healthcare institutions.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method and system for early autism screening based on fNIRS and motor feature dual-modal fusion, characterized in that, include: The neural signal acquisition module is used to acquire near-infrared spectral signals of brain function in subjects under standardized social interaction tasks; The behavioral video acquisition module is used to synchronously record videos of subjects' behaviors under the same task; The brain network analysis unit is connected to the neural signal acquisition module and is used to construct a dynamic brain function connectivity map based on the brain functional near-infrared spectral signal, and to analyze brain network features through graph attention network analysis and output neural modality classification probability. The action analysis unit, connected to the behavior video acquisition module, is used to extract human key point sequences from the behavior video, and to model action temporal dependencies through a bidirectional long short-term memory network, and output the behavior modality classification probability. The multimodal fusion decision unit is connected to the brain network analysis unit and the action analysis unit, respectively, and is used to dynamically weight and fuse the classification probabilities of the neural modality and the classification probabilities of the behavioral modality based on the attention mechanism, and output the final classification decision.

2. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 1, characterized in that, The brain network analysis unit includes: The functional connectivity graph construction module is used to divide the preprocessed brain functional near-infrared spectral signals into time windows, calculate the wavelet coherence coefficient between any two channel signals in each time window, and construct a functional connectivity graph, where nodes are fNIRS channels and edge weights are the connection strength between channels. The graph attention network module is used to receive the functional connection graph, assign attention weights to the connection between each node and its neighboring nodes through a self-attention mechanism, and output the neural modality classification probability after aggregation by a multi-layer graph attention network and graph pooling.

3. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 2, characterized in that, In the graph attention network module, each layer of the graph attention network calculates the attention coefficient between each node in the graph and all its neighboring nodes through a self-attention mechanism, and dynamically assigns different importance weights to different neighbor connections in order to focus on abnormal functional connection patterns related to autism.

4. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 1, characterized in that, The motion analysis unit includes: The pose estimation module is used to input the behavior video frame by frame into the AlphaPose pose estimation model, detect and output the two-dimensional coordinate sequence of human key points frame by frame; The temporal modeling module is used to input the key point coordinate sequence into a bidirectional long short-term memory network, learn the contextual dependencies of actions in both forward and backward directions, and output the behavior modality classification probability after passing through a fully connected layer and Softmax.

5. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 4, characterized in that: The key points of the human body include a total of 17 points: nose, shoulder, elbow, wrist, hip, knee, and ankle.

6. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 1, characterized in that, The multimodal fusion decision unit includes: The attention weight generation module is used to concatenate the graph global feature vector extracted by the brain network analysis unit with the last hidden state of the bidirectional long short-term memory network extracted by the action analysis unit, input it into the attention network, and generate neural modality weights and behavioral modality weights. The weighted fusion module is used to calculate the fused classification probability according to the formula, and output the final classification decision based on the fused classification probability.

7. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 6, characterized in that, The sum of the neural modality weights and the behavioral modality weights is 1, and the weight values ​​are dynamically generated based on the current test samples, reflecting the relative reliability of neural information and behavioral information for the final diagnosis.

8. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 1, characterized in that, The neural signal acquisition module is a portable near-infrared spectroscopy imaging device, which focuses on acquiring blood oxygen dynamics signals of the social brain network in the prefrontal cortex and temporoparietal junction.

9. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 1, characterized in that, The behavioral video acquisition module includes at least one front camera and one side camera for synchronously recording the subject's behavioral videos from multiple angles.

10. The method and system for early autism screening based on fNIRS and motor feature bimodal fusion as described in claim 1, characterized in that, The final classification decision includes two outputs: "high risk of autism spectrum disorder" and "typical development".

Images