Early risk prediction method for children with autism based on postures and facial expressions

Abstract

The invention discloses an early risk prediction method for autistic children based on postures and facial expressions, and aims to construct a deep neural network framework fusing a facial expression image sequence and body posture motion information for the pain point that existing ASD early recognition depends on a subjective doctor observation scale and lacks objective quantitative indexes. A specific convolutional neural network (CNN) branch is designed to extract facial expression features, a time sequence modeling network (such as Transform) is utilized to analyze attitude sequence features, an attention mechanism is combined to fuse multi-modal information, and finally automatic and objective recognition of natural flowing fine abnormal expressions and engraving attitudes in social interaction of young children is realized. And an effective intelligent tool is provided for early screening and auxiliary diagnosis of ASD.

Description

Technical Field

[0001] This invention belongs to the technical field of medical artificial intelligence, specifically relating to an early risk prediction method for children with autism based on posture and facial expressions. Background Technology

[0002] Current early screening for autism spectrum disorder (ASD) heavily relies on subjective assessments by clinicians using scales such as ADOS. This method has significant limitations: subtle abnormalities exhibited by young children in natural social interactions (such as 0.3-second interruptions in micro-expressions or stereotyped gestures) are difficult to detect and quantify visually, leading to a misdiagnosis rate exceeding 30%. While existing AI-assisted methods attempt to incorporate computer vision technology, they are limited by incomplete data acquisition—a single sensor (such as an RGB camera) cannot simultaneously capture high-precision facial micro-expressions and three-dimensional body posture motion flow, resulting in a lack of multimodal collaborative foundation for subsequent analysis. More critically, traditional models, when processing multimodal data, only use hard feature concatenation or weighted averaging (such as early fusion networks), completely ignoring cross-modal spatiotemporal dependencies (e.g., stiff shoulder movements accompanying a child's fake smile), causing semantic conflicts in the fusion layer due to heterogeneous data (corresponding to the failure scenario of the "attention fusion layer" module in the diagram). This coarse-grained fusion further makes the recognition module sensitive to individual differences (such as differences in facial expressions across different cultural backgrounds), resulting in insufficient reliability of the output results (corresponding to the clinical implementation bottleneck of the "ASD anomaly recognition" module in the figure). Therefore, there is an urgent need for an end-to-end solution that can achieve synchronous and collaborative data acquisition through multimodal sensors, namely RGB cameras for acquiring facial expression video streams and inertial measurement units (IMUs) for acquiring body posture motion sequences, and design an adaptive cross-modal interaction mechanism to ultimately achieve the goal of objectively quantifying ASD features. Summary of the Invention

[0003] The main objective of this invention is to overcome the shortcomings and deficiencies of the prior art and provide a method for early risk prediction of autism in children based on posture and facial expression, which can be used to achieve objective early screening of autism spectrum disorder (ASD) by fusing facial expression and body posture data.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: One aspect of the present invention provides a method for early risk prediction of autism in children based on posture and facial expressions, comprising the following steps: The video stream of facial expressions and the sequence of body posture movements of subjects during natural interactions were simultaneously acquired using multiple source sensors. A spatiotemporal dilated convolutional network is used to extract the temporal features of facial micro-expressions in the facial expression video stream; The spatiotemporal dynamics of body posture at key skeletal points in the body posture motion sequence are encoded using a posture feature extraction network. The temporal features and body posture spatiotemporal dynamic features are deeply fused by a cross-modal dynamic attention fusion engine. The fused features are then output in parallel via a fully connected layer as micro-expression interruption index and stereotyped motion frequency coefficient. By constructing a learnable weighted fusion layer, the micro-expression interruption index and stereotyped movement frequency coefficient are weighted and fused to obtain the comprehensive risk score of ASD. The scoring threshold is set and the subjects are classified into ASD risk levels.

[0005] As a preferred technical solution, the multi-source sensor includes an image sensing unit for acquiring facial expression video streams and a three-dimensional motion sensing unit for acquiring body posture motion sequences; the image sensing unit uses an RGB camera and / or an infrared camera; the three-dimensional motion sensing unit uses a stereo vision system and / or an inertial measurement unit.

[0006] As a preferred technical solution, the spatiotemporal dilated convolutional network is a dual-stream architecture, including several layers of 3D convolutional neural networks, which respectively process the original RGB frame sequence and the pre-computed optical flow frame sequence, and fuse the two features through an early or late strategy. The 3D convolutional neural network is obtained by expanding the 2D convolutional neural network through an inflation mechanism, specifically: increasing the two-dimensional convolutional kernel K in the pre-trained 2D convolutional neural network. 2D ∈R Kh×Kw Extending along the time dimension generates a 3D convolutional kernel K. 3D (t,h,w)∈R Kt×Kh×Kw Where R represents the set of real numbers, K t , K h , K w These represent the dimensions of the convolutional kernel in the time, height, and width directions, respectively. The early fusion strategy specifically involves fusing the original RGB frame sequence with the pre-computed optical flow frame sequence before inputting it into the spatiotemporal dilated convolutional network. The post-fusion strategy is as follows: the original RGB frame sequence and the pre-computed optical flow frame sequence are processed independently by the spatiotemporal dilated convolutional network, and then the semantic features are generated and fused.

[0007] As a preferred technical solution, the three-dimensional convolution kernel K 3D (t,h,w)∈R Kt×Kh×Kw Used for processing the input facial expression video stream V∈R T×H×W×C Performing 3D convolution operations yields the temporal feature tensor F∈R of facial expressions. T′×H′×W′×C′Where C and C' represent the number of input and output channels, respectively; T, H and W are the input dimensions of time, height and width, respectively; and T′, H′ and W′ are the output dimensions of time, height and width, respectively. in: ; When the number of channels C = 1, the single-channel convolution formula is: ; Where X is the input tensor, t, x, y are the starting center coordinates of the current output point Y(t, x, y) in the input data, and t', i, j are the offsets based on these starting coordinates.

[0008] As a preferred technical solution, the posture feature extraction network extracts spatiotemporal node feature sequences (V) from the input body posture motion sequence through a series of stacked ST-Conv Block modules. t-M+1 , ..., V t To conduct joint learning; In each ST-Conv Block module, temporal dependencies are first captured using one-dimensional convolution and gated linear units (GLUs) through temporally gated convolution. Next, spatial graph convolution is applied to aggregate the spatial relationships between nodes based on predefined graph weights W. Finally, features are extracted again through temporally gated convolution units, and V is output. l+1 As input to the next ST-ConvBlock module, it is finally output through the output layer; The time-gated convolution is specifically: ; Among them, V l Indicates the first l The layer's output vector serves as the input to the entire GLU operation, V. temp (l+1) Indicates the first l +1 is a temporary vector, where * denotes a matrix multiplication operation. Represents element-wise product. σ For the sigmoid activation function, τ GLU W represents a gated linear unit operation. out W gate These are learnable 1D convolutional kernel weights.

[0009] As a preferred technical solution, the cross-modal dynamic attention fusion processor calculates the attention weights between different modalities through self-learning, specifically as follows: In cross-modal attention, vision → language: ; In cross-modal attention, audio → language: ; Modality fusion output language modality l j Integrating visual information: ; ; ; Where j, i, k represent the indices of the language, visual, and audio sequences, respectively; m, p, d represent the length of the visual and audio sequences and the dimension of the features, respectively; and I j V represents the j-th language feature vector. i a represents the i-th visual feature vector. i Represents the i-th audio feature vector α i,j V→L Let α be the attention weight, representing the correlation between the j-th linguistic feature and the i-th visual feature. i,j A→L W represents the attention weight between the j-th language feature and the i-th audio feature. Q L W is the query transformation matrix used for language modalities. K V W is the key transformation matrix used for visual modality. K A W is the key transformation matrix used for audio modalities. V V W is the value transformation matrix used for visual modality. V A This is the value transformation matrix used for audio modalities. Let j be the representation of the j-th language feature after being enhanced by visual information. Let j be the representation of the j-th language feature after it has been enhanced with audio information. This represents the final fused representation of the j-th language feature.

[0010] As a preferred technical solution, the ASD risk score is obtained by weighting and fusing the micro-expression interruption index and stereotyped movement frequency coefficient by constructing a learnable weighted fusion layer, specifically as follows: The two coefficients, the micro-expression interruption index and the stereotyped motion frequency coefficient, output in parallel from the fully connected layer will be normalized: ; ; Among them, MEI min MEI max These are the minimum and maximum values ​​of the microexpression interruption index (SMF) obtained statistically from a large-scale training dataset. min SMF maxThey are the minimum and maximum values of the stereotyped movement frequency coefficient statistically obtained from a large-scale training dataset.

[0011] Calculate the ASD comprehensive risk score as follows: ; where ASRS is the calculated ASD comprehensive risk score, with a value range between [0, 1]. The higher the score, the higher the ASD risk; α and β are learnable weight parameters, and α + β = 1.

[0012] As a preferred technical solution, the setting of the scoring threshold and the classification of the ASD risk level for the subject are specifically as follows: Map the continuous ASD comprehensive risk score to a discrete risk level: When ASRS ≤ T1, it is considered that the ASD risk level of the subject is low risk; When T1 < ASRS ≤ T2, it is considered that the ASD risk level of the subject is medium risk; When ASRS > T2, it is considered that the ASD risk level of the subject is high risk; where ASRS is the ASD comprehensive risk score, and T1 and T2 are the scoring thresholds obtained from the clinically annotated training data.

[0013] Another aspect of the present invention also provides an early risk prediction system for autistic children based on posture and facial expression, which applies the above-mentioned early risk prediction method for autistic children based on posture and facial expression, and includes a multi-source sensor, a spatio-temporal dilated convolutional network, a posture feature extraction network, a cross-modal dynamic attention fusion device, a learnable weighted fusion layer, and an ASD risk level classification module; The multi-source sensor is used to synchronously collect the facial expression video stream and the body posture motion sequence in the natural interaction of the subject; The spatio-temporal dilated convolutional network is used to extract the temporal features of facial micro-expressions in the facial expression video stream; The posture feature extraction network is used to encode the spatio-temporal dynamics features of the body posture of the skeletal key points in the body posture motion sequence; The cross-modal dynamic attention fusion device is used to perform deep feature fusion on the temporal features and the spatio-temporal dynamics features of the body posture. The fused features are parallelly output as the micro-expression interruption index and the stereotyped movement frequency coefficient through a fully connected layer; The learnable weighted fusion layer is used to perform weighted fusion on the micro-expression interruption index and the stereotyped movement frequency coefficient to obtain the ASD comprehensive risk score; The ASD risk level classification module is used to set the scoring threshold and classify the ASD risk level of the subject.

[0014] In another aspect, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for early risk prediction of autism in children based on posture and facial expressions.

[0015] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) Objective quantification: Using computer vision technology to automatically capture and quantify subtle behavioral features that are difficult for the human eye to perceive (such as a 0.3-second micro-expression interruption), overcoming the limitations of subjective scale assessment.

[0016] (2) Multimodal collaboration: Through an innovative cross-modal dynamic attention fusion mechanism, the intrinsic relationship between facial expressions and body posture is deeply explored, which improves the specificity and accuracy of recognition.

[0017] (3) End-to-end automation: A complete closed loop is formed from data collection to risk assessment, which reduces the reliance on external scoring and expert experience and is conducive to large-scale universal screening.

[0018] (4) Strong generalization ability: The model is not sensitive to individual differences (such as expression differences caused by cultural background), the output results are more stable and reliable, and it has strong clinical applicability. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating an early risk prediction method for children with autism based on posture and facial expressions, according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the structure of the facial micro-expression recognition network according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of the posture stereotype behavior analysis network according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of the multimodal fusion network according to an embodiment of the present invention. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0021] Example: This embodiment provides an early risk prediction method for children with autism based on posture and facial expressions. It simultaneously acquires video streams of facial expressions and body posture sequences from children's natural interactions using multi-source sensors. A dual-branch deep evaluation network is constructed: a temporal dilated convolutional network (I3D) is used to extract temporal features of facial micro-expressions, while a posture feature extraction network (joint graph spatiotemporal gating network, ST-GCN) is used to encode the spatiotemporal dynamic patterns of skeletal key points. A cross-modal dynamic attention fusion mechanism (based on a Multimodal Transformer architecture) is used to achieve deep fusion of the dual features, adaptively capturing the co-correlation patterns between expressions and postures (such as the coupling phenomenon between abnormal smiles and stereotyped arm swings). The fused features are output in parallel through fully connected layers as micro-expression interruption indices and stereotyped movement frequency coefficients. Finally, a weighted fusion is used to achieve fine-grained ASD risk quantification assessment. The flowchart is shown below. Figure 1 As shown.

[0022] I. Multi-source sensors; The multi-source sensor includes an image sensing unit and a three-dimensional motion sensing unit; The image sensing unit is used to acquire facial expression video streams containing facial regions, and employs an RGB camera and / or an infrared camera. The three-dimensional motion sensing unit is used to collect body posture motion sequences, that is, the motion trajectory data of the body joints of the target object in three-dimensional space; a stereo vision system and / or an inertial measurement unit are used.

[0023] II. Facial Micro-expression Recognition Networks: I3D (e.g.) Figure 2 (As shown).

[0024] The backbone feature extraction network of the spatiotemporal dilated convolutional network in this embodiment is a multi-layer 3D convolutional neural network that can simultaneously process the spatial dimension (single-frame facial region) and temporal dimension (inter-frame muscle dynamic changes) of video frame sequences.

[0025] The network employs a dual-stream architecture: one input is the original RGB frame sequence to capture static appearance features, and the other input is the pre-computed optical flow frame sequence to explicitly model motion dynamics; the two features are integrated through a late or early fusion operation (⊕) to achieve complementary encoding of appearance and motion information, and finally output action classification labels.

[0026] The early fusion strategy specifically involves fusing the two feature streams before inputting them into the backbone feature extraction network of the spatiotemporal dilated convolutional network. The post-fusion strategy is as follows: after the two features are processed independently by the spatiotemporally dilated convolutional network to generate high-level semantic features, they are fused before the fully connected layer.

[0027] The 3D convolutional neural network expands a mature 2D convolutional neural network (i.e., a 2D image classification network, such as Inception) into a 3D convolutional structure through an inflation mechanism.

[0028] The operation of the dilation mechanism is as follows: The two-dimensional convolutional kernel K in the pre-trained 2D convolutional neural network (such as Inception or ResNet) is... 2D ∈R Kh×Kw Extending along the time dimension generates a 3D convolutional kernel K. 3D ∈R Kt×Kh×Kw Where R represents the set of real numbers, K t , K h , K w These represent the dimensions of the convolutional kernel in the time, height, and width directions, respectively, with the time dimension K being K. t The weight initialization adopts a parameter duplication strategy, that is, the original 2D convolutional kernel weights are repeated K times in the time dimension. t This allows each time slice to share the same spatial filtering parameters. This operation processes the input video segment V∈R. T×H×W×C The spatiotemporal feature tensor F∈R is output through three-dimensional convolution operations. T′×H′×W′×C′ Where C and C' represent the number of input and output channels, respectively; T, H, and W are the input dimensions of time, height, and width, respectively; and T′, H′, and W′ are the output dimensions of time, height, and width, respectively.

[0029] , formula (1); 3D convolution can simultaneously process spatial dimensions (H, W) and temporal dimensions (T), capturing continuous changes in muscle movement. When the number of channels C = 1, the formula for single-channel convolution is: , formula (2); Where X is the input tensor, t, x, y are the starting center coordinates of the current output point Y(t, x, y) in the input data, and t', i, j are the offsets based on these starting coordinates.

[0030] III. Posture Feature Extraction Network (Posture Stereotype Behavior Analysis Network): ST-GCN.

[0031] The core processing flow of ST-GCN (Spatiotemporal Graph Convolutional Network) is completed by ST-Conv Block modules stacked in series. Each ST-Conv Block module processes the input spatiotemporal node feature sequence (V) t-M+1 , ..., V tJoint learning is performed as follows: First, temporal gated-conv (C=64) is used to capture sequence dynamics, which effectively models temporal dependencies using 1-D convolutions and gated linear units (GLUs). Next, spatial graph-conv (C=16) is applied to aggregate spatial relationships between nodes based on predefined graph structure weights W. Finally, features are extracted again using temporal gated convolution, and V is output. l+1 This serves as input for the next module. Multiple such blocks progressively extract pose feature sequences, and their final output is fed into the output layer to achieve the prediction target.

[0032] like Figure 3 As shown, the core method of ST-GCN (Spatiotemporal Graph Convolutional Network) is to jointly model spatiotemporal graph sequences through cascaded ST-Conv Block modules: each module alternately performs spatial graph convolution (aggregating the topological relationships between nodes) and temporally gated convolution (using 1D convolution with gated mechanism to capture dynamic temporal dependencies) to achieve hierarchical feature abstraction.

[0033] Among them, V l Indicates the first l The layer's output vector serves as the input to the entire GLU operation, V. temp (l+1) Indicates the first l +1 is a temporary vector, where * denotes a matrix multiplication operation. Represents element-wise product. σ For the sigmoid activation function, τ GLU W represents a gated linear unit operation. out W gate It uses learnable 1D convolutional kernel weights (number of channels C=64). Its core innovation lies in the temporally gated convolutional unit, mathematically represented as: , formula (3).

[0034] IV. Multimodal Fusion Network: MulT.

[0035] MulT (Multimodal Transformer) is an innovative multimodal interaction model, its core being the crossmodal attention mechanism. This differs from traditional predefined word-level alignment methods (such as...). Figure 4Unlike the upper half of the image, MulT abandons the limitation that alignment must be pre-defined and strongly correlated. It allows segments from different modalities, such as visual, linguistic, and audio, to dynamically pay attention to each other, calculating attention weights between modalities through self-learning (as shown by the varying thickness of the dashed lines in the lower half of the image). For example, when understanding the word "spectacle," the model can simultaneously focus on the visual information of "raising an eyebrow" in the video and the emphasized segment in the audio, ignoring irrelevant information (such as the visual part of "uninformative"). This dynamic, end-to-end cross-modal interaction mechanism effectively overcomes the challenge of strict temporal or semantic alignment between modalities, greatly improving the understanding ability and performance of multimodal fusion.

[0036] Let the visual sequence be V = {v1, v2, ..., v...} m The language sequence L = {l1, l2, ..., l} n}, audio sequence A={a1,a2,...,a p}, Figure 4 The dashed arrows (orange / red) correspond to Attention. V→L and Attention A→L The weight distribution (the thicker the line, the greater the weight).

[0037] The core formula is as follows: Cross-modal attention (visual → language): , formula (4); Cross-modal attention (audio → language): , formula (5); Modality fusion output language modality l j Integrating visual information: , formula (6); , formula (7); , formula (8); In the formula v i T W V and l j T W L Modal feature mapping was implemented (implicitly included in the dot product of the original image formula). The final fused output l^j dynamically aggregates cross-modal information (such as "spectacle" simultaneously associating visual / audio segments). Here, j, i, k represent the indices of the language, visual, and audio sequences, respectively; m, p, d represent the length of the visual / audio sequence and the dimension of the feature, respectively; and I... j V represents the j-th language feature vector.i a represents the i-th visual feature vector. i Represents the i-th audio feature vector α i,j V→L Let α be the attention weight, representing the correlation between the j-th linguistic feature and the i-th visual feature. i,j A→L W represents the attention weight between the j-th language feature and the i-th audio feature. Q L W is the query transformation matrix used for language modalities. K V W is the key transformation matrix used for visual modality. K A W is the key transformation matrix used for audio modalities. V V W is the value transformation matrix used for visual modality. V A This is the value transformation matrix used for audio modalities. Let j be the representation of the j-th language feature after being enhanced by visual information. Let j be the representation of the j-th language feature after it has been enhanced with audio information. This represents the final fused representation of the j-th language feature.

[0038] V. Fine-grained ASD risk quantification assessment.

[0039] Fine-grained ASD risk quantification assessment establishes a learnable weighted fusion layer to integrate the two objective quantitative indicators—micro-expression interruption index and stereotyped movement frequency coefficient—into a comprehensive ASD risk score, and then classifies the risk level based on the scoring threshold. The specific steps are as follows: (1) Indicator normalization; First, the MEI and SMF coefficients of the parallel output of the fully connected layer are normalized to ensure they fall within a similar numerical range (e.g., [0, 1]) to eliminate the influence of dimensions and facilitate subsequent weighted fusion. The normalization formula is as follows: , formula (9); , formula (10); Among them, MEI min MEI max SMF min SMF max These are the minimum and maximum values ​​of the microexpression interruption index and stereotyped movement frequency coefficient obtained statistically from a large-scale training dataset.

[0040] (2) Adaptive weighted fusion; The normalized indicators are used to calculate the final ASD comprehensive risk score through an adaptive weighted summation formula, as follows: , formula (11); Where ASRS is the calculated comprehensive risk score, with a value range typically between [0, 1]. A higher score indicates a higher risk of ASD. α and β are learnable weight parameters that satisfy α+β=1. (3) Risk level classification (fine-grained assessment); To provide more clinically relevant and granular assessments, continuous ASRS scores are mapped to discrete risk levels. For example, a three-level classification can be established using the following threshold: 1. Low Risk: ASRS≤T1; This indicates that the child's behavioral characteristics are not significantly different from those of a typical developing child, and routine follow-up is recommended.

[0041] 2. Medium Risk: T1 <ASRS≤T2; This indicates that the child is exhibiting some degree of abnormal behavior, and it is recommended to observe them closely and consider having a professional conduct an initial screening.

[0042] 3. High Risk: ASRS > T2; This indicates that the child is exhibiting significant ASD-related behavioral characteristics, and immediate referral to a specialist for a comprehensive clinical diagnosis is strongly recommended.

[0043] The scoring thresholds T1 and T2 are also based on clinically labeled training data.

[0044] In another embodiment of this application, an early risk prediction system for autism children based on posture and facial expressions is provided, the system comprising: (1) A data acquisition module (multi-source sensor) is configured to simultaneously acquire facial image sequences and body posture sequences of the target object. The data acquisition module includes an image sensing unit and a three-dimensional motion sensing unit. The image sensing unit is used to capture video streams containing facial regions. The three-dimensional motion sensing unit is used to acquire motion trajectory data of the target object's body joints in three-dimensional space. The image sensing unit includes RGB cameras, infrared cameras, etc. The three-dimensional motion sensing unit includes inertial measurement units (IMUs), etc.

[0045] (2) Expression feature extraction module (spatiotemporal dilated convolutional network), which is connected to the data acquisition module and configured to process the facial image sequence to extract temporal features that characterize the dynamic changes of facial muscles; the expression feature extraction module includes a spatiotemporal dilated convolutional network, which is used to jointly learn spatial appearance features and temporal motion features from a continuous frame sequence.

[0046] (3) Posture feature extraction module (posture feature extraction network), which is connected to the data acquisition module and configured to process the body posture sequence to extract spatiotemporal features that characterize the body joint movement pattern; the posture feature extraction module includes a spatiotemporal graph neural network, which is used to model the spatial connection relationship and temporal motion dependency between joints based on the topological structure of the human skeletal joint.

[0047] (4) Cross-modal fusion assessment module, including a cross-modal attention fusion unit and a risk quantification unit. A cross-modal attention fusion unit is configured to calculate the interaction attention weights between the temporal features and the spatiotemporal features, and dynamically fuse the features according to the weights to generate a fused feature vector. A risk quantification unit is configured to output an index for quantitatively assessing the risk of autism based on the fused feature vector. The risk quantification unit also calculates the micro-expression interruption index and the stereotyped movement frequency coefficient in parallel; it performs a weighted fusion of the micro-expression interruption index and the stereotyped movement frequency coefficient to generate a comprehensive risk score. A risk grading unit is also configured to compare the comprehensive risk score with a preset threshold range and output the corresponding discrete risk level information.

[0048] It should be noted that the system provided in the above embodiments is only an example of the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure can be divided into different functional modules to complete all or part of the functions described above. The system can be applied to an early risk prediction method for autism children based on posture and facial expression in the above embodiments.

[0049] In another embodiment of this application, a computer-readable storage medium is also provided, storing a computer program that, when executed by a processor, implements a method for early risk prediction of autism in children based on posture and facial expressions, as described in the above embodiment. Specifically: The video stream of facial expressions and the sequence of body posture movements of subjects during natural interactions were simultaneously acquired using multiple source sensors. A spatiotemporal dilated convolutional network is used to extract the temporal features of facial micro-expressions in the facial expression video stream; The spatiotemporal dynamics of body posture at key skeletal points in the body posture motion sequence are encoded using a posture feature extraction network. The temporal features and body posture spatiotemporal dynamic features are deeply fused by a cross-modal dynamic attention fusion engine. The fused features are then output in parallel via a fully connected layer as micro-expression interruption index and stereotyped motion frequency coefficient. By constructing a learnable weighted fusion layer, the micro-expression interruption index and stereotyped movement frequency coefficient are weighted and fused to obtain the comprehensive risk score of ASD. The scoring threshold is set and the subjects are classified into ASD risk levels.

[0050] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0051] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for early risk prediction of autism in children based on posture and facial expressions, characterized in that, It includes the following steps: Synchronously collect the facial expression video stream and the body posture motion sequence during the natural interaction of the subject through multi-source sensors; Adopt a spatio-temporal dilated convolutional network to extract the temporal features of facial micro-expressions in the facial expression video stream; Use a pose feature extraction network to encode the body pose spatio-temporal dynamics features of the bone key points in the body pose motion sequence; Perform feature depth fusion on the temporal features and the body pose spatio-temporal dynamics features through a cross-modal dynamic attention fusion器, and the fused features are parallelly output as the micro-expression interruption index and the stereotyped movement frequency coefficient through a fully connected layer; Construct a learnable weighted fusion layer to perform weighted fusion on the micro-expression interruption index and the stereotyped movement frequency coefficient to obtain the ASD comprehensive risk score, set a scoring threshold, and divide the ASD risk level of the subject.

2. The method for early risk prediction of autism in children based on posture and facial expressions according to claim 1, characterized in that, The multi-source sensors include an image sensing unit for collecting the facial expression video stream and a three-dimensional motion sensing unit for collecting the body pose motion sequence; the image sensing unit uses an RGB camera and / or an infrared camera; the three-dimensional motion sensing unit uses a stereo vision system and / or an inertial measurement unit.

3. The method for early risk prediction of autism in children based on posture and facial expressions according to claim 1, characterized in that, The spatio-temporal dilated convolutional network is a two-stream architecture, including several layers of 3D convolutional neural networks, which respectively process the original RGB frame sequence and the pre-computed optical flow frame sequence, and fuse the features of the two paths through an early or late strategy; The 3D convolutional neural network is obtained by expanding the 2D convolutional neural network through an inflation mechanism, specifically: increasing the two-dimensional convolutional kernel K in the pre-trained 2D convolutional neural network. 2D ∈R Kh×Kw Extending along the time dimension generates a 3D convolutional kernel K. 3D (t,h,w)∈R Kt×Kh×Kw Where R represents the set of real numbers, K t , K h , K w These represent the dimensions of the convolutional kernel in the time, height, and width directions, respectively. The early fusion strategy is specifically: before inputting the spatio-temporal dilated convolutional network, fuse the original RGB frame sequence and the pre-computed optical flow frame sequence; The late fusion strategy is specifically: after the original RGB frame sequence and the pre-computed optical flow frame sequence are independently processed by the spatio-temporal dilated convolutional network, fuse them after generating semantic features.

4. The method for early risk prediction of autism in children based on posture and facial expressions according to claim 3, characterized in that, The three-dimensional convolution kernel K 3D (t,h,w)∈R Kt×Kh×Kw Used for processing the input facial expression video stream V∈R T ×H×W×C Performing 3D convolution operations yields the temporal feature tensor F∈R of facial expressions. T′×H′×W′×C′ Where C and C' represent the number of input and output channels, respectively; T, H and W are the input dimensions of time, height and width, respectively; and T′, H′ and W′ are the output dimensions of time, height and width, respectively. in: ; When the number of channels C = 1, the single-channel convolution formula is: ； where X is the input tensor, t, x, y are the starting center coordinates of the current output point Y(t, x, y) in the input data, and t', i, j are the offsets based on this starting coordinate.

5. The method for early risk prediction of autism in children based on posture and facial expressions according to claim 1, characterized in that, The posture feature extraction network extracts spatiotemporal node feature sequences (V) from the input body posture motion sequence through cascaded stacked ST-Conv Block modules. t-M+1 , ..., V t To conduct joint learning; In each ST-Conv Block module, temporal dependencies are first captured using one-dimensional convolution and gated linear units (GLUs) through temporally gated convolution. Next, spatial graph convolution is applied to aggregate the spatial relationships between nodes based on predefined graph weights W. Finally, features are extracted again through temporally gated convolution units, and V is output. l+1 As input to the next ST-Conv Block module, it is eventually output through the output layer; The time-gated convolution is specifically: ； Among them, V l Indicates the first l The layer's output vector serves as the input to the entire GLU operation, V. temp (l+1) Indicates the first l +1 is a temporary vector, where * denotes a matrix multiplication operation. Represents element-wise product. σ For the sigmoid activation function, τ GLU W represents a gated linear unit operation. out W gate These are learnable 1D convolutional kernel weights.

6. The method for early risk prediction of autism in children based on posture and facial expressions according to claim 1, characterized in that, The cross-modal dynamic attention fusion器 calculates the attention weights between different modalities through self-learning, specifically: In cross-modal attention, vision → language: ; In cross-modal attention, audio → language: ; Modality fusion output language modality l j Integrating visual information: ； ； ； Where j, i, k represent the indices of the language, visual, and audio sequences, respectively; m, p, d represent the length of the visual and audio sequences and the dimension of the features, respectively; and I... j V represents the j-th language feature vector. i Let a represent the i-th visual feature vector. i Represents the i-th audio feature vector α i,j V→L Let α be the attention weight, representing the correlation between the j-th linguistic feature and the i-th visual feature. i,j A→L , representing the attention weight between the j-th language feature and the i-th audio feature, W Q L W is the query transformation matrix used for language modalities. K V W is the key transformation matrix used for visual modality. K A W is the key transformation matrix used for audio modalities. V V W is the value transformation matrix used for visual modality. V A This is the value transformation matrix used for audio modalities. Let j be the representation of the j-th language feature after being enhanced by visual information. Let j be the representation of the j-th language feature after it has been enhanced with audio information. This represents the final fused representation of the j-th language feature.

7. The method for early risk prediction of autism in children based on posture and facial expressions according to claim 1, characterized in that, The construction of a learnable weighted fusion layer to perform weighted fusion on the micro-expression interruption index and the stereotyped movement frequency coefficient to obtain the ASD risk score is specifically: Normalize the two coefficients of the micro-expression interruption index and the stereotyped movement frequency coefficient parallelly output by the fully connected layer: ； ； Among them, MEI min MEI max These are the minimum and maximum values ​​of the microexpression interruption index (SMF) obtained statistically from a large-scale training dataset. min SMF max These are the minimum and maximum values ​​of the stereotyped motion frequency coefficients obtained statistically from a large-scale training dataset; Calculate the ASD comprehensive risk score, as shown in the following formula: ； where ASRS is the calculated ASD comprehensive risk score, the value range is between [0, 1], and the higher the score, the higher the ASD risk; α and β are learnable weight parameters, and α + β = 1.

8. The method for early risk prediction of autism in children based on posture and facial expressions according to claim 1, characterized in that, The setting of the scoring threshold and the division of the ASD risk level of the subject are specifically: Map the continuous ASD comprehensive risk score to a discrete risk level: When ASRS ≤ T1, it is considered that the ASD risk level of the subject is low risk; When T1 < ASRS ≤ T2, it is considered that the ASD risk level of the subject is medium risk; When ASRS>T2, the subject's ASD risk level is considered high. Here, ASRS is the comprehensive risk score for ASD, and T1 and T2 are the score thresholds obtained from training data based on clinical annotations.

9. A system for predicting early risk of autism in children based on posture and facial expressions, characterized in that, The method for early risk prediction of autism in children based on posture and facial expression according to any one of claims 1-8 includes a multi-source sensor, a spatiotemporally dilated convolutional network, a posture feature extraction network, a cross-modal dynamic attention fusion fusion device, a learnable weighted fusion layer, and an ASD risk level classification module. The multi-source sensor is used to simultaneously acquire video streams of facial expressions and sequences of body postures during the subject's natural interactions. The spatiotemporal dilated convolutional network is used to extract temporal features of facial micro-expressions in the facial expression video stream; The posture feature extraction network is used to encode the spatiotemporal dynamic features of the skeletal key points in the body posture motion sequence; The cross-modal dynamic attention fusion unit is used to perform deep feature fusion of the temporal features and the spatiotemporal dynamic features of body posture. The fused features are output in parallel through a fully connected layer as micro-expression interruption index and stereotyped motion frequency coefficient. The learnable weighted fusion layer is used to weight and fuse the micro-expression interruption index and the stereotyped movement frequency coefficient to obtain the ASD comprehensive risk score; The ASD risk level classification module is used to set scoring thresholds and classify subjects' ASD risk levels.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements a method for early risk prediction of autism in children based on posture and facial expressions as described in any one of claims 1-8.

Images