A child autism intelligence auxiliary analysis method and system based on typical development behavior alignment

Abstract

The embodiment of the present application provides a kind of autism intelligence auxiliary analysis method and system based on typical development behavior alignment, it is related to the technical field of autism auxiliary analysis technology.The method includes: obtaining the facial video data of subject under multi-paradigm experiment;With the preset prior action space as the benchmark, the alignment degree of the multi-modal behavior of the subject and the prior action space is determined, and the first feature sequence is generated based on the alignment degree;The first feature sequence is subjected to timing feature extraction and global integration to obtain global embedding representation;According to the global embedding representation, the probability of autism spectrum disorder is determined.Through the present application, the problem of low precision of autism auxiliary analysis is solved, and the effect of improving the precision of auxiliary analysis is achieved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of autism auxiliary analysis, specifically to a method and system for intelligent auxiliary analysis of childhood autism based on alignment with typical developmental behaviors. Background Technology

[0002] Autism Spectrum Disorder (ASD) is a complex neurodevelopmental disorder characterized by impaired social communication and repetitive, stereotyped behaviors. Early auxiliary analysis of ASD is crucial for subsequent clinical intervention and improved prognosis. With the development of artificial intelligence technology, using computer vision to analyze children's facial videos under specific experimental paradigms to capture behavioral abnormalities has become a promising research direction.

[0003] However, existing data-driven methods generally face a fundamental technical contradiction in practical applications: on the one hand, the behavior of the ASD group exhibits extremely high heterogeneity, making it difficult to characterize using a unified model; on the other hand, children with typical developmental delays (TD) also show normal fluctuations in attention and behavioral changes in experiments.

[0004] Existing methods typically extract and splice high-dimensional raw behavioral features (such as gaze coordinates, emotion classification, head posture parameters, etc.). This "feature chaos" approach fails to establish a stable reference benchmark, making it difficult for the model to distinguish between the "normal behavioral fluctuations" of children with TD and the "atypical pathological deviations" of children with ASD. This results in a large number of false positive results (false alarms), which seriously limits the credibility of its clinical application. Summary of the Invention

[0005] This invention provides a method and system for intelligent auxiliary analysis of childhood autism based on alignment with typical developmental behaviors, in order to at least solve the problem of low accuracy in auxiliary analysis in related technologies.

[0006] According to one embodiment of the present invention, a method for intelligent auxiliary analysis of childhood autism based on alignment with typical developmental behaviors is provided, comprising: Acquire facial video data of subjects in multi-paradigm experiments; Based on a preset prior behavior space, the alignment degree between the subject's multimodal behavior and the prior behavior space is determined, and a first feature sequence is generated based on the alignment degree. Temporal feature extraction and global integration are performed on the first feature sequence to obtain a global embedding representation; The probability of autism spectrum disorder is determined based on the global embedding representation.

[0007] In one exemplary embodiment, generating the first feature sequence based on the alignment includes: Quantitative features of gaze modality, emotion modality, and head posture modality are extracted from the facial video data; The quantitative features of the gaze modality, head posture modality, and emotion modality are weighted and fused to generate the first feature; The first feature is assembled in chronological order to obtain the first feature sequence.

[0008] In one exemplary embodiment, extracting the quantization features of the gaze modality includes: Divide the screen into multiple preset areas; Determine the target region set based on the stimulus presentation status of the current experimental paradigm; Within a statistical timestamp, the number of video frames in which the subject's gaze coordinates fall within the target region set is used to obtain the quantitative characteristics of the gaze modality.

[0009] In one exemplary embodiment, extracting quantitative features of the emotion modality includes: Perform facial emotion recognition on each video frame within the timestamp; The number of video frames identified as having a neutral emotion within the timestamp is counted to obtain the quantitative characteristics of the emotion modality.

[0010] In one exemplary embodiment, extracting the quantization features of the head pose modality includes: Obtain the head pose Euler angles for each video frame within the timestamp, whereby the Euler angles include yaw angle, pitch angle, and roll angle; The number of video frames within the timestamp where the yaw angle, pitch angle, and roll angle are all within a preset stable threshold range is counted to obtain the quantitative characteristics of the head posture modality.

[0011] In an exemplary embodiment, temporal feature extraction and global integration of the first feature sequence to obtain a global embedding representation includes: The first feature sequence is input into a long short-term memory network layer to obtain a hidden state sequence; The hidden state sequence is divided into multiple fragment-level data, each fragment-level data corresponding to a different experimental paradigm; Within each of the fragment-level data, max pooling is performed along the feature dimension to obtain a globally context-aware fragment-level descriptor sequence; The fragment-level descriptor sequence is input into a multi-head self-attention module to calculate the association weights between fragments of different experimental paradigms, and then weighted aggregates them to generate the global embedding representation.

[0012] According to another embodiment of the present invention, a child autism intelligent auxiliary analysis system based on typical developmental behavior alignment is provided, comprising: The data acquisition module is used to acquire facial video data of subjects in multi-paradigm experiments; The feature alignment module, connected to the data acquisition module, is used to determine the alignment degree between the subject's multimodal behavior and the prior behavior space based on a preset prior behavior space, and to generate a first feature sequence based on the alignment degree. The encoding module, connected to the feature alignment module, is used to perform temporal feature extraction and global integration on the first feature sequence to obtain a global embedding representation; An auxiliary analysis module, connected to the encoding module, is used to receive the global embedding representation and determine the probability of autism spectrum disorder based on the global embedding representation.

[0013] In one exemplary embodiment, generating the first feature sequence based on the alignment includes: Quantitative features of gaze modality, emotion modality, and head posture modality are extracted from the facial video data; The quantitative features of the gaze modality, head posture modality, and emotion modality are weighted and fused to generate the first feature; The first feature is assembled in chronological order to obtain the first feature sequence.

[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is also provided, wherein a computer program is stored therein, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0015] According to yet another embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0016] This invention filters out the inherent normal fluctuation noise of the TD population by aligning with typical developmental behaviors, amplifies the discrimination boundary of atypical pathological deviations, improves feature recognition, and significantly reduces the false alarm rate. Therefore, it can solve the problem of low accuracy in auxiliary analysis and achieve the effect of improving the accuracy of auxiliary analysis. Attached Figure Description

[0017] Figure 1 This is a flowchart of a method for intelligent auxiliary analysis of childhood autism based on alignment with typical developmental behaviors according to an embodiment of the present invention; Figure 2This is a structural block diagram of a child autism intelligent auxiliary analysis system based on typical developmental behavior alignment according to an embodiment of the present invention; Figure 3 This is a schematic diagram of screen area division and experimental paradigm target area according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the encoder processing architecture according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0019] In the following description, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0020] Furthermore, in this application, directional terms such as "upper," "lower," "left," and "right" may be defined relative to the orientation of the components shown in the accompanying drawings. It should be understood that these directional terms can be relative concepts, used for relative description and clarification, and may change accordingly depending on the orientation of the components in the accompanying drawings.

[0021] In this application, unless otherwise expressly specified and limited, the term "connection" should be interpreted broadly. For example, "connection" can be a fixed connection, a detachable connection, or an integral part; it can be a direct connection or an indirect connection through an intermediate medium. Furthermore, the term "coupled" can refer to an electrical connection that enables signal transmission.

[0022] As used herein, “about,” “approximately,” or “approximately” includes the stated value and the average value within an acceptable range of deviation from the given value, wherein the acceptable range of deviation is determined by a person skilled in the art taking into account the measurement under discussion and the error associated with the measurement of the given quantity (i.e., the limitations of the measurement system).

[0023] Example 1 This embodiment provides an intelligent auxiliary analysis method and system for childhood autism spectrum disorder based on alignment with typical developmental behaviors. This method uses a pre-constructed rule-based typical developmental behavior space as a reference benchmark to align and quantify the multimodal behaviors of the subjects. Combined with a temporal modeling architecture of a long short-term memory network encoder for multi-head self-attention enhancement, it can extract highly discriminative pathological features from long-term multi-paradigm behavioral videos. This solves the technical problems of low specificity and sensitivity of auxiliary analysis systems in the prior art due to blurred feature boundaries and the inability of models to effectively capture transient abnormalities. It achieves the beneficial effect of improving the accuracy and reliability of auxiliary analysis for autism spectrum disorder.

[0024] Reference Figure 1 The document illustrates a flowchart of a method for intelligent auxiliary analysis of childhood autism spectrum disorder based on alignment with typical developmental behaviors according to an embodiment of the present invention, specifically including the following steps: S100 acquires facial video data of subjects in a multi-paradigm experiment through a data acquisition module.

[0025] The data acquisition module is equipped with a high-definition camera connected to the host system via a USB 3.0 interface. Responding to acquisition commands issued by the experimental control software, this module captures a continuous facial video stream of the subjects during the multi-paradigm visual induction experiment at a preset frame rate (e.g., 30 frames / second) and resolution (e.g., 1920x1080 pixels). The multi-paradigm experiment consists of a series of sequentially presented visual stimulus phases designed to systematically stimulate the subjects' behavioral performance in areas such as attention, social interaction, and emotional response. The phase sequence includes (but is not limited to) blank paradigm, overlapping paradigm, face-gazing paradigm, and exogenous stimulating paradigm. The raw YUV format video stream captured by the camera is encoded in real-time into H.264 format and encapsulated in an MP4 container. Simultaneously, a timecode generator synchronized with the video frames writes millisecond-level timestamp information into the data stream metadata. In order to perform subsequent batch processing, the data acquisition module is further equipped with a data segmenter. The data segmenter cuts the continuous video stream into a series of non-overlapping data blocks according to the preset analysis window duration (for example, 1 second, corresponding to 30 frames of video) to obtain timestamp units.

[0026] For example, a subject participates in a multi-paradigm experiment lasting 60 seconds; the front-end camera of the data acquisition module records at a sampling rate of 30 frames per second (FPS), generating a total of 1800 frames of video data; subsequently, the data segmenter parses the video stream into 60 independent timestamp data units, labeled as 1 second (30 frames) per time window. Each timestamp unit Logically, all include from arrive The experiment used 30 consecutive video frames. The experimental control software recorded the precise time points of the paradigm shift (e.g., frame 451, i.e., ...). The starting frame (marking the beginning of the "overlapping paradigm") is used to associate these paradigm labels with their corresponding timestamp units. Based on this, subsequent modules process the timestamp units. At that time, it can be found from the associated metadata that it belongs to the "blank paradigm", while in the processing timestamp unit When you look up the information, you can find that it belongs to the "overlapping paradigm".

[0027] S200 uses the prior behavioral space of a typical developmental group as a benchmark. The feature alignment module extracts and quantifies the alignment degree between the subject's multimodal behavior and the prior behavioral space, and fuses them to generate a single-dimensional first feature sequence.

[0028] The feature alignment module performs a nonlinear mapping of the high-dimensional, heterogeneous original behavioral feature space to a one-dimensional, normalized "alignment" space. Based on a predefined physical reference frame, this module quantifies the extent to which the current subject's behavior within each time stamp unit conforms to or deviates from this reference frame, which represents a large number of typical developmental (TD) children's normative behavioral patterns. Specifically, it includes the following sub-steps: S210 extracts quantitative features of gaze modality, emotion modality, and head posture modality for each timestamp in the facial video data.

[0029] Inside the feature alignment module, three parallel processing subunits extract features of three key behavioral modalities from the video frame sequence contained in each timestamp unit. The calculation method can directly reflect the degree of alignment with the normative behavior of the TD group.

[0030] First, the gaze alignment quantization subunit extracts gaze features. The input to this subunit is 30 frames of images from a single timestamp unit. Its internal logic first virtually divides the display screen into a 3x3 grid, generating nine rectangular regions. In response to the current experimental paradigm label parsed from the input metadata, this subunit dynamically loads the corresponding target region set from a pre-set lookup table. .

[0031] For example, assuming a screen resolution of 1920x1080 pixels, the size of each 3x3 grid area is 640x360 pixels, and the lookup table is defined as follows: If the normal form label is "blank normal form" Set as It is used to assess the ability to maintain a sustained gaze at the center of the screen.

[0032] If the paradigm label is "overlapping paradigm" Set as It is used to assess the efficiency of attention shift from the center to the periphery.

[0033] If the paradigm label is "Face Gazing Paradigm". Set as It is used to assess attention to social stimuli.

[0034] If the paradigm label is "exogenous modeling paradigm". Set as It is used to assess the ability to respond to environmental cues.

[0035] For each frame within that timestamp unit, a gaze estimation algorithm based on a convolutional neural network is invoked to output the screen coordinates of the gaze point in that frame. The counter is then initialized to zero. In a loop, for each of the 30 frames, the gaze coordinates are determined. Does it fall into the currently loaded Within the defined area. If so, the counter is incremented; after the loop ends, the final value of the counter is used as the line-of-sight feature value. Output. For example, in timestamps. Within the (overlap paradigm), if the gaze coordinates of 22 frames fall into region 4 or 5, then the output gaze feature value is... The count is 22 (count), and so on.

[0036] Secondly, the emotion alignment and quantization subunit extracts emotion features. This subunit receives the same 30 frames of image input and contains a facial emotion recognition model capable of recognizing seven basic emotions. This subunit also initializes a counter. In the frame-by-frame processing loop of the 30 images, each frame is fed into the emotion recognition model. If the model outputs a classification result of "neutral," the counter is incremented by one. After the loop ends, the value of the counter is used as the emotion feature value. Output. For example, in timestamps. Within the (face gaze paradigm), if 28 frames are classified as "neutral," then the output emotion feature value is... The count is 28.

[0037] Next, the head pose alignment and quantization subunit extracts pose features. This subunit uses a 3D facial keypoint regression-based model to estimate the head pose Euler angles (yaw, pitch, and roll) frame by frame from the input 30 frames of images. Internally, this subunit hard-codes a set of stable threshold ranges, such as the yaw angle... Pitch angle Roll angle Then, a counter is initialized. Subsequently, in the frame-by-frame processing loop, it is checked whether the three Euler angles of the current frame simultaneously satisfy the aforementioned range constraints; if so, the counter is incremented. After the loop ends, the counter value is used as the attitude feature value. Output. For example, in timestamps. Within the (blank paradigm), if the pose remains completely within the stable threshold for 25 frames, then the output pose feature value is... The count is 25.

[0038] S220 performs weighted fusion of the quantitative features of gaze modality, emotion modality, and head posture modality to generate the first feature of the single dimension under the timestamp.

[0039] The fusion unit receives data from three parallel sub-units at the same timestamp. The three scalar eigenvalues ​​output below , and This unit has a set of preset weighting coefficients. And satisfy Preferably, these weights are set as follows: (dimensionless) to reflect the greater importance of gaze behavior in diagnosis. This unit performs a weighted summation operation, the mathematical form of which is: This operation maps three independent alignment counts to a comprehensive single-dimensional canonical developmental alignment feature. Since this operation is a linear combination, the output is... This is the "weighted alignment frame number", and its value range is between [0, 30].

[0040] For example, when input is received At that time, the fusion unit performs the operation. The scalar value 19.75 is the final output of this timestamp.

[0041] S230, the first feature of each single dimension under each timestamp is spliced ​​together in chronological order to form the first feature sequence.

[0042] The sequence builder module has a first-in-first-out (FIFO) buffer to receive and temporarily store each first feature generated by the fusion unit in chronological order. ; throughout the entire video (total After processing the timestamps, this module will store all scalar values ​​in the buffer. Arrange them according to the order in which they entered the buffer, constructing a one-dimensional buffer of length [ ]. The floating-point array, i.e., the first characteristic sequence This array is passed to the subsequent timing coding module.

[0043] For example, for a 60-second experiment The sequence builder ultimately outputs a one-dimensional vector containing 60 floating-point numbers, for example... Each element value in this sequence reflects the degree to which the subject's behavior aligns with the TD norm within the corresponding second.

[0044] S300 inputs the first feature sequence into a multi-head self-attention enhanced long short-term memory network encoder to extract temporal features and integrate them globally, and outputs the global embedding representation of the subject.

[0045] The encoding module internally instantiates a multi-head self-attention enhanced long short-term memory (MHSA-LSTM) network model. This module is configured to process the input TD-Aligned temporal feature sequence. Deep temporal modeling was performed to extract a fixed-dimensional global embedding representation that could summarize the subject's overall behavioral patterns throughout the experiment. This process was decomposed into S310, S320, and S330.

[0046] S310, the first feature sequence is input into the Long Short-Term Memory (LSTM) network layer to obtain the hidden state sequence.

[0047] sequence The LSTM layer, containing 128 hidden units, is fed into the encoding module; at time step At that time, the LSTM unit receives and the initial zero vector hidden state With cell state The first hidden state vector is generated through the gated logic operations of its internal input gate, forget gate, and output gate. and cell state At each subsequent time step The output of the LSTM unit at the previous time step and current input As input, iteratively update its state and output. This process continues. After 1 time step, this layer outputs a dimension of Hidden state sequence matrix ; where each 128-dimensional vector All of them are encoded from the start to the current time step. Context dependency information.

[0048] S320: Based on the switching time point of the multi-paradigm experiment, the hidden state sequence is divided into multiple fragment-level data corresponding to different experimental paradigms. Within each fragment-level data, max pooling is performed along the feature dimension to obtain a globally context-aware fragment-level descriptor sequence.

[0049] Based on the experimental paradigm switching time points recorded in S100, a segmenter module processes the input hidden state sequence matrix. Perform row splitting. For example, in an experiment containing four paradigms, the sequence... It is divided into four sub-matrices: (Lines 1-15) (Lines 16-30) (Lines 31-45) and (Lines 46-60).

[0050] Next, the pooling unit performs max pooling operations on each of the four sub-matrices: for each sub-matrix (e.g., ... , dimension Pooling operations are performed independently along their column directions (i.e., feature dimensions); in order to generate pooled descriptors The elements ( (from 1 to 128), this cell compares submatrices. The Middle The column contains all 15 element values, and the maximum value is assigned to it. This operation ensures that the strongest signal activations occurring on each feature dimension (regardless of the instant they occur at) are preserved throughout the duration of the paradigm. After operating on all four submatrices, a dimension of... Fragment-level descriptor sequence matrix .

[0051] S330 inputs the fragment-level descriptor sequence into the multi-head self-attention (MHSA) module, calculates the association weights between fragments of different experimental paradigms, performs weighted aggregation, and generates a global embedding representation.

[0052] Fragment-level descriptor sequence matrix The input matrix is ​​fed into the MHSA module, which has eight parallel attention heads. In each head, the input matrix... Each row vector Multiplied by three independent trainable weight matrices respectively This generates the query, key, and value vectors for the header; by calculating the scaled dot product attention of the query vector and all key vectors, and applying the Softmax function, a... The attention weight matrix is ​​used, where each row of the weight matrix represents the attention distribution of one paradigm to all other paradigms. The value vectors are weighted and summed based on the weight matrix to generate the output of the head. The outputs of the eight heads are then concatenated and passed through a final linear transformation layer.

[0053] Finally, in order to generate a single global embedding representation, the mean pooling layer modifies the output of the MHSA module. The matrix is ​​averaged row-wise to generate a 128-dimensional vector. The vector It not only encodes salient features within each paradigm, but also incorporates interactions and dependencies between different paradigms through weighted aggregation of self-attention mechanisms, thus forming the final representation of the subject's overall behavioral pattern.

[0054] S400 takes the global embedding representation as input to a pre-trained classification model and outputs the probability that the subject belongs to autism spectrum disorder.

[0055] The auxiliary analysis module receives the 128-dimensional global embedding vector output by the encoding module. Internally, this module contains a pre-trained classifier consisting of two fully connected layers. The first layer maps the 128-dimensional input to 64 dimensions and applies the ReLU activation function; the second layer maps the 64-dimensional input to 2 dimensions; then, the Softmax function is applied to the output of the second layer to generate a two-dimensional probability vector. ,in The second element in the vector The probability that a subject belongs to the autism spectrum disorder is used as the final analysis result output; for example, if the output is [0.08, 0.92], the system determines that the subject has a 92% probability of belonging to the ASD group.

[0056] Example 2 Unlike Embodiment 1, the feature alignment module in this embodiment generates a multi-channel temporal feature sequence. This sequence not only includes the "compliance" with the canonical behavior, but also explicitly quantifies the "deviation" from the canonical behavior, thereby providing richer and more discriminative information for the subsequent temporal coding module.

[0057] Specifically, in this embodiment, step S200 is replaced by S210, and method S210 includes the following sub-steps: S211, construct a temporal multimodal probability gauge space.

[0058] First, the data processing pipeline analyzes a large number of TD children's training sample videos, processing each timestamp... Extract raw floating-point feature vectors containing 5 dimensions from each frame. ,in and To normalize the screen view coordinates to the [0,1] interval (dimensionless), Euler angles for head posture (unit: degrees).

[0059] Subsequently, for each experimental paradigm (e.g., the "face gaze paradigm"), the original feature vectors of all TD samples at all timestamps under that paradigm are aggregated; at this point, the statistical modeling unit fits the dataset using a multivariate Gaussian distribution, which is calculated by the mean vector of these feature vectors. Covariance Matrix To parameterize the probability gauge space under this paradigm; where the 5-dimensional mean vector This includes the expected behavior of the TD group under this paradigm, and the 5x5 covariance matrix. This quantifies the fluctuation range of each behavioral feature (diagonal elements) and their linear dependencies (off-diagonal elements).

[0060] For example, for the "face gaze paradigm", the statistical modeling unit calculates the mean vector based on tens of thousands of frames of TD children's data. and a specific 5x5 covariance matrix The mean vector indicates that, under this paradigm, children with TD (Total Disturbance) have an average gaze slightly off-center from the screen, with a slight average head tilt. The covariance matrix may reveal that when the head turns to the left (yaw decreases), the gaze also tends to move to the left side of the screen. (Reduce) the statistical correlation.

[0061] S212, calculates multi-channel alignment and offset features.

[0062] When conducting online analysis on new subjects, the feature alignment module is used at each time stamp. Two processing channels will be started in parallel: (1) Compliance Channel: The function and implementation of this channel are exactly the same as S210 and S220 in Example 1. Its output is a scalar value based on rule counting that measures the "compliance" of the behavior. .

[0063] (2) Deviation Channel: This channel is used to quantify the degree of deviation between the subject's behavior and the statistical norms of the TD group; for timestamps For each frame within the frame, extract its original feature vector. The system first queries the current paradigm and loads the corresponding... and The mathematical operation unit then calculates the Mahalanobis distance between the frame's action vector and the normative probability canonical space. : The physical meaning of this operation lies in the fact that it projects the original feature vectors onto a new space where the features are uncorrelated and have a variance of 1, and then calculates the Euclidean distance; therefore, it quantifies the degree to which a data point statistically deviates from the center of the distribution; a timestamp Deviation characteristics within This is the average Mahalanobis distance over the 30 frames within that timestamp.

[0064] For example, under the "face gaze paradigm", the input is the behavioral vector of a subject in a certain frame. Subsequently, the mathematical operation unit is based on the pre-stored... and Calculate the Mahalanobis distance for this frame. (Dimensionless); and for a typical TD child, their behavioral vector might be: The calculated Mahalanobis distance may only be .

[0065] S213 generates a multi-channel time-series feature sequence.

[0066] Vector splicer at each timestamp The output of the conformity channel Output of the deviation channel Combined into a two-dimensional feature vector .

[0067] Then, collect all the two-dimensional vectors of timestamps sequentially to form a vector of dimension 1. Multi-channel time-series feature sequence matrix .

[0068] In subsequent steps, this two-dimensional temporal feature sequence The data is fed into the MHSA-LSTM encoder. The input dimension of the encoder's LSTM layer is set to 2 accordingly, and then step-by-step processes such as piecewise max pooling and multi-head self-attention integration are performed. The relevant steps are consistent with those in Example 1 and will not be described in detail here.

[0069] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0070] This embodiment also provides an intelligent auxiliary analysis system for children with autism based on alignment with typical developmental behaviors. This system is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the apparatus described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0071] Figure 2 This is a structural block diagram of a child autism intelligent auxiliary analysis system based on typical developmental behavior alignment according to an embodiment of the present invention, such as... Figure 2 As shown, the system includes: The data acquisition module is used to acquire facial video data of subjects in multi-paradigm experiments; The feature alignment module, connected to the data acquisition module, is used to determine the alignment degree between the subject's multimodal behavior and the prior behavior space based on a preset prior behavior space, and to generate a first feature sequence based on the alignment degree. The encoding module, connected to the feature alignment module, is used to perform temporal feature extraction and global integration on the first feature sequence to obtain a global embedding representation; An auxiliary analysis module, connected to the encoding module, is used to receive the global embedding representation and determine the probability of autism spectrum disorder based on the global embedding representation.

[0072] In an optional embodiment, generating the first feature sequence based on the alignment includes: Quantitative features of gaze modality, emotion modality, and head posture modality are extracted from the facial video data; The quantitative features of the gaze modality, head posture modality, and emotion modality are weighted and fused to generate the first feature; The first feature is assembled in chronological order to obtain the first feature sequence.

[0073] The following examples illustrate this point.

[0074] A smart auxiliary analysis system for childhood autism based on alignment with typical developmental behaviors consists of a data acquisition module, a feature alignment module, an MHSA-LSTM encoding module, and an auxiliary analysis module.

[0075] Data acquisition module: used to acquire facial video data of subjects under a pre-set multi-paradigm experiment and divide the acquired facial video data into non-overlapping time windows.

[0076] Feature alignment module: Connected to the data acquisition module, it is used to extract multimodal behavioral features such as gaze, facial emotion, and head posture from facial video data, align and quantify these features with the prior behavioral space of the typical developmental (TD) group, and fuse them to generate a one-dimensional typical developmental alignment (TD-Aligned) temporal feature sequence.

[0077] The MHSA-LSTM encoding module, connected to the feature alignment module, receives temporal feature sequences, extracts basic temporal features, performs data segmentation and local maximum feature extraction according to the experimental paradigm, calculates cross-paradigm global attention, and outputs a global embedding representation.

[0078] The auxiliary analysis module is connected to the MHSA-LSTM encoding module. It receives the global embedding representation, inputs it into the trained classification model, and outputs the probability that the subject is identified as having autism spectrum disorder (ASD), providing intelligent auxiliary analysis results.

[0079] The intelligent auxiliary analysis method for childhood autism based on alignment with typical developmental behaviors in this embodiment is executed based on the above system and includes the following steps: Step S1: The data acquisition module acquires facial video data of the subjects in the multi-paradigm experiment.

[0080] Step S2: The feature alignment module extracts and quantifies the spatial alignment between multimodal behavior and the TD population, and fuses them to obtain a single-dimensional TD-Aligned temporal feature.

[0081] Step S3: The MHSA-LSTM encoding module sequentially extracts the TD-Aligned temporal features, performs segmented max pooling, and cross-paradigm self-attention processing to obtain a global embedding representation.

[0082] Step S4: The auxiliary analysis module inputs the global embedding representation into the pre-trained classification model and outputs the intelligent auxiliary analysis results for autism.

[0083] The implementation method of step S2, "extracting and quantifying the spatial alignment between multimodal behavior and TD population using the feature alignment module," is as follows: S21 (Gaze Alignment Quantization): Divide the screen into 9 rectangular regions. For each time stamp t, dynamically define the target region set at that time stamp based on the stimulus presentation status of the current experimental paradigm (see appendix). Figure 3 The number of video frames within a time stamp t in which the subject's gaze coordinates fall into the target area set is recorded as the gaze feature value gt.

[0084] S22 (Emotion Alignment Quantization): Facial emotion recognition is performed on each frame within timestamp t. The number of frames within timestamp t that are classified as "neutral" emotion is counted and denoted as the emotion feature value et.

[0085] S23 (Head Attitude Alignment Quantization): Estimate the head attitude Euler angles (yaw, pitch, roll) for each frame within timestamp t. Set a preset stability threshold and count the number of video frames where all three Euler angles are within the stability threshold range, denoted as the attitude feature value pt.

[0086] S24 (Feature Fusion): The three quantized features mentioned above are weighted and fused to calculate the single-dimensional TD-Aligned feature xt under the timestamp t; the calculation formula is as follows: The temporal feature sequence X of the entire video segment is obtained by concatenating the sequences in chronological order.

[0087] The step S3, "MHSA-LSTM encoding module extracts global embedding representation" (see appendix) Figure 4 The implementation method is as follows: S31 (Basic Temporal Extraction): Input the feature sequence X into the Long Short-Term Memory (LSTM) network layer, and output the hidden state sequence corresponding to each timestamp through the LSTM memory unit.

[0088] S32 (Paradigm Segmentation and Segmented Max Pooling): Based on the switching time points of the multi-paradigm experiments, the hidden state sequence is explicitly segmented into variable-length data segments corresponding to different experimental paradigms. Within each segment, max pooling is performed on the hidden state in each dimension to extract the most salient feature activation values, resulting in a compressed segment-level descriptor sequence.

[0089] S33 (Cross-Paradigm Global Integration): The fragment-level descriptor sequence is input into the Multi-Head Self-Attention (MHSA) module, which calculates the Query, Key, and Value sequences and calculates the association weights between fragments of different paradigms through scaling dot products. After mean and normalization operations, a global embedding representation for characterizing the global behavior of the subject is output.

[0090] The intelligent auxiliary analysis method for childhood autism based on alignment with typical developmental behaviors includes a model building and training phase before steps S1 to S4. The specific implementation method is as follows: S41. Obtain multi-paradigm facial video samples of children with known developmental typicality (TD) and autism spectrum disorder (ASD) with known classification labels, and extract TD-Aligned temporal features according to the method described in step S2 to construct a training dataset.

[0091] S42, build the MHSA-LSTM encoder and subsequent classifier network.

[0092] S43 (Design A-SupCon Loss): Design an asymmetric supervised contrastive loss function. During loss function calculation, an asymmetric positive-negative sample mapping logic is established: clustering is performed only on samples from the TD group in the feature space (reducing intra-class distance for TD); for samples from the ASD group, intra-class clustering is not enforced to maintain the inherent behavioral heterogeneity distribution of the ASD group.

[0093] S44 uses an asymmetric supervised contrastive loss function to iteratively train the MHSA-LSTM encoder multiple times, enabling the model to maximize the feature boundary between the canonical behavior of the TD group and the anomalous behavior of the ASD group while avoiding representation collapse. Then, with the encoder parameters fixed, the classifier is trained using a cross-entropy loss function until the model converges, resulting in a well-trained intelligent auxiliary analysis model.

[0094] It should be noted that the above modules can be implemented by software or hardware. For the latter, they can be implemented in the following ways, but are not limited to: all the above modules are located in the same processor; or, the above modules are located in different processors in any combination.

[0095] Embodiments of the present invention also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to perform the steps in any of the above method embodiments when executed.

[0096] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0097] Embodiments of the present invention also provide an electronic device including a memory and a processor, the memory storing a computer program and the processor being configured to run the computer program to perform the steps in any of the above method embodiments.

[0098] In one exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0099] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for analyzing the intelligence of children with autism based on alignment with typical developmental behaviors, characterized in that, include: Acquire facial video data of subjects in multi-paradigm experiments; Based on a preset prior behavior space, the alignment degree between the subject's multimodal behavior and the prior behavior space is determined, and a first feature sequence is generated based on the alignment degree. Temporal feature extraction and global integration are performed on the first feature sequence to obtain a global embedding representation; The probability of autism spectrum disorder is determined based on the global embedding representation.

2. The method according to claim 1, characterized in that, The step of generating the first feature sequence based on the alignment includes: Quantitative features of gaze modality, emotion modality, and head posture modality are extracted from the facial video data; The quantitative features of the gaze modality, head posture modality, and emotion modality are weighted and fused to generate the first feature; The first feature is assembled in chronological order to obtain the first feature sequence.

3. The method according to claim 2, characterized in that, The quantization features extracted from the gaze modality include: Divide the screen into multiple preset areas; Determine the target region set based on the stimulus presentation status of the current experimental paradigm; Within a statistical timestamp, the number of video frames in which the subject's gaze coordinates fall within the target region set is used to obtain the quantitative characteristics of the gaze modality.

4. The method according to claim 2, characterized in that, Extracting quantitative features of emotional modalities includes: Perform facial emotion recognition on each video frame within the timestamp; The number of video frames identified as having a neutral emotion within the timestamp is counted to obtain the quantitative characteristics of the emotion modality.

5. The method according to claim 2, characterized in that, The quantization features extracted from head pose modalities include: Obtain the head pose Euler angles for each video frame within the timestamp, whereby the Euler angles include yaw angle, pitch angle, and roll angle; The number of video frames within the timestamp where the yaw angle, pitch angle, and roll angle are all within a preset stable threshold range is counted to obtain the quantitative characteristics of the head posture modality.

6. The method according to claim 1, characterized in that, Temporal feature extraction and global integration are performed on the first feature sequence to obtain a global embedding representation, including: The first feature sequence is input into a long short-term memory network layer to obtain a hidden state sequence; The hidden state sequence is divided into multiple fragment-level data, each fragment-level data corresponding to a different experimental paradigm; Within each of the fragment-level data, max pooling is performed along the feature dimension to obtain a globally context-aware fragment-level descriptor sequence; The fragment-level descriptor sequence is input into a multi-head self-attention module to calculate the association weights between fragments of different experimental paradigms, and then performs weighted aggregation to generate the global embedding representation.

7. A child autism intelligent auxiliary analysis system based on alignment with typical developmental behaviors, characterized in that, include: The data acquisition module is used to acquire facial video data of subjects in multi-paradigm experiments; The feature alignment module, connected to the data acquisition module, is used to determine the alignment degree between the subject's multimodal behavior and the prior behavior space based on a preset prior behavior space, and to generate a first feature sequence based on the alignment degree. The encoding module, connected to the feature alignment module, is used to perform temporal feature extraction and global integration on the first feature sequence to obtain a global embedding representation; An auxiliary analysis module, connected to the encoding module, is used to receive the global embedding representation and determine the probability of autism spectrum disorder based on the global embedding representation.

8. The system according to claim 7, characterized in that, The step of generating the first feature sequence based on the alignment includes: Quantitative features of gaze modality, emotion modality, and head posture modality are extracted from the facial video data; The quantitative features of the gaze modality, head posture modality, and emotion modality are weighted and fused to generate the first feature; The first feature is assembled in chronological order to obtain the first feature sequence.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program is configured to perform the method described in any one of claims 1 to 6 when executed.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the method as described in any one of claims 1 to 6.

Images