Autism recognition system, storage medium and device based on hybrid deep learning

Abstract

The present invention provides an autism recognition system, storage medium and device based on hybrid deep learning, which relate to the field of computer-aided recognition technology, including: a data acquisition module, used to acquire continuous video data; a skeleton key point extraction module, used to identify each skeleton key point of a human target and its corresponding position according to the video data as a skeleton sequence; a recognition and classification module, used to input the skeleton sequence graph into a 2sG-ALSTM network architecture, classify the skeleton data in the original skeleton sequence based on a partitioning strategy, divide the adjacency matrix of the graph with self-loops into multiple matrices, and then map from the posture space to the feature space, extract posture sequence features related to upper body and head movements from the feature space, input the posture sequence features into an LSTM network, and automatically assign different attention weights to different frames in the posture sequence features in the temporal attention module of the LSTM network, and finally output the recognition and classification results.

Description

Technical Field

[0001] The present disclosure relates to the field of computer-aided recognition technology, and in particular to an autism recognition system, storage medium, and device based on hybrid deep learning. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] Autism Spectrum Disorder (ASD) is a rapidly growing neurodevelopmental disorder that mainly manifests in early childhood. Timely intervention is crucial for the growth and development of ASD children, but traditional auxiliary clinical screening methods are time-consuming and labor-intensive and lack measurable indicators.

[0004] Computer vision (CV) technology is increasingly being used to analyze and identify human behavior, providing a more objective and efficient means for ASD detection. Researchers have developed various protocols, such as name-calling response, expressing needs with index finger pointing (ENIFP), and robot-assisted protocol (RAP) to analyze key behaviors such as head, finger, and facial movements. These protocols help assess the quality of joint attention and social communication, which is enhanced by CV capabilities. In addition, trials also use biomarkers such as eye tracking, head movement, and motor movements to identify ASD characteristics.

[0005] However, protocols focused on a single biomarker may not fully capture the complexity of social interactions and cognitive behaviors. Traditional scales such as the ADI-R, ASIEP-3, or STAT, while commonly used to assess ASD symptoms, require prolonged direct observation by clinicians. While deep learning has emerged as a key tool to enhance ASD screening, aiming to surpass traditional performance benchmarks. However, there is still room for improvement in modeling the temporal dynamics of complex movements.

[0006] The inability of existing 3DCNNs to achieve higher performance can be attributed to the inherent characteristics of the activities in the collected datasets. Unlike behavioral datasets, it is difficult for CNNs to extract useful information from the background to accurately identify behavioral activities. Summary of the invention

[0007] In order to solve the above problems, the present invention proposes an autism recognition system, storage medium and device based on hybrid deep learning, a hybrid deep learning 2sG-ALSTM network based on video skeleton behavior analysis, combines dual-stream graph convolution with attention-enhanced LSTM, extracts spatiotemporal features from long-term behaviors, and uses the PCB dataset to enhance the system-assisted screening process for ASD in young children.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions:

[0009] Autism recognition system based on hybrid deep learning, including:

[0010] A data acquisition module, used to acquire data set information obtained based on a parent-child duo block game protocol, wherein the data set information is continuous video data;

[0011] The skeleton key point extraction module is used to identify each skeleton key point of the human target and its corresponding position based on the high-resolution network according to the video data as a skeleton sequence;

[0012] The recognition and classification module is used to input the skeleton sequence in the form of a graph into the 2sG-ALSTM network architecture. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on the partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, it is mapped from the posture space to the feature space, and the posture sequence features related to the upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the temporal attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output.

[0013] According to some embodiments, the present disclosure adopts the following technical solutions:

[0014] A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions, and when the computer instructions are executed by a processor, the program is executed by the processor to implement the following steps:

[0015] Acquire data set information obtained based on a parent-child duo block game protocol, where the data set information is continuous video data;

[0016] According to the video data, each skeleton key point of the human target and its corresponding position are identified based on a high-resolution network as a skeleton sequence;

[0017] The skeleton sequence is input into the 2sG-ALSTM network architecture in the form of a graph. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on the partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, it is mapped from the posture space to the feature space, and the posture sequence features related to the upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the temporal attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] An electronic device includes: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the program to implement the following steps:

[0020] Acquire data set information obtained based on a parent-child duo block game protocol, where the data set information is continuous video data;

[0021] According to the video data, each skeleton key point of the human target and its corresponding position are identified based on a high-resolution network as a skeleton sequence;

[0022] The skeleton sequence is input into the 2sG-ALSTM network architecture in the form of a graph. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on the partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, it is mapped from the posture space to the feature space, and the posture sequence features related to the upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the temporal attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] This paper proposes an autism identification system based on hybrid deep learning, and proposes a parent-child group game (PCB) protocol to construct a dataset. PCB is designed to capture specific ASD-related behaviors of young children and provide standardized dataset guidance for future consistent evaluation; a comprehensive annotated PCB video dataset, which is more extensive than previous datasets in terms of the number of participants and the duration of a single session. This dataset records the interactive behaviors of young children and provides an important resource for fine-grained behavioral analysis for early screening of ASD;

[0025] This paper proposes a hybrid deep learning framework 2sG-ALSTM for video skeleton behavior analysis. The framework combines dual-stream graph convolution with attention-enhanced LSTM to extract spatiotemporal features from long-term behaviors. The framework shows excellent performance in action recognition, robustness, and spatiotemporal domain feature extraction; the attention layer can calculate attention weights based on input values, allowing the model to flexibly focus on changes in input values. This method helps the model capture key information in the input sequence more accurately, thereby improving model performance; the PCB dataset is used to enhance the screening and discrimination process of ASD in young children.

[0026] Compared with other traditional methods, the method disclosed in this paper has achieved higher performance. Combined with the 2sG-ALSTM technology, it can more effectively retain the spatiotemporal features in the data set. In the simulation, the comparison between the two skeleton-based methods further confirmed this and obtained a relatively higher accuracy rate compared with the skeleton-based LSTM method. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0028] Figure 1 An example frame of an experimental scene in a dataset collected for an embodiment of the present disclosure;

[0029] Figure 2 Observe the operation process of the parent-child interaction props of the embodiment of the present disclosure;

[0030] Figure 3 2sG-ALSTM network architecture diagram of an embodiment of the present disclosure;

[0031] Figure 4 is an example diagram of different neighborhoods of a node in an embodiment of the present disclosure;

[0032] Figure 5 The GCN network block structure of the embodiment of the present disclosure;

[0033] Figure 6 This is a diagram of the existing LSTM network architecture of an embodiment of the present disclosure;

[0034] Figure 7 LSTM model diagram with attention mechanism according to an embodiment of the present disclosure;

[0035] Figure 8 It is an example framework of the experimental scenario of ASD children in the embodiment of the present disclosure;

[0036] Fig. 9 This is an example framework of the TD children experimental scenario in the embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0038] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0039] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0040] Example 1

[0041] In one embodiment of the present disclosure, a hybrid deep learning-based autism recognition system is provided, including a data acquisition module, a skeleton key point extraction module, and a recognition and classification module. The following describes in detail the method content specifically executed in each module:

[0042] 1. A data acquisition module is used to obtain data set information based on a parent-child duo block game protocol, where the data set information is continuous video data.

[0043] Specifically, a novel parent-child dyad block play (PCB) protocol based on kinesiology and neuroscience research is proposed to identify behavioral patterns that distinguish ASD from typically developing toddlers. In the context of PCB, a large video dataset was compiled, including 40 toddlers diagnosed with ASD and 89 typically developing toddlers playing block play with their parents.

[0044] Based on the parent-child dyad block play protocol and video behavior recognition to assess the interaction between young children and their environment, with a particular focus on their head, hand, and block movements in a structured environment, the PCB protocol is a structured task sequence designed to assess and engage young children, especially children with autism, in a controlled experimental environment. A standardized environment was established to enable social attention and cognitive abilities to be consistently assessed through play. The scene settings and interaction architecture are as follows:

[0045] like Figure 1 As shown, design an observation platform with an area of ​​about 10-15m 2 The observation room for the parent-child dyad experiment was equipped with a suitable table and two chairs. The experimental materials consisted of 10 cubic bricks and a box of children's irregular bricks. Different building block games were designed for children of different ages. Before the dyad experiment began, the children were provided with props and corresponding instructions in the task manual to help parents and children conduct the dyad experiment. In the observation room, a standard RGB camera with recording function was installed to capture the facial expressions and body movements of the children and parents. By collecting this data, we can analyze the behavior of both parents and children and evaluate the quality of the interactive activities.

[0046] The process of observing parent-child dyads through task completion involves setting specific tasks that require cooperation between the child and the parent. The purpose of the task is to assess the interaction during task completion. The observation process is divided into three sessions, each of which lasts approximately 8 minutes. Figure 2 As shown. The PCB protocol is divided into four different stages:

[0047] Phase 1: Children and parents were given ten cubes and an instruction sheet and were asked to complete a set task in approximately three minutes. At the end of the time, the observer prompted the participant to complete the task. Completing the task within the allotted time was not mandatory.

[0048] Phase 2: The observer retrieved the cube and instructions from the participant, provided the child with a box of irregular bricks, and allowed the child to play freely for approximately 3 minutes. The session began with a voice prompt from the observer.

[0049] Phase 3: Participants were asked to pack up and organize the previously used bricks for approximately two minutes. The session began after the observer gave an audio prompt.

[0050] Phase 4: includes ending the recording process.

[0051] Then it is collected as the Parent-Child Building Block Game (PCB4ASD-ED) dataset, which includes 187 videos, including 97 ASD video clips and 90 TD video clips, each video clip is about 20 seconds long. This dataset is almost twice as large as the previous benchmark dataset SSBD (68 videos), and there is no shared video between the two datasets.

[0052] Since the PCB4ASD-ED dataset is continuous and long-term video behavior data, each segment in the video needs to be manually labeled. In terms of data acquisition equipment, the camera parameters used to acquire the video are fixed, and the entire dataset is converted to a rate of 17 frames per second. After conversion, there are a total of 72,418 frames in the dataset. This dataset focuses on the theme of parent-child dyads in building block games and is collected from different individuals. However, there are also videos from different environments related to the same topic. There are a total of 129 participants, including 40 ASD children and 89 normally developing children.

[0053] Furthermore, the video frames in the collected dataset are Gaussian smoothed to reduce the details in the visual appearance of the subject. This blurs the image, reduces noise and improves object recognition by giving more weight to the central pixels and less weight to the surrounding pixels.

[0054] 2. Skeleton key point extraction module, which is used to identify each skeleton key point of the human target and its corresponding position based on the high-resolution network according to the video data as a skeleton sequence;

[0055] First, since the video contains interactions between people and objects, it is necessary to detect the subject performing the action. To this end, the Faster R-CNN network is used to implement child detection. The Faster R-CNN network is an object detection model with 13 convolutional layers, 13 ReLU layers, and 4 pooling layers to detect people in each frame. The input image passes through the backbone network to extract feature maps, and then the RPN network uses these feature maps to generate candidate regions. Finally, the candidate regions are classified and regressed to bounding boxes through the detection network. The detection network uses the RoI pooling layer to convert candidate regions of different sizes into fixed-size feature vectors. The output includes the coordinates of the bounding box, object, and category prediction.

[0056] The output of FasterR-CNN is then fed to HRNet (High Resolution Network), which predicts the position of each skeletal keypoint of the identified human target and performs skeleton keypoint extraction; HRNet has multiple parallel branches, each of which uses different convolution kernel sizes and strides to extract features of different scales and generate multiple scale feature maps. Feature maps of different scales are then fused at the pixel and channel levels to obtain richer and more accurate feature representations. Finally, a fully connected layer with multiple output nodes obtains the coordinates and confidence of each corresponding keypoint. Both models are pre-trained on the COCO dataset with 80 categories. The network takes a 1333×800 tensor as input and outputs a 17×3 tensor as a skeleton sequence.

[0057] 3. The recognition and classification module is used to input the skeleton sequence in the form of a graph into the 2sG-ALSTM network architecture. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on the partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, it is mapped from the posture space to the feature space, and the posture sequence features related to the upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the temporal attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output.

[0058] Specifically, the present disclosure proposes a 2sG-ALSTM network architecture, a human skeleton action recognition method based on GCN (two-stream graph convolution) and LSTM, wherein feature extraction is performed based on GCN (two-stream graph convolution). First, the normalized skeleton sequence x can be represented in the form of a graph G = {N, E}, where N is a set of joint nodes, N = [n1, n2, ..., n k ], k is the number of nodes across T frames, and E represents the edges connecting the nodes. Intra-frame edges are defined based on the natural connections between joints, and inter-frame edges are defined based on the connections of the same joints between consecutive frames.

[0059] GCN updates the features of the root node by aggregating the local set of spatial nodes, and uses the partitioning strategy and residual blocks to preliminarily define the layer-by-layer propagation rules of the two-stream GCN (2s-GCN), specifically:

[0060] First, the acquired skeleton sequence is preprocessed, and the T-frame skeleton sequence x = {x raw |t=1,…,T} involves normalization to improve stability and accelerate convergence during training. Specifically, the raw feature vector x of the t-th frame raw The two sub-vectors {x raw,m |m=1,2}, representing different parts of the body (upper body and head). These subvectors are labeled according to their part membership, where the label variable m is set to 1 for upper body and 2 for head. To ensure that each subvector is normalized independently, the normalization process is performed on x raw,m The mathematical expression is:

[0061]

[0062] in and σ(x raw,m ) are x raw,m The mean and standard.

[0063] Furthermore, the partitioning strategy considered is limited to skeleton data with complex topological structures in the spatial dimension. Specifically, G t = {N t ,E t} represents the spatial graph of the skeleton in frame t, where the root node v ti The neighbor set of ti )={v tj |d(v ti ,v tj )≤1}. Here, i and j represent node labels, d(v ti ,v tj ) corresponds to v ti to v tj It is worth noting that different neighbor sets may differ in the number and order of nodes, which makes it infeasible to directly implement kernel sharing. To overcome this challenge, two partitioning strategies are designed to divide the neighbor set into a fixed number of K subsets and map them to the corresponding nodes through the mapping function l ti :N(v ti )→{1,…,K} assigns the label {1,…,K} to each node v tj ∈N(v ti ) to complete.

[0064] One strategy is to partition the neighbor set based on the distance of each node to a specified root node, mainly used for key points in the head. This method is called distance partitioning and involves partitioning the neighbor set into subgroups based on the minimum path length from each internal node to the root node. Formally, distance partitioning can be expressed as:

[0065] l ti (v tj )=d(v ti ,v tj )+1

[0066] Among them l ti (v tj ) is N(v ti )Node v tj This method divides the neighbor set of a node into two different subsets, namely the root node and its 1-neighbor nodes.

[0067] Another strategy, i.e., multi-scale space partitioning, solves the weight bias problem of neighboring nodes that are relatively far away, e.g. Figure 4 As shown. Node self-loops in GCNs lead to more possible loops, which may amplify the deviation and cause the skeleton key point sequence to be dominated by signals from local body parts. Self-loops also cause the model to be unable to capture long-range key point dependencies of high-order polynomials. To address this problem, different k values ​​are assigned to different adjacency matrices to obtain different scales. This allocation method is applied to the skeleton key points of the upper body. This can be achieved through the mathematical formula:

[0068]

[0069] After the skeleton data is classified, each part of the skeleton data is subjected to graph convolution. The spatial aggregation strategy of graph convolution can usually be expressed mathematically as follows:

[0070]

[0071] Where X(v tj ) is the node v tj The input features of . W(·) is a weight function composed of the label l ti (v tj ) is distributed from K weights. ti (v tj ) is the number of neighbors v tj , which normalizes the feature representation. out (v ti ) represents node v tj The output of the graph convolutional layer. Based on the partitioning strategy, the adjacency matrix A of the skeleton graph with self-loops can be divided into K matrices {A k|k=1,…,K}. Mathematically, this can be expressed as A=∑ k A k For illustration, both distance partitioning and spatial configuration partitioning can be expressed as A1=I, where I is the identity matrix. Similarly, the degree matrix Λ can also be decomposed into K matrices {D k |k=1,…,K} graph topology calculation formula can be expressed as:

[0072]

[0073] Where σ represents the activation function. is the symmetric normalized k-neighborhood.

[0074] In the multi-scale space partitioning strategy, a new adjacency matrix is ​​defined The following equation is obtained:.

[0075]

[0076]

[0077]

[0078] Where min is the minimum function. According to the above formula, the formula of 2S-GCN on the entire input feature map is X = R NTC , where N, T, and C are the number of joints, frames, and channels.

[0079] The GCN module first maps the input from the posture space to the feature space, and then extracts the posture sequence features related to the upper body and head movements from the feature space. Then, the GCN block is used to extract features in this feature space, and a residual connection is added between every 3 GCN blocks; then the network module is able to directly learn the residual instead of the target posture. Finally, a residual connection is added between the input posture and the output posture to ensure that the network learns the difference between them. This residual connection is designed to improve the accuracy of posture feature extraction. The GCN block architecture is as follows Figure 5 shown.

[0080] Then, an adaptive fusion module is used to assign weights for the fusion of multiple features. Although it is common to add or concatenate multimodal features in many studies, in our task, the role of the arms is obviously more important than the secondary role of the head. Therefore, a weight assignment mechanism is designed that considers the hierarchical relationship between features.

[0081] set up represents the 256-dim feature vector obtained by mapping the output of the mth part in the tth frame of the multilayer perceptron. This formula is used to fuse multiple features of the tth frame, and the mathematical expression is:

[0082]

[0083] where α m is the spatial importance weight assigned to the label m, which is learned by network adaptive learning. Restricted to 1 and α m ∈[0,1], and define

[0084]

[0085] Where λ is a reinforcement factor that controls the magnitude of the change in α. ω is a set of parameters used to iteratively optimize the model, initialized to 0. ω can be learned via standard back-propagation.

[0086] Furthermore, the information provided by frames in a skeleton sequence is not of equal value. Key frames contain the most discriminative information, while other frames provide contextual information. For example, in the "block stacking" action of the ASD parent-child dyad, the "hands approaching" sub-stage is considered more important than the "arms opening" sub-stage. To address this issue, the present disclosure designs a temporal attention module in the LSTM network to automatically assign different attention weights to different frames. The temporal attention module allows the model to more accurately capture the valuable information provided by the key frames in the sequence, thereby improving the performance of the model.

[0087] LSTM is a variant of RNN and has shown excellent ability in modeling long-term temporal dependencies in sequence modeling. The LSTM used contains three gates: input gate i t , forget the door t and output gate o t Each gate interacts with other gates to improve the information analysis ability of the LSTM model. The structure of LSTM is as follows Figure 6 shown.

[0088] Unit memory c t The temporal dynamics are shown through the recurrent edges that are self-connected by their weights and are connected to the hidden state H t Together. The functional definition of the LSTM unit is as follows:

[0089] i t =σ(W xi X t +W hi H t-1 +b i )

[0090] f t =σ(W xf X t +W hf H t-1 +bf )

[0091] o t =σ(W xo X t +W ho H t-1 +b o )

[0092] u t =tanh(W xc X t +W hc H t-1 +b c )

[0093] C t =f t ⊙C t-1 +i t ⊙u t

[0094] H t =o t ⊙tanh(C t )

[0095] Where ⊙ represents the Hadamard product. σ is the sigmoid activation function. t is the modulation input.

[0096] The attention layer transforms the hidden state h = [h k ,h k+1 ,…,h k+w-1 ] T ,h i ∈R 1×n As its input, and based on this input, calculate a set of attention weights α that represent the influence of each hidden state on the result k ,α k+1 ,…,α k+w-1 The model then weights and sums the inputs to obtain the resulting vector l k The architecture of the attention layer is as follows Figure 7 To improve the performance of the model, this attention mechanism allows the model to focus more on the important parts of the input sequence and less on the irrelevant parts.

[0097] The attention mechanism can be expressed as follows:

[0098] W a =W[x t h t-1 ]+b

[0099]

[0100]

[0101] l k =ReLU(∑α i h i )

[0102] Where W a ∈R w×n represents the weight matrix. i,j Represents the element W in the matrix a b∈R w×n and w∈R w×n is a learnable parameter. k is the result vector. During model training, the model inherently learns the impact of each input element on the output and generates attention weights for each time step. As the sliding window moves, the input sequence value changes, but the attention layer can calculate the attention weights based on the input value, allowing the model to flexibly focus on the changes in the input value. This method helps the model capture the key information in the input sequence more accurately, thereby improving the model performance and ultimately outputting the recognition classification results.

[0103] Experimental simulation

[0104] By analyzing the subjects' behaviors while participating in block play activities, their ability to construct block structures under parental guidance can be assessed. Figure 8 The different interactions between children and mothers when building blocks at different time periods are shown. It shows that children with ASD often need the help of their mothers and have difficulty meeting the task requirements. In contrast, TD children are able to complete the building of blocks independently. The experimental results show that this method has a high accuracy in identifying ASD patients. In order to ensure reliable and objective observation of the subjects' interaction ability, standardized scenarios and structured assessments are used. In this case, upper body movements and head movements are focused on, and an accuracy of approximately 0.79 and an unweighted average recall of 0.77 are obtained (Table 1).

[0105] Table 1 shows the complete data used for each category. The first and second categories are videos of head movements and upper body actions, respectively.

[0106]

[0107] Table 2 shows the 10-fold results using different physical features for screening. The worst performance is using a single feature from category 1 (head) or category 2 (upper body). The reason may be that the number of features detected using a single feature is small. As shown in the “Fused Features” column of Table 1, the recognition performance is improved by up to 6 (from 0.73 to 0.79) after using the adaptive fusion module. This shows the effect of fusing information from multiple feature maps, thereby avoiding irrelevant data. As shown in the “WithAttention” column of Table 2, using the Attention mechanism in the model can lead to a performance enhancement of about 3.

[0108] The attention mechanism further enhances the model's ability to capture long-range frame behavioral features, overcoming the performance degradation caused by long sequence analysis. The three rows of data labeled "ASD data", "TD data", and "full data" in Table 1 show that models trained using only ASD or TD toddlers as samples perform poorly when tested using the full data (including ASD and TD data). This finding suggests that it is necessary to increase the diversity of samples and models that rely on single symptom samples with suboptimal generalization performance.

[0109] The results showed that children with ASD showed significant differences in block play behaviors compared to TD children. Specifically, toddlers with ASD lacked fluency and spontaneity in their behavioral interactive movements, especially when building block structures. In addition, there were significant differences in the ability to complete block play tasks under parental guidance and showed less interactivity.

[0110] Table 2 Evaluation results of all frameworks: local descriptor based methods, articulated pose based methods and CNN based methods.

[0111]

[0112] In all categories, the proposed method outperforms the other methods. It is worth emphasizing that the proposed method achieves higher performance compared to the CNN-based methods, as LSTM is considered to be superior in extracting temporal features. Overall, the combination of 2SG-ALSTM technique can more effectively preserve the spatiotemporal features in the dataset. This is further confirmed by the comparison of the two skeleton-based methods, which achieved relatively high accuracy compared to the skeleton-based LSTM method (11). It is worth noting that the 3DCNN was unable to achieve higher performance, which can be attributed to the inherent characteristics of the activities in the collected dataset. Unlike behavioral datasets, it is difficult for CNNs to extract useful information from the background to accurately identify behavioral activities.

[0113] Example 2

[0114] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the program is executed by the processor to implement the following steps:

[0115] Acquire data set information obtained based on a parent-child duo block game protocol, where the data set information is continuous video data;

[0116] According to the video data, each skeleton key point of the human target and its corresponding position are identified based on a high-resolution network as a skeleton sequence;

[0117] The skeleton sequence is input into the 2sG-ALSTM network architecture in the form of a graph. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on the partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, it is mapped from the posture space to the feature space, and the posture sequence features related to the upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the temporal attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output.

[0118] Example 3

[0119] In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the program to implement the following steps:

[0120] Acquire data set information obtained based on a parent-child duo block game protocol, where the data set information is continuous video data;

[0121] According to the video data, each skeleton key point of the human target and its corresponding position are identified based on a high-resolution network as a skeleton sequence;

[0122] The skeleton sequence is input into the 2sG-ALSTM network architecture in the form of a graph. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on the partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, it is mapped from the posture space to the feature space, and the posture sequence features related to the upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the temporal attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output.

[0123] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0124] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0125] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. Autism recognition system based on hybrid deep learning, characterized by: include: A data acquisition module, used to acquire data set information obtained based on a parent-child duo block game protocol, wherein the data set information is continuous video data; The parent-child dyad block game protocol is a structured task sequence. The facial expressions and body movements of children and parents when performing the task sequence are captured by a camera, and multiple video clips are compiled as a data set; The skeleton key point extraction module is used to identify each skeleton key point of the human target and its corresponding position based on the high-resolution network according to the video data as a skeleton sequence; A recognition and classification module is used to input the skeleton sequence in the form of a graph into a 2sG-ALSTM network architecture. The 2sG-ALSTM network architecture is a human skeleton action recognition method based on GCN and LSTM. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on a partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, the posture space is mapped to the feature space, and posture sequence features related to upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the time attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output; The partitioning strategy is limited to skeleton data with complex topological structures in the spatial dimension. Two partitioning strategies are designed to divide the neighbor set into a fixed number of K subsets, which are completed by assigning labels to each node through a mapping function, including a strategy for partitioning the neighbor set and a multi-scale space partitioning strategy; The strategy for partitioning the neighbor set is: based on the distance from each node to the specified root node, the key point partitioning for the head is called distance partitioning, including dividing the neighbor set into subgroups according to the minimum path length from each internal node to the root node; multi-scale space partitioning is used for key points of local body parts, different adjacency matrices are assigned different k values, and different scales are obtained.

2. The autism recognition system based on hybrid deep learning as claimed in claim 1, characterized in that: Based on the video data, it is segmented into frame images, and the images are input into the Faster R-CNN network. The feature map is extracted through the backbone network, and then the RPN network is used to generate human candidate regions using the feature map. Classification and bounding box regression are performed through the detection network part, and candidate regions of different sizes are converted into feature vectors of fixed size. The output includes the coordinates of the bounding box, object and category prediction to achieve human recognition.

3. The autism recognition system based on hybrid deep learning as claimed in claim 2, characterized in that: The output of the FasterR-CNN network is then input into the HRNet network. HRNet has multiple parallel branches. Each branch uses different convolution kernel sizes and step sizes to extract features of different scales, generate multiple scale feature maps, and then fuse the feature maps of different scales at the pixel and channel levels. The coordinates and confidence of the key points corresponding to each skeleton are obtained in the fully connected layer as a skeleton sequence.

4. The autism recognition system based on hybrid deep learning as claimed in claim 1, characterized in that: The skeleton sequence is plotted as It is expressed in the form of, N is the set of joint nodes, E represents the edge connecting the nodes, the intra-frame edge is defined based on the natural connection between joints, and the inter-frame edge is defined based on the connection of the same joint between consecutive frames.

5. The autism recognition system based on hybrid deep learning as claimed in claim 1, characterized in that: To extract features from posture data, the GCN module first maps the input from the posture space to the feature space. Then, the GCN block is used to extract features in the feature space, and residual connections are added between every 3 GCN blocks. Then, the residual is directly learned instead of the target posture. Finally, a residual connection is added between the input posture and the output posture to learn the difference between them. Then, the weight distribution mechanism considering the hierarchical relationship between the features is used for adaptive fusion. The fused features are used to output the prediction probability for classification.

6. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the following steps are implemented: Acquire data set information obtained based on a parent-child duo block game protocol, where the data set information is continuous video data; According to the video data, each skeleton key point of the human target and its corresponding position are identified based on a high-resolution network as a skeleton sequence; The skeleton sequence is input into the 2sG-ALSTM network architecture in the form of a graph. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on the partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, it is mapped from the posture space to the feature space, and the posture sequence features related to the upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the temporal attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output.

7. An electronic device, characterized in that: include: A processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the program to implement the following steps: Acquire data set information obtained based on a parent-child duo block game protocol, where the data set information is continuous video data; According to the video data, each skeleton key point of the human target and its corresponding position are identified based on a high-resolution network as a skeleton sequence; The skeleton sequence is input into the 2sG-ALSTM network architecture in the form of a graph. First, the skeleton data in the original skeleton sequence is classified into upper body data and head data based on the partitioning strategy, and the adjacency matrix of the graph with self-loops is divided into multiple matrices. Then, it is mapped from the posture space to the feature space, and the posture sequence features related to the upper body and head movements are extracted from the feature space. The posture sequence features are input into the LSTM network. In the temporal attention module of the LSTM network, different attention weights are automatically assigned to different frames in the posture sequence features, and finally the recognition and classification results are output.

Images