Method, device and machine-assisted system for autism children behavior recognition and motion prediction

Abstract

The application discloses a method, device and diagnosis and treatment machine auxiliary system for autism child behavior recognition and motion prediction, the method comprises the following steps: (1) acquiring original skeleton video data of the child from different angles; (2) inputting the acquired original skeleton video data into a view adaptive conversion unit to adaptively convert to a consistent coordinate system to obtain the optimal representation of the human skeleton; (3) inputting the optimal representation of the human skeleton into a multi-scale feature extraction unit to extract high-performance spatiotemporal features of the human skeleton; (4) inputting the high-performance spatiotemporal features into a multi-task learning unit to perform behavior recognition and motion prediction on the human body, and obtaining behavior classification results and motion prediction results. The application can predict and intervene the future motion while recognizing the behavior.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to methods, devices and diagnostic machine-assisted systems for behavior recognition and motion prediction in children with autism. Background Technology

[0002] Autism spectrum disorder (ASD) is one of the most common developmental disorders, characterized by impairments in social communication and interaction, and restricted, repetitive interests and behaviors. In my country, there are approximately 200,000 children with ASD each year. Studies suggest that ASD can be detected as early as 14 months of age, and has a high degree of certainty before age 3. However, recent surveys show that over 70% of children with ASD do not receive effective intervention or diagnosis before age 4.

[0003] Early identification and screening for children with ASD primarily takes two forms. The first involves guardians closely observing the child's behavior, recording and completing questionnaires, which are then used by experienced physicians for diagnosis and evaluation. This method requires significant manpower, relies on recall, and has a degree of subjectivity. The second method utilizes computer vision-based identification and diagnostic systems. These systems collect and process children's behavioral data to train decision-making models for abnormal behaviors, thus enabling screening.

[0004] However, current machine vision-based ASD behavior recognition methods have the following shortcomings: (1) In terms of the collection and processing of ASD children's behavior data: one method is to use wearable devices to collect information such as speed and acceleration from multiple parts of the child's arms, shoulders, elbows, etc., which usually causes discomfort or even aversion in children, resulting in unreliable information in the collected data; another method is to use video capture devices to collect children's motion data, but ignores the problem of low recognition accuracy caused by data collected by the device from different perspectives, and the need to consume a lot of manpower and computing costs based on manually defined perspective consistency standards; (2) In terms of feature extraction and model structure for ASD children's behavior data: ASD children exhibit weak or obvious chaotic behavior that is significantly different from the behavior explored in the current behavior recognition field. Ordinary feature extraction methods and model structures are not suitable for the analysis and modeling of ASD children's behavior features. Because existing methods mainly focus on how to extract high-order semantic information from sequences, without considering the different contributions of human skeletal structure in different behaviors, the recognition effect of specific behaviors of ASD children is not good. For example, the stereotyped movement "repeatedly twisting the arm" that is common in children with ASD is a coordinated movement of the arm and wrist joints, that is, the relationship between different parts of the human body, and it is not necessary to model all the joints; (3) In terms of the diagnosis and intervention of children with ASD behavior: the existing models are insufficient in processing the movement data and extracting features of children with ASD, resulting in low diagnostic accuracy. In addition, there are very few reports on visual models that can predict and intervene in future movements while recognizing the extreme behaviors of children with ASD, such as "aggression" and "self-harm". That is, the existing research separates the ASD behavior recognition task and the movement prediction task, resulting in insufficient understanding of movement by the model, which affects the efficiency of deep network-assisted diagnosis and intervention. Summary of the Invention

[0005] This invention provides a method and device for behavior recognition and movement prediction in children with autism, as well as a diagnostic machine-assisted system that can predict and intervene in future movements while recognizing behaviors.

[0006] The technical solution of the present invention is as follows:

[0007] A method for behavioral recognition and movement prediction in children with autism includes:

[0008] (1) Obtain raw skeleton video data of children from different angles;

[0009] (2) Input the acquired raw skeleton video data into the view adaptive transformation unit to adaptively transform it to a consistent coordinate system to obtain the optimal representation of the human skeleton;

[0010] (3) Input the optimal representation of the human skeleton into the multi-scale feature extraction unit to extract high-performance spatiotemporal features of the human skeleton.

[0011] (4) Input high-performance spatiotemporal features into the multi-task learning unit to perform human behavior recognition and motion prediction, and obtain behavior classification results and motion prediction results.

[0012] Step (2) includes:

[0013] (2-1) Based on the rotation rules of vectors in a three-dimensional coordinate system, the rotation matrix is ​​obtained.

[0014]

[0015] In the formula, Let α represent the set of trainable parameters. t ,β t ,θ t These are three angular parameters representing rotation around the X, Y, and Z axes, respectively.

[0016] (2-2) The original skeleton video data is translated and rotated according to the translation vector and rotation matrix to obtain the optimal representation of the human skeleton;

[0017] The j-th joint s' of the t-th frame after conversion t,j Represented as:

[0018]

[0019] In the formula, γ t Represents the trainable translation parameters; s t,j This represents the human body coordinates before the transformation; s' t,j This represents the transformed human body coordinates; x' t,j ,y' t,j ,z' t,j These represent the X, Y, and Z coordinates of the transformed frame t and the j-th joint, respectively.

[0020] Furthermore, the view adaptive transformation unit uses an AGC-LSTM network to learn the translation vector and rotation parameters.

[0021] To enable the view adaptive transformation unit to automatically learn trainable parameters and γ t When training the view adaptive transformation unit, the loss function is set as follows:

[0022]

[0023] Based on the input of the skeleton, different transformation parameters are learned to generate different optimal representations for each frame.

[0024] Step (3) includes:

[0025] (3-1) Spatial feature modeling operators are used to extract spatial features at the joint scale and body part scale of the human body, respectively, including:

[0026] For human joint scale: combining the i-th joint s i and the j-th joint s j The characteristics of the relationship between them are represented as f i,j Use the following formula for f i,j Perform iterative calculations to extract long-short features:

[0027]

[0028] f i,j =MLP([s i ,s j ])

[0029] In the formula, MLP(·) represents a multilayer perceptron that maps joint features, and uses the “Concat” function in the Pandas library to concatenate features;

[0030] The i-th joint s is calculated using the following formula. i and the j-th joint s j Adjacency matrix A i,j :

[0031]

[0032] In the formula, T represents the transpose of the matrix; k is used for summation and counting, with a range of [1, J], where J represents the total number of human joints labeled in the dataset;

[0033] The adjacency matrix A is calculated using the following formula. cp :

[0034] A cp =A⊙W+A'⊙W'+A”⊙W”

[0035] In the formula, W is the corresponding multi-order trainable weight matrix; ⊙ is the element-based product operation; A is the graph convolution of the skeleton; A' represents the first-order polynomial of A; A” represents the second-order polynomial of A; W' represents the first-order polynomial of W; W” represents the second-order polynomial of W.

[0036] Finally, the spatial characteristics of human joint scale are obtained using the following formula:

[0037]

[0038] In the formula, γ is the order; R(·) is the rule function; S' t,jW represents the input to the multi-scale feature extraction unit. i,j Represents the weight matrix;

[0039] For human body parts at the scale: use the neighbor adjacency matrix A hp =diag(A)⊙W replaces the adjacency matrix A cp , diag(·) is a diagonal transformation matrix, and the other calculation methods are the same as those for extracting spatial features at the human joint scale;

[0040] (3-2) Temporal feature modeling operators are used to extract temporal features at the human joint scale and the human body part scale, respectively, including:

[0041] For the human joint scale, one-dimensional convolution (Conv) is used to extract features along the time direction of the input sequence, and then the ReLU activation function is used for nonlinear transformation to obtain the temporal features of the human joint scale. The calculation formula is as follows:

[0042] T-HJ(S' t,j )=R(Conv[HJ(S' t,j );k,s])

[0043] In the formula, T-HJ(S' t,j ) represents the time features at the human joint scale, and k and s represent the kernel size and stride, respectively;

[0044] The method for extracting temporal features at the human body part scale is the same as the method for extracting temporal features at the human body joint scale.

[0045] (3-3) Copy all columns of the features at the human body part scale once so that they can match the size of the features at the human joint scale; use the Concat function to concatenate the features at the human joint scale and the human body part scale to obtain the final high-performance spatiotemporal features of the multi-scale feature extraction unit.

[0046] Step (4) includes:

[0047] (4-1) The high-performance spatiotemporal features are converted into feature vectors using an average pooling layer, and the softmax layer is used as a classifier to output the behavior classification results.

[0048] (4-2) The high-performance spatiotemporal features are re-extracted using the graph convolution operator, and the output features are refined to location X. p Speed ​​X v and acceleration X a Three types of outputs are normalized to obtain a refined representation of the features of the re-extracted frame t.

[0049]

[0050] The output F(t+1) of the motion prediction in frame t+1 is obtained by the following formula:

[0051]

[0052] In the formula, F(t) represents the pose of the t-th frame. The label represents the behavior classification result of frame t, and GRU stands for Gated Recursive Calculation.

[0053] The obtained behavioral recognition and motion prediction results can assist in the diagnosis and screening of children with ASD and the intervention of aggressive behaviors.

[0054] The present invention also provides a device for behavioral recognition and movement prediction in children with autism, comprising:

[0055] The View Adaptive Transformation Unit (VAT-GCN) is used to automatically learn transformation parameters and transform human body data collected from different perspectives into a unified perspective to obtain the optimal representation of the human skeleton.

[0056] The Multi-Scale Feature Extraction Unit (MS-GCN) is used to extract features from the optimal representation of the human skeleton to obtain high-performance spatiotemporal features.

[0057] The multi-task learning unit is used to perform behavior recognition and motion prediction based on the high-performance spatiotemporal features, and to obtain behavior classification results and motion prediction results.

[0058] The behavioral classification results are used for the auxiliary diagnosis and screening of children with ASD, and the motor prediction results are used for the intervention of children's aggressive behaviors.

[0059] The aforementioned view adaptive transformation unit uses an AGC-LSTM network to learn transformation parameters. These transformation parameters include a translation vector and a rotation matrix, which transform the input original skeleton sequence to a unified viewpoint, obtaining the optimal representation of the human skeleton. Each frame of the input original skeleton sequence can be transformed to a unified coordinate system using the above transformation parameters, resulting in the optimal representation of the human skeleton.

[0060] The multi-scale feature extraction unit includes a human joint spatiotemporal feature extraction module and a human body part spatiotemporal feature module; both spatiotemporal feature extraction modules contain a spatial feature modeling operator (HJ-GCN or HP-GCN) and a temporal feature modeling operator (T-HJ or T-HP).

[0061] Based on the differences in the human skeleton during movement, the multi-scale feature extraction unit divides the optimal representation of the human skeleton from the view adaptive transformation unit into two scales to deeply explore the relationship between the skeleton and human movement. The two scales are: the top focuses on mining the spatiotemporal features between the joints of the human body, and the bottom aims to mine the spatiotemporal features between different parts of the human body. Each human body is labeled with 20 joints and 10 parts. Each scale includes a spatial feature modeling operator (HJ-GCN or HP-GCN) and a temporal feature modeling operator (T-HJ or T-HP). A fusion scheme is used between the two scales to learn features and output a high-performance feature representation.

[0062] The multi-task learning unit includes an action recognition module and a motion prediction module. The action recognition module uses an average pooling layer to convert the high-performance spatiotemporal features output by the multi-scale feature extraction unit into feature vectors, and uses a Softmax layer as a classifier to output action classification results. The motion prediction module uses a graph convolution operator to re-extract the high-performance spatiotemporal features to obtain a refined representation of the re-extracted features. Taking the pose, the re-extracted features, and the corresponding action classification result labels as input, it extracts hidden features and uses a gated recursive unit to calculate and obtain the motion prediction result for the next frame.

[0063] This invention also provides a diagnostic machine-assisted system for behavior recognition and motion prediction in children with autism, comprising a computer memory, a computer processor, and a computer program stored in the computer memory and executable on the computer processor. The computer memory contains a view adaptive transformation unit, a multi-scale feature extraction unit, and a multi-task learning unit. When the computer processor executes the computer program, it performs the following steps: inputting raw skeleton video data of the child from different angles into the view adaptive transformation unit to adaptively transform it to a consistent coordinate system to obtain the optimal representation of the human skeleton; inputting the optimal representation of the human skeleton into the multi-scale feature extraction unit to extract high-performance spatiotemporal features of the human skeleton; and inputting the high-performance spatiotemporal features into the multi-task learning unit to perform behavior recognition and motion prediction on the human body to obtain behavior classification results and motion prediction results.

[0064] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0065] This invention integrates a view adaptive transformation unit into a GCN-based framework. By automatically learning transformation parameters, it transforms the human body to the optimal unified viewpoint, thereby improving the representation of the human 3D skeleton. This overcomes the difficulties and variability in data collection for children with ASD in specific scenarios, saves computational costs, and indirectly improves the accuracy of diagnosis and intervention for children with ASD.

[0066] The multi-scale feature extraction unit of this invention decomposes the behavioral characteristics of children with ASD into two scales, combines the different contributions of the human skeletal structure in different movements, models the spatial features of the human body and the temporal features of consecutive frames, and uses a special fusion method to obtain consistent spatiotemporal features. These improvements increase the targeting of feature extraction to the behavior of children with ASD, and the high-performance features obtained directly improve the accuracy of diagnosis and reduce the error of the intervention system by approximately 0.6.

[0067] The multi-task learning unit and self-supervised learning scheme of this invention can simultaneously process action recognition and motion prediction tasks, and output the results of both tasks in parallel. High-performance features from the action recognition task can be used to predict motion, while features extracted from the motion prediction task can help understand the actions. Attached Figure Description

[0068] Figure 1 This is a connection block diagram of each unit in the general graph neural network of the present invention;

[0069] Figure 2 This is a structural diagram of the view adaptive transformation unit;

[0070] Figure 3 This describes the specific process of adaptive transformation of the skeleton model.

[0071] Figure 4 This is a structural diagram of a multi-scale feature extraction unit;

[0072] Figure 5 A connection structure diagram for a multi-task human activity learning unit;

[0073] Figure 6 The diagram shows the joints and parts of the human body, where (a) shows the relationship between any joint and other joints, and (b) shows the relationship between each part and other parts.

[0074] Figure 7 This is a rendering of the view adaptive transition unit;

[0075] Figure 8 This is an overall flowchart of the present invention;

[0076] Figure 9 This is a diagram illustrating the motion prediction task of the present invention, where the first row represents the actual motion labels and the second row represents the prediction results of the present invention. Detailed Implementation

[0077] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be noted that the embodiments described below are intended to facilitate the understanding of the present invention and do not limit it in any way.

[0078] The purpose of this invention is to overcome the shortcomings of existing vision-based technologies and effectively assist in the diagnosis and intervention of children with ASD. It provides a method for assisted diagnosis and prediction based on a general graph neural network of the human skeleton, combining behavior recognition and motion prediction technologies for the assisted diagnosis and intervention of children with ASD.

[0079] To address the aforementioned shortcomings (1), this invention provides an adaptive adjustment scheme for skeleton representation integrated into a graph neural network (GCN). This scheme automatically transforms the human body to a unified viewpoint based on the learned features, thereby overcoming the impact of viewpoint changes on the accuracy of behavior recognition and saving manual processing costs.

[0080] To address the aforementioned deficiency (2), this invention provides several GCN-based graph operators to describe the positional relationships and abstract dynamics between different joints and parts of the human body. By learning two types of human body graphs and temporal information, multi-scale spatiotemporal features are obtained. A feature extraction unit invented based on the behavioral characteristics of children with ASD enables the network to more comprehensively understand the behavior of children with ASD and obtain high-performance spatiotemporal features.

[0081] To address the aforementioned shortcomings (3), this invention provides a practical multi-task learning framework and a multi-objective self-supervised learning scheme, employing a feature sharing method to improve network learning efficiency. Specifically, labels obtained from action recognition tasks are used for motion prediction, while detailed features generated during motion prediction are used for action recognition. This solves the problem of low accuracy in behavior recognition and long-term motion prediction in specific scenarios (family, kindergarten, etc.), providing more reliable data for the diagnosis and intervention of children with ASD. Simultaneously, this invention allows for a thorough study of multi-task human activity understanding models, providing a new reference method for human behavior analysis.

[0082] Figure 1 This is a diagram showing the connection structure between the various units in the Vers-GNN (Vers-GNN) of this invention. The Vers-GNN includes a View Adaptive Transformation Unit (VAT-GCN), a Multi-Scale Feature Extraction Unit (MS-GCN), and a Multi-Task Learning Unit. The View Adaptive Transformation Unit automatically learns transformation parameters and transforms human body data collected from different viewpoints to a unified viewpoint, obtaining the optimal representation of the human skeleton to address the impact of view changes. The Multi-Scale Feature Extraction Unit consists of multiple graph convolution operators, processed by a bidirectional fusion scheme, to extract features from the optimal representation of the human skeleton, obtaining high-performance spatiotemporal features. The Multi-Task Learning Unit promotes the mutual reinforcement of behavior recognition and motion prediction tasks, obtaining behavior classification and motion prediction results. The behavior classification results are used for the diagnosis and screening of children with ASD, and the motion prediction results are used for intervention of aggressive behaviors in children with ASD.

[0083] Figure 2 This is a structural diagram of the view adaptive transformation unit of the invention. The AGC-LSTM network is used to learn the transformation parameters. These transformation parameters include a translation vector and a rotation matrix, used to transform the original skeleton sequence of the input. The AGC-LSTM network is a publicly disclosed model structure, implemented by replacing the input, hidden states, and unit memories in a standard LSTM with a graphical structure. Figure 3 This describes the transformation process of the original skeleton through the view adaptive transformation unit. Each frame of the original input skeleton can be transformed to a unified coordinate system using the above transformation parameters to obtain the optimal representation of the human skeleton.

[0084] Figure 4 The connection structure of each part in the multi-scale feature extraction unit of the invention is used to learn the features of the aforementioned view adaptive transformation unit. Based on the mutual differences in the human skeleton during movement, this unit divides the skeleton from the view adaptive transformation unit into two scales to deeply explore the relationship between the skeleton and human movement. The two scales are: the top focuses on mining the spatiotemporal features between the joints of the human body, and the bottom aims to mine the spatiotemporal features between different parts of the human body. The joints and different parts of the human body are as follows: Figure 6 As shown, each human body is labeled with 20 key points and 10 body parts. Each scale includes a spatial feature modeling operator (HJ-GCN or HP-GCN) and a temporal feature modeling operator (T-HJ or T-HP). A fusion scheme is used between the two scales to learn features to output a high-performance feature representation.

[0085] Figure 5 This is a connected structure for a multi-task learning unit. For behavior recognition in children with ASD, a standard Softmax layer is used as a classifier to output the final diagnostic result. For motion prediction, the obtained high-performance features are re-extracted by a newly invented feature extractor, and three types of unified representations are output in parallel to predict motion for intervention in ASD-related aggressive behaviors. This unit correlates the results of behavior recognition and motion prediction; that is, the results of behavior recognition are used to assist motion prediction, and the results of motion prediction are used to improve the accuracy of behavior recognition.

[0086] Furthermore, such as Figure 8 As shown, the method for human behavior recognition and motion prediction of the present invention includes:

[0087] The S100 uses three Kinect V2 sensors to acquire raw skeleton video data of the child from different angles (only one is needed for model testing) and initializes all layers of the model.

[0088] S101, a. Input the acquired raw skeleton data into the view adaptive transformation unit to adaptively transform it to a consistent coordinate system and obtain the optimal representation of the human skeleton.

[0089] b. Furthermore, the view adaptive transformation unit primarily uses translation and rotation to transform the original skeleton. Specifically, based on the rotation rules of vectors in the three-dimensional coordinate system, the rotation matrix is ​​obtained.

[0090]

[0091] In the formula, Let α represent the set of trainable parameters. t ,β t ,θ t These are three angular parameters representing rotation around the X, Y, and Z axes. These parameters are adaptively learned during model training based on the child's specific movements. Through translation and rotation, the original input human skeleton is transformed to a consistent coordinate system. The coordinates of the j-th joint in the t-th frame after transformation are represented as follows:

[0092]

[0093] In the formula, γ t Represents the trainable translation parameters; s t,j This represents the human body coordinates before the transformation; s t ' ,j This represents the transformed human body coordinates; x′ t，j y′ t，j , z′ t，j These represent the X, Y, and Z coordinate values ​​of the transformed frame t and the j-th joint, respectively.

[0094] c. Furthermore, considering the modeling of spatiotemporal elements, the view adaptive transformation unit uses an AGC-LSTM network as the modeling layer to learn the aforementioned translation and rotation parameters. The AGC-LSTM network is a publicly available model structure, implemented by replacing the inputs, hidden states, and unit memories in a standard LSTM with a graphical structure.

[0095] d. Integrate the above view adaptive transformation method into a GCN-based network to output the optimal skeleton representation. Use GCN-based layers to model the translation and rotation processes. After the GCN-based layers, the translation vector and rotation matrix can be obtained. Both the translation vector and rotation matrix are obtained by multiplying the weights of the AGC-LSTM network by the features of the network's hidden layers, and adding the network's bias.

[0096] S102, after obtaining the optimal skeleton representation, a multi-scale feature extraction unit effectively learns the skeleton's features. Specifically:

[0097] a. The input to the multi-scale feature extraction unit is the human skeleton information output by the aforementioned view adaptive transformation unit. The human skeleton representation is divided into two scales to deeply explore the relationship between the skeleton and human movement. These two scales are... Figure 4 The text describes the human body in two parts: the upper part and the lower part, representing the joints and body parts. A diagram illustrating the joints and body parts is provided below. Figure 6 As shown. Figure 6 (a) The aim is to study the relationship between any joint and other joints. Figure 6 In (b), the human skeleton is divided into ten parts, aiming to study the relationships between each part and other parts. Skeletal features are mined at two scales using two branches. Each of these two branches includes a spatial feature modeling operator (HJ-GCN or HP-GCN) and a temporal feature modeling operator (T-HJ or T-HP). Therefore, the invented method enables the network to learn the features of human joints and human body parts separately.

[0098] The spatial feature modeling operator (HJ-GCN or HP-GCN) is an improvement based on the standard graph convolution (GCN). The specific improvement method is as follows:

[0099] b. The HJ-GCN operator is used to model the spatial features of the human joint scale (i.e., the first scale mentioned above). The specific method is as follows:

[0100] ① First, combine the i-th joint s i and the j-th joint s j The characteristics of the relationship between them are represented as f i,j The following formula is used to iteratively calculate and aggregate long-short features. That is, in the k-th iteration, using... and To generate Then use To calculate and

[0101]

[0102] f i,j =MLP([s i ,s j ])

[0103] In the formula, MLP(·) represents a multilayer perceptron that maps joint features, using the "Concat" function from the Pandas library to concatenate the features, f i,j s iand s j The characteristics between them.

[0104] After that, each joint s i Integrate with all previous information. The adjacency matrix A of the i-th joint and the j-th joint. i,j The following formula is used to calculate and explore the relationships between distant joints (such as the wrist and ankle).

[0105]

[0106] In the formula, T represents the transpose of the matrix; k is used for summation and counting, with a range of [1, J], where J represents the total number of human joints labeled in the dataset.

[0107] ②Furthermore, another adjacency matrix A is introduced. cp To capture the spatial relationships of joints that contribute significantly to the analysis of motion, the adjacency matrix is ​​represented by the following third-order polynomial. This process is used to uncover complex relationships between multiple joints.

[0108] A cp =A⊙W+A'⊙W'+A”⊙W”

[0109] In the formula, W is the corresponding multi-order trainable weight matrix; ⊙ is the element-based product operation; A is the graph convolution of the skeleton; A' represents the first-order polynomial of A; A” represents the second-order polynomial of A; W' represents the first-order polynomial of W; W” represents the second-order polynomial of W.

[0110] ③ Furthermore, the GCN operator shown in the following formula is used to mine the rich spatial features of human joints.

[0111]

[0112] In the formula, γ is the order; R(·) is the rule function; S' t,j W represents the input to the multi-scale feature extraction unit. i,j This represents the weight matrix.

[0113] c. The HP-GCN operator is used to model spatial features at the human body part scale (i.e., the second scale mentioned above). The specific method is as follows:

[0114] Similar to the HJ-GCN operator mentioned above, the adjacency matrix A is used. hp =diag(A)⊙W captures the relationships between body parts, where diag(·) is a diagonal transformation matrix. To avoid redundant features between body parts, high-order polynomials are not used in the HP-GCN operator. The output HP(S') of HP-GCN is obtained using the same principle as formulas ② and ③ in b above. t,j A).

[0115] d. The structure of the temporal feature modeling operator (T-HJ or T-HP) is "one classical convolution layer + one ReLU function". The temporal feature modeling operator is used to mine the temporal relationships between frames. The specific method is as follows:

[0116] ① First, the temporal features corresponding to HJ-GCN are modeled and denoted as "T-HJ". Its structure is as follows: Features along the temporal direction of the input sequence are extracted using standard one-dimensional convolution (Conv), and then a nonlinear transformation is performed using the ReLU activation function. The aim is to learn the temporal information gathered by HJ-GCN from each frame in the previous stage. The output of "T-HJ" is obtained from the following formula: T-HJ(S' t,j ).

[0117] T-HJ(S' t,j )=R(Conv[HJ(S' t,j );k,s])

[0118] In the formula, k and s represent the kernel size and stride, respectively.

[0119] ② Further, similar to the T-HJ operator mentioned above, the temporal feature modeling unit corresponding to the HP-GCN operator is denoted as "T-HP", which is used to learn the temporal information of human body parts in each frame aggregated by the HP-GCN operator mentioned above. The network features are output using the formula in d① above.

[0120] e. The aforementioned multi-scale feature fusion scheme is used to exchange hidden layer features between the two scales mentioned above, obtaining consistent multi-scale spatiotemporal features. Multi-scale features flow bidirectionally within each unit, as detailed below:

[0121] ① Based on the above definitions of human joints and human body parts, the features belonging to the same human body part are averaged. The "Concat" function is used to fuse the averaged features, and the feature is copied once to obtain the features at the human body part scale. Then, all features at the human joint scale are concatenated to output the final high-performance spatiotemporal features of the multi-scale feature extraction unit.

[0122] S103, Multi-task school unit such as Figure 5 As shown. This network aims to identify human behavior and predict human movement in parallel. Its steps are as follows:

[0123] a) To identify human behavior, an average pooling layer is used to convert the high-performance spatiotemporal features output by the multi-scale feature extraction unit into feature vectors. A standard Softmax layer is used as the classifier to output the classification result. The classification result is also used to assist in predicting human movement.

[0124] b. The steps of the aforementioned task of predicting human movement are as follows:

[0125] ① Use a standard graph convolution operator to re-extract the high-performance spatiotemporal features output by the MS-GCN above. Figure 5 (Below HJ-GCN) its output features are refined into three types of unified representations, including position X. p Speed ​​X v and acceleration X a The acceleration X mentioned a This method is used to perceive subtle motions in a sequence. L2 normalization is used to normalize the three outputs, resulting in a refined representation of the re-extracted t-th frame.

[0126]

[0127] ② Further, the motion of future frames is predicted sequentially along the time axis. The input to the motion prediction unit consists of the pose F(t) of frame t, and tags from action recognition The process involves extracting hidden features using the "HJ-GCN" operator and modeling temporal information using a gated recurrent unit (GRU). The output F(t+1) of the motion prediction for frame t+1 is obtained by the following formula:

[0128]

[0129] In the formula, With GCN operators It is the same structure. The input is S t ' ,j , The input is the pose F(t) of the t-th frame.

[0130] Figure 9 This is a diagram illustrating the motion prediction task of the present invention, where the first row represents the actual motion labels and the second row represents the prediction results of the present invention.

[0131] It is important to note that the motion prediction process of this invention does not incur additional computational costs. This is because the prediction of each subsequent frame utilizes features from the previous frame, making the overall prediction continuous and self-cyclical.

[0132] Furthermore, the entire network employs a self-supervised learning and training method. The principle is as follows:

[0133] ① To enable the view adaptive transformation unit to automatically learn trainable parameters and γ t Its loss function is set as follows:

[0134]

[0135] Based on the skeleton input, different transformation parameters are learned to generate different optimal representations for each frame. It is worth noting that these representations may come from different perspectives; they are merely abstract features.

[0136] ② For motion prediction, the loss function is set as follows:

[0137]

[0138] Where X and X represent the predicted frame and the ground frame, ||·|| F Let λ denote the Frobenius norm. λ is a weighted average used to determine the relative importance of the two tasks, ensuring that the predicted motion can be used for action recognition.

[0139] ③ For action recognition, the loss function is set as follows:

[0140]

[0141] Where Y and Y are the predicted label and the true label of the action, respectively, and f(·) is the cross-entropy function. It is a parameter that decreases linearly with iteration.

[0142] S104, based on the obtained behavioral recognition and motion prediction results, assists in the diagnostic screening and intervention of aggressive behaviors in children with ASD.

[0143] It is important to note that the aforementioned units, connected as shown in the diagram, form an end-to-end system for ASD child behavior recognition and motion prediction. During training, the behavior recognition and motion prediction tasks are performed separately with different parameter settings.

[0144] Furthermore, for the action recognition task, six identical spatiotemporal feature extraction blocks are used in the multi-scale feature extraction unit. Each extraction block contains the four units mentioned above: "HJ-GCN", "T-HJ", "HP-GCN", and "T-HP". The feature dimensions of the six spatiotemporal feature extraction blocks are set to 32, 64, 128, 128, 256, and 256, respectively. One-dimensional standard convolutions are used in the extraction blocks with a kernel size of 7, a stride of 2 for the second and fourth blocks, and a stride of 1 for the remaining blocks. The tradeoff weight λ of the self-supervised training scheme is set to 0.5. The model uses the standard Adam optimizer with an initial learning rate of 0.005 and a batch size of 128.

[0145] Furthermore, for the motion prediction task, four of the aforementioned spatiotemporal extraction blocks were used. The feature dimensions were 64, 64, 128, and 256, respectively. The weight λ was set to 0.8, and the stride sizes of the one-dimensional standard convolutions used in all spatiotemporal extraction blocks were set to 1, 1, 2, and 2, respectively. The model used the standard Adam optimizer with an initial learning rate of 0.0001 and a batch size of 32.

[0146] The advantages of this invention are:

[0147] The view-adaptive transformation unit of this invention integrates into a GCN-based framework. By automatically learning transformation parameters, it transforms the human body to an optimal unified viewpoint, thereby improving the representation of the 3D human skeleton. The effect of the view-adaptive transformation unit is as follows: Figure 7 As shown, this method overcomes the difficulties and variability in data collection for children with ASD in specific scenarios, saves computational costs, and indirectly improves the accuracy of diagnosis and intervention for children with ASD.

[0148] The multi-scale feature extraction unit of this invention decomposes the behavioral characteristics of children with ASD into two scales, combines the different contributions of the human skeletal structure in different movements, models the spatial features of the human body and the temporal features of consecutive frames, and uses a special fusion method to obtain consistent spatiotemporal features. These improvements increase the targeting of feature extraction to the behavior of children with ASD, and the high-performance features obtained directly improve the accuracy of diagnosis and reduce the error of the intervention system by approximately 0.6.

[0149] The multi-task learning unit and self-supervised learning scheme of this invention can simultaneously process action recognition and motion prediction tasks, and output the results of both tasks in parallel. High-performance features from the action recognition task can be used to predict motion, and features extracted from the motion prediction task can help understand the action.

[0150] The embodiments described above provide a detailed explanation of the technical solutions and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, additions, and equivalent substitutions made within the scope of the principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for autism children behavior recognition and motion prediction, characterized in that, Comprise: (1) acquire the original skeleton video data of children from different angles; (2) input the acquired original skeleton video data into a view adaptive conversion unit to adaptively convert to a consistent coordinate system to obtain the optimal representation of the human skeleton; (3) input the optimal representation of the human skeleton into a multi-scale feature extraction unit to extract high-performance spatiotemporal features of the human skeleton, including: (3-1) using a spatial feature modeling operator to extract the spatial features of the human joint scale and the human part scale, respectively, including: For human joint scales: joint th and joint th characteristics are represented as , long-short features are extracted using the following iterative calculation on : ； ； In the formula, The multi-layer perceptron representing the mapping of joint features uses the "Concat" function in the Pandas library to concatenate the features; The adjacency matrix of the first and second joints and the first and second joints is calculated by the following formula: ； In the formula, T represents the transpose symbol of a matrix; k For the sum count, the range is [1, J], and J represents the total number of labeled body joints in the data set. The adjacency matrix is calculated using the following formula : ； wherein is the corresponding multi-order trainable weight matrix; is an element-wise product operation; is a graph convolution of the skeleton; represents a first-order polynomial of represents a second-order polynomial of represents a first-order polynomial of represents a second-order polynomial of Finally, the spatial features of the human joint scale are obtained by the following formula: ； In the formula, is the order; is the Rule function; represents the input of the multi-scale feature extraction unit, represents a weight matrix; For human body part scale: use neighbor adjacency matrix Instead of adjacency matrix , is a diagonal transformation matrix, and other calculations are the same as the spatial feature extraction method of human body joint scale; (3-2) using a time feature modeling operator to extract the time features of the human joint scale and the human part scale, respectively, including: For the human joint scale, one-dimensional convolution is used to extract the features of the input sequence along the time direction, and then a ReLU activation function is used for nonlinear transformation to obtain the time features of the human joint scale, and the calculation formula is: ； In the formula, denotes the time characteristics of the human joint scale, and respectively represent the convolution kernel size and the step length; The time feature extraction method of the human part scale is the same as that of the human joint scale; (3-3) copy all columns of the human part scale feature once to make it match the size of the human joint scale feature; use the Concat function to concatenate the features of the human joint scale and the human part scale to obtain the final high-performance spatiotemporal features of the multi-scale feature extraction unit; (4) input the high-performance spatiotemporal features into a multi-task learning unit to perform behavior recognition and motion prediction on the human body to obtain behavior classification results and motion prediction results.

2. The method for autism children behavior recognition and motion prediction according to claim 1, characterized in that, Step (2) includes: (2-1) According to the rotation rule of the vector in the three-dimensional coordinate system, the rotation matrix is obtained : ； wherein denote a set of trainable parameters, are three angle parameters for rotations around the X, Y, Z axes, respectively; (2-2) translating and rotating the original skeleton video data according to the translation vector and the rotation matrix to obtain the optimal representation of the human skeleton; converted 1st frame 1st joint is represented as: ； wherein, represents a trainable translation parameter; represents a pre-transformed human coordinate representation; represents a post-transformed human coordinate representation; respectively represent the post-transformed X, Y, Z coordinates of the t frame, the j joint.

3. The method for autism children behavior recognition and motion prediction according to claim 1, characterized in that, Step (4) includes: (4-1) using an average pooling layer to convert the high-performance spatiotemporal features into a feature vector, using a Softmax layer as a classifier to output behavior classification results; (4-2) re-extract the high-performance spatio-temporal features using a graph convolution operator and refine the output features into three types of position , velocity and acceleration , normalize the three types of outputs, and obtain a refined representation of the re-extracted features of the first t frame : ； The first t+ 1 The output of the motion prediction of a frame is obtained from the equation: ； In the formula, represents the first t frame of the posture, represents the first t frame of the behavior classification result, and GRU represents a gated recurrent calculation.

4. An apparatus for autism child behavior recognition and motion prediction, characterized in that, The method for identifying and predicting the behavior of autistic children according to any one of claims 1-3, comprising: a view adaptive conversion unit for automatically learning conversion parameters and converting human data collected from different angles to a unified perspective to obtain the optimal representation of the human skeleton; a multi-scale feature extraction unit for extracting features from the optimal representation of the human skeleton to obtain high-performance spatiotemporal features; a multi-task learning unit for behavior recognition and motion prediction based on the high-performance spatiotemporal features to obtain behavior classification results and motion prediction results.

5. The device for behavior recognition and motion prediction of autistic children as claimed in claim 4, wherein, The view adaptive conversion unit uses an AGC-LSTM network to learn conversion parameters; the conversion parameters include a translation vector and a rotation matrix to convert the input original skeleton sequence to a unified perspective to obtain the optimal representation of the human skeleton.

6. The device for autism child behavior recognition and motion prediction according to claim 4, characterized in that, The multi-scale feature extraction unit includes a human joint spatiotemporal feature extraction module and a human part spatiotemporal feature module; both spatiotemporal feature extraction modules include a spatial feature modeling operator and a time feature modeling operator.

7. The device for behavior recognition and motion prediction of autistic children as claimed in claim 4, wherein The multi-task learning unit comprises a behavior recognition module and a motion prediction module; the behavior recognition module converts high-performance spatiotemporal features output by the multi-scale feature extraction unit into a feature vector by using an average pooling layer, uses a Softmax layer as a classifier, and outputs a behavior classification result; the motion prediction module reextracts the high-performance spatiotemporal features by using a graph convolution operator, obtains a fine representation of the reextracted features, takes a posture, the reextracted features, and a corresponding behavior classification result label as inputs, extracts hidden features, and then calculates by using a gated recurrent unit to obtain a motion prediction result of a next frame.

8. A diagnosis and treatment machine-assisted system for autism children behavior recognition and motion prediction, characterized in that, A computer storage, a computer processor, and a computer program stored in the computer storage and executable on the computer processor are used to perform the method for behavior recognition and motion prediction of autistic children according to any one of claims 1-3, the computer storage stores a view adaptive conversion unit, a multi-scale feature extraction unit, and a multi-task learning unit, and the computer processor realizes the following steps when executing the computer program: inputting acquired original skeleton video data of children at different angles into the view adaptive conversion unit to adaptively convert to an optimal representation of a human skeleton in a consistent coordinate system; inputting the optimal representation of the human skeleton into the multi-scale feature extraction unit to extract high-performance spatiotemporal features of the human skeleton; and inputting the high-performance spatiotemporal features into the multi-task learning unit to perform behavior recognition and motion prediction on the human body, and obtain a behavior classification result and a motion prediction result.

Images