Autistic child behavior recognition method based on micro-expression and motion behavior analysis

Abstract

The invention discloses an autistic child behavior recognition method based on micro-expression and motion behavior analysis, and the method comprises the steps: collecting a free activity video of a child through a camera, and carrying out the size transformation and pixel normalization of a video frame; extracting spatial-temporal characteristics of skeleton points of the whole body by adopting a cascaded differential convolution long-short-term memory unit, and generating skeleton point coordinates through a full connection layer; performing time sorting, space normalization and interpolation expansion on the skeleton point queue to enhance data continuity; and fusing expression and attitude features based on a lightweight ResNet18 network, and outputting a classification result and confidence in combination with a Softmax layer. According to the method, the subjective defect of a traditional method is overcome through a skeleton point detection technology, the objectivity, accuracy and real-time performance of autism behavior recognition are remarkably improved by combining spatial-temporal feature extraction and an efficient classification model, and the method is suitable for early screening and auxiliary diagnosis.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to a method, system and product for recognizing the behavior of children with autism, and particularly to a method, system and product for recognizing the behavior of children with autism based on micro-expression and motor behavior analysis. Background Technology

[0002] Autism spectrum disorder (ASD), a neurodevelopmental disorder, is characterized by core symptoms encompassing deficits in social interaction, atypical sensorimotor patterns, and restricted repetitive motor movements (SMMs). Traditional research has focused on stereotyped motor movements (SMMs) such as hand-clapping and body rocking, but the behavioral representations of children with ASD extend far beyond this. Neurodevelopmental abnormalities in facial expressions manifest as reduced eye contact and abnormal activation patterns of facial muscles (such as stiff expressions or inappropriate smiles in social situations). In the body movement dimension, there are motor coordination impairments, which may present as abnormal gait, difficulty maintaining posture, or spontaneous body twisting. Behavioral patterns involve complex characteristics such as narrow interests, impaired environmental adaptation, and ritualistic behaviors. Research by the World Health Organization has confirmed that these multidimensional representations are closely related to neuroplasticity, and early identification plays a decisive role in improving prognosis.

[0003] However, existing clinical diagnostic methods rely heavily on subjective assessment and manual observation, resulting in low efficiency, poor consistency, and difficulty in quantification. This is particularly true in the early screening stage before the age of two, where misdiagnosis and missed diagnosis rates remain high. Furthermore, certain facial expressions or body postures of autistic patients often only appear in specific situations, making them difficult to observe manually in a timely manner.

[0004] To address the inefficiencies and low accuracy of traditional methods, current research is increasingly incorporating artificial intelligence, particularly computer vision and deep learning technologies, to automatically recognize facial expressions, body movements, and behavioral patterns in children with ASD. For example, some existing technologies use convolutional neural networks (CNNs) to extract posture and behavioral information from video image sequences, and utilize long short-term memory networks (LSTMs) or attention mechanisms to model the temporal series of behaviors to determine the presence of stereotyped behaviors (SMMs). A representative study is the "Autism Stereotyped Behavior Detection System Based on Posture Recognition," which uses pose estimation technology to track and analyze upper limb movement trajectories to detect whether children with ASD exhibit clapping, shaking, or other similar behaviors.

[0005] Despite the progress made by the above technologies in automated recognition, the following key technical defects still exist in practical applications: poor recognition accuracy and robustness, reliance on high-quality large datasets, lack of real-time performance and interpretability, and limited motion capture dimensions. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention proposes a method, system, and product for behavioral recognition of autistic children based on micro-expression and motor behavior analysis. It can automatically capture and classify children's micro-expression and movement information, and also has high recognition accuracy and high real-time processing efficiency.

[0007] The technical solution adopted by the present invention is: a method for behavioral recognition of autistic children based on micro-expression and motor behavior analysis, comprising the following steps: Step 1: Obtain a video of free movement featuring only the child being tested; Step 2: Perform image frame preprocessing on each frame of the video, including resizing the image frame and normalizing the pixel values ​​of the image frame; Step 3: Based on the video frame after the image frame preprocessing, the skeleton point extraction module is used to detect and extract the two-dimensional spatial coordinates of the whole body skeleton points of the child under test, including facial skeleton points and body skeleton points. Step 4: Perform bone point processing on the two-dimensional spatial coordinates of the whole body skeleton points extracted from each frame of the video, including forming a bone point queue according to the video frame time order, performing spatial normalization on the bone points in the bone point queue, and performing interpolation processing on the bone point queue to obtain a longer extended bone point queue. Step 5: Based on the expanded skeletal point queue, classify and identify the children to be tested.

[0008] Preferably, the skeleton point extraction module includes four cascaded differential convolutional long short-term memory units, a Flatten layer, and two cascaded fully connected layers; The four cascaded differential convolutional long short-term memory units are each composed of a DC-LSTM layer, a BN layer, a ReLU layer, and a Maxpool layer connected in sequence. The Flatten layer is used to flatten the spatiotemporal features extracted by the DC-LSTM layer into a one-dimensional vector, and then the two cascaded fully connected layers perform nonlinear transformation and abstraction on these high-dimensional, integrated spatiotemporal features. The two cascaded fully connected layers consist of two layers. The first layer contains 128 hidden neurons that transform abstract features through nonlinear transformation. The second layer inputs the features into a skeletal point prediction head and finally generates multiple outputs, representing the coordinates of various skeletal points throughout the child's body.

[0009] Preferably, the DC-LSTM layer consists of an input gate i, a forget gate f, a cell gate g, an output gate o, and a storage cell state C; gates i, f, and g control the update of the cell state C, while gate o determines the proportion of cell memory transmitted to the hidden state output H. For the input video at time t, its working principle is as follows: ; Among them, X t Represents the input video frame tensor, ΔH t-1 Ct represents the difference between the hidden states at time t-1 and time t-2, and Ct represents the storage unit tensor. and Each of these represents a weight kernel tensor for the corresponding gate k=i, f, g, o, used to perform the convolution operation. These represent the bias terms for the corresponding gates k=i, f, g, and o, respectively; ⊙ and ⊙ represent the convolution operation and the Hadamard product, respectively; This represents the Sigmoid activation function.

[0010] As a preferred option

[0011] Among them, threshold Used to promote increased sparsity, and to flexibly control the sparsity of DC-LSTM modules.

[0012] Preferably, in step 3, each frame of the child's entire body is broken down into individual skeletal points and stored in chronological order in a fixed length. The skeleton point queue, in which This represents the sampling window length of the input video frame sequence; the length is always [value missing]. The skeleton point queue is spatially normalized so that the coordinates of the topmost skeleton point remain at the center of the image, thus making the overall spatial distribution of the skeleton point queue closer to the center of the image; then the skeleton point queue is interpolated to obtain a longer extended skeleton point queue.

[0013] As a preferred embodiment, step 4 includes the following sub-steps: Step 4.1: Extract facial expression features from facial bone points and body features from body bone points; Step 4.2: Perform feature fusion on facial expression features and body features, and then input them into the autism classifier for autism classification.

[0014] Preferably, in step 4.2, the autism classifier consists of a Conv layer, a BN layer, a Conv layer, and a Softmax layer connected in sequence; the final output is the classification result of autism and its corresponding confidence level.

[0015] This invention also provides a behavior recognition system for children with autism based on micro-expression and motor behavior analysis, comprising: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the aforementioned method for behavioral recognition of autistic children based on micro-expression and motor behavior analysis.

[0016] The present invention also provides a behavior recognition product for children with autism based on micro-expression and motor behavior analysis, including computer program instructions, which, when the computer program instructions are run on a computer, cause the computer to execute the behavior recognition method for children with autism based on micro-expression and motor behavior analysis.

[0017] The present invention also provides a non-volatile computer-readable storage medium containing a computer program that, when executed by one or more processors, causes the processors to perform the aforementioned method for behavioral recognition of autistic children based on micro-expression and motor behavior analysis.

[0018] Compared to existing technologies, the benefits of this invention include: (1) This invention, by employing skeletal point detection technology, can automatically extract and analyze the two-dimensional spatial coordinates of facial and trunk skeletal points during a child's free movement, avoiding errors caused by subjective assessment and manual observation in traditional diagnostic methods. This method not only improves the objectivity of diagnosis, but also improves the accuracy of diagnosis because the skeletal point data only contains joint coordinate information and is not affected by environmental factors such as lighting, background, and clothing. (2) The method of the present invention can process video data in real time and quickly output classification results and their corresponding confidence scores, which greatly shortens the time required for diagnosis. Compared with traditional methods that rely on parent reports or manual annotation of video clips, the present invention greatly improves diagnostic efficiency and makes large-scale screening possible; (3) This invention, by designing a skeletal point extraction module comprising four cascaded differential convolutional long short-term memory (DC-LSTM) units and two cascaded fully connected layers, fully considers the spatiotemporal characteristics of video frames and achieves multi-level feature extraction and fusion. This design not only captures the spatial location information of skeletal points but also captures their dynamic features that change over time, thereby enhancing the expressive power of features and improving the accuracy of behavior recognition; (4) The interpolation and spatial normalization techniques used in the skeleton point processing stage and the lightweight ResNet18 network structure used in the classification and recognition stage of this invention make the method of this invention more efficient and practical in practical applications. Attached Figure Description

[0019] To more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of the basic process of the method provided in the embodiments of the present invention; Figure 2 The skeleton point extraction network based on the DC-LSTM module provided in this embodiment of the invention; Figure 3 This is a structural diagram of a DC-LSTM module provided in an embodiment of the present invention; Figure 4 This is a skeletal point feature extraction and autism classification network diagram provided in an embodiment of the present invention; Figure 5 This diagram illustrates the classification results of children's behavioral patterns within videos and the corresponding confidence levels in an experiment according to an embodiment of the present invention. Detailed Implementation

[0021] To better understand the technical content and advantages of this invention, the following detailed description, in conjunction with the accompanying drawings and embodiments, provides a method for identifying autistic behaviors based on skeletal point detection. The specific implementation methods, structures, features, and objectives are detailed below. Furthermore, specific features, structures, or characteristics in one or more embodiments can be combined in any suitable form.

[0022] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. Terms such as “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a circuit structure, article, or device comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such article or device. Without further limitation, an element defined by the phrase “comprising one…” does not exclude the presence of additional identical elements in the article or device that includes the element.

[0023] The following will provide a detailed description of an autism behavior recognition method based on skeletal point detection provided in this embodiment, with reference to the accompanying drawings.

[0024] like Figure 1 and Figure 4 As shown, this invention relates to a method for identifying autistic behaviors based on skeletal point detection, specifically including the following steps: Step 1: Video input.

[0025] First, acquire video footage of the child's free movement captured by a camera. Specifically, obtain 5 minutes of free movement video from a fixed position using a surveillance camera in a kindergarten classroom or home. Strictly speaking, the camera must only capture the child in its field of view, showing all parts of the child's body, and the child's behavior must not be intentionally guided by a guardian or external environment. The video frame rate must be at least 30 fps to analyze and extract complete facial expression and posture skeletal point information.

[0026] Step 2: Image frame preprocessing.

[0027] In the autism behavior recognition method based on skeletal point detection, image frame preprocessing for each frame of the video is a crucial step. This step includes resizing the image frames and normalizing the pixel values ​​of the image frames.

[0028] Step 2.1: Image frame size transformation.

[0029] Since each frame in a video may have a different size, this invention first performs a uniform transformation on the image frames to ensure consistency and efficiency in subsequent processing. Specifically, a standard size can be selected (e.g., 640 pixels wide and 480 pixels high), and the original image frames are then transformed to this standard size using a scaling algorithm (such as bilinear interpolation, nearest neighbor interpolation, etc.). This process ensures that all image frames have the same size in subsequent processing, thus facilitating subsequent skeleton point detection and behavior recognition.

[0030] Step 2.2: Normalize image frame pixel values.

[0031] The pixel values ​​of an image frame typically vary within a large range (e.g., 0-255), which can interfere with subsequent skeleton point detection algorithms. To eliminate this effect, this invention normalizes the pixel values ​​of the image frame. Specifically, each pixel value of the image frame can be divided by 255, thereby transforming the pixel value range to between 0 and 1. This process helps reduce the impact of pixel value variations on the skeleton point detection algorithm, improving the algorithm's accuracy and stability.

[0032] Step 3: Extract all-body skeletal points, including facial and body skeletal points.

[0033] In autism behavior recognition methods based on skeletal point detection, the extraction of skeletal points throughout the body is one of the core steps.

[0034] In one implementation, such as Figure 2 As shown, a meticulously designed skeletal point extraction module is employed, comprising four cascaded Differential Convolutional Long Short-Term Memory (DC-LSTM) units and two cascaded fully connected layers (FC). Each of the four cascaded DC-LSTM units consists of a sequentially connected DC-LSTM layer, a BN layer, a ReLU layer, and a Maxpool layer. Taking full account of the spatiotemporal characteristics of the video frames after image frame preprocessing, the module achieves spatiotemporal feature extraction of the entire skeletal points of the child under test through multi-level feature extraction and fusion. Two cascaded fully connected layers (FC) first flatten the spatiotemporal features extracted by the DC-LSTM into one-dimensional vectors using a flattened layer. Then, the two cascaded FC layers perform nonlinear transformations and abstractions on these high-dimensional, integrated spatiotemporal features. The first cascaded FC layer contains 128 hidden neurons and transforms the abstract features through nonlinear transformations. The second layer inputs the features into a skeletal point prediction head, ultimately generating multiple outputs representing the coordinates of various skeletal points throughout the child's body.

[0035] The skeletal point extraction module contains approximately 420,000 trainable parameters. It fully considers the spatiotemporal characteristics of the video frames after the image frames are preprocessed. Through multi-level feature extraction and fusion, it realizes the spatiotemporal feature extraction of the whole body skeletal points of the child to be tested.

[0036] In one implementation, such as Figure 3As shown, the DC-LSTM unit is responsible for capturing the spatiotemporal dependencies in the video frames after image frame preprocessing and for extracting features from the whole-body skeletal points of the child being tested. Each DC-LSTM unit uses a 3×3 convolutional kernel and is configured with 8, 16, 32, and 64 hidden nodes respectively to progressively extract multi-level features from the input data. This hierarchical structure design enables the network to progressively learn spatiotemporal features from low to high layers, thereby enhancing the model's expressive power. At the output of each DC-LSTM unit, batch normalization is first applied to accelerate network convergence and improve training stability, followed by the introduction of nonlinearity through the ReLU (Rectified Linear Unit) activation function. To reduce the spatial dimension of the feature map and enhance the translation invariance of the features, the output of each module is also downsampled through a max-pooling layer.

[0037] In its implementation, a DC-LSTM cell consists of an input gate i, a forget gate f, a cell gate g, an output gate o, and a storage cell state C. In this structure, gates i, f, and g control the updating of the cell state C, while gate o determines the proportion of cell memory that is transferred to the hidden state output H.

[0038] While traditional ConvLSTM excels at capturing spatiotemporal dependencies, its computational cost remains high, especially when processing long video frames. However, the inherent sparsity of long-duration video footage of children's free movement presents a significant opportunity to reduce the computational cost associated with network processing. A thorough analysis of each network stage leads to an important conclusion: convolution operations constitute the largest computational expense in the ConvLSTM architecture. For a single ConvLSTM unit, the input consists of two elements: the input tensor X at the current time step. t And the hidden state H from the previous moment t-1 The first input part, Xt, is based on video frames and the output from the previous time step after ReLU activation, naturally exhibiting sparsity. In contrast, the second input part, H... t-1 Primarily dense. Based on the above analysis, DC-LSTM aims to further optimize computation by introducing sparsity into the hidden states. Unlike traditional ConvLSTM, this invention does not directly use H... t-1 Instead, it utilizes H t-1 and H t-2 The variation between these values ​​serves as a cyclic input feature. Furthermore, this embodiment sets a threshold for this variation. This promotes increased sparsity and facilitates flexible control of the sparsity of the DC-LSTM module. Based on a threshold... changing The formula for Ht-1 is as follows:

[0039] The specific DC-LSTM unit can be expressed by the formula:

[0040] Among them, ⊙ and ⊙ represent the convolution operation and the Hadamard product, respectively. X t Represents the input video frame tensor, ΔH t-1 Ct represents the difference between the hidden states at time t-1 and time t-2, and Ct represents the storage unit tensor. and Each of these represents a weight kernel tensor for the corresponding gate k=i, f, g, o, used to perform the convolution operation. These represent the bias terms for the corresponding gates k=i, f, g, and o, respectively; ⊙ and ⊙ represent the convolution operation and the Hadamard product, respectively; This represents the Sigmoid activation function. It is introduced by... H t-1 As module input and intermediate variable, and based on threshold By locally zeroing out the components, this modification enables high temporal sparsity in convolutional operations. It avoids the need to accumulate previous matrix-vector multiplication results by employing an incremental encoder to process the hidden paths, thus distinguishing this method from traditional LSTM-based network structures. Consequently, the computational and memory overhead associated with inducing and utilizing temporal sparsity is significantly reduced.

[0041] After processing by four DC-LSTM units, the spatiotemporal features of the video frame are passed to two cascaded fully connected layers for further processing. The first fully connected layer contains 128 hidden neurons, which maps high-dimensional features to low-dimensional space through nonlinear transformations (such as the ReLU activation function) to reduce the dimensionality of the features and extract key information.

[0042] The second fully connected layer generates multiple outputs, each representing candidate coordinates of various skeletal points throughout the child's body. Notably, the operations in the fully connected layer are repeated L times, where L represents the sampling window length of the input video frame sequence. This repetitive execution mechanism ensures that the network can effectively process time-series data, thereby more accurately extracting the coordinates of the child's skeletal points from the video frames.

[0043] Step 4: Skeletal point processing.

[0044] In autism behavior recognition methods based on skeletal point detection, skeletal point processing of the two-dimensional spatial coordinates of the whole-body skeletal points extracted from each frame of a video is a crucial step. This invention proposes an effective skeletal point processing scheme, which includes assembling a skeletal point queue according to the temporal order of video frames, splitting the skeletal points, spatially normalizing the skeletal point queue, and interpolating the skeletal point queue to obtain a longer extended skeletal point queue. Specific implementation includes: Step 4.1: Construct the skeleton point queue.

[0045] The two-dimensional spatial coordinates of each skeletal point of the child's entire body extracted from each video frame are stored in a skeletal point queue of fixed length L in chronological order. Here, L represents the sampling window length of the input video frame sequence, which determines the number of video frames that the skeletal point queue can contain. By storing the skeletal point coordinates in chronological order, the temporal information between video frames can be preserved, which is crucial for subsequent behavior recognition.

[0046] Step 4.2: Skeletal Point Decomposition The skeletal points are split into facial skeletal points and body skeletal points based on the region to which they belong.

[0047] Step 4.3: Spatial normalization.

[0048] To eliminate spatial discrepancies in the coordinates of skeletal points and improve the accuracy of behavior recognition, this invention performs spatial normalization on the skeletal point queue. Specifically, the topmost skeletal point in the queue (such as a head skeletal point) can be selected as a reference point, and its coordinates are adjusted to align with the center of the image. Then, the coordinates of other skeletal points are adjusted accordingly based on the position of this reference point, making the spatial distribution of the entire skeletal point queue closer to the image center. This spatial normalization helps reduce the impact of environmental factors such as lighting and shooting angle on behavior recognition.

[0049] Step 4.4: Interpolation processing of the skeleton point queue.

[0050] To increase the temporal density of skeletal point data and improve the precision and continuity of behavior recognition, this invention performs temporal interpolation processing on the skeletal point queue. Specifically, this invention employs linear interpolation, quadratic interpolation, or other suitable interpolation methods to insert new skeletal point frames between adjacent skeletal point frames. The coordinates of these newly inserted skeletal point frames are obtained through interpolation calculations, and they fill the time intervals between the original skeletal point frames, thus making the skeletal point data more continuous and dense in time. This method yields a longer extended skeletal point queue. This extended skeletal point queue not only contains information from the original skeletal points but also information from the new skeletal points obtained through temporal interpolation, thereby improving the precision and accuracy of behavior recognition.

[0051] Step 5: Feature extraction, fusion, and classification.

[0052] In autism behavior recognition methods based on skeletal point detection, feature extraction and classification are key steps in ultimately determining the behavioral patterns of the children being tested.

[0053] In one implementation, an efficient and accurate feature extraction and classification scheme is adopted. This scheme is based on an expanded skeleton point queue, uses a ResNet18 network to extract facial expression features from facial skeleton points and behavioral features from body skeleton points, then performs feature fusion, and adds a softmax layer at the end for classification.

[0054] ResNet18 is a deep convolutional neural network that excels in image classification, object detection, and other fields. This invention chooses ResNet18 as the backbone network for feature extraction primarily because of its strong feature extraction capabilities and fast inference speed. This invention uses an expanded skeletal point queue as input to ResNet18. This queue contains the coordinates of the child's entire skeletal structure in video frames, and after interpolation in the temporal dimension, it exhibits higher temporal density and continuity.

[0055] In the ResNet18 network, the input data first undergoes a series of convolutional and pooling layers for feature extraction. These convolutional and pooling layers learn the spatial and temporal features of the skeletal point data and gradually refine deeper feature representations. Through multiple layers of convolution and pooling operations, a highly abstract and robust feature map is obtained.

[0056] To classify the feature maps, this invention adds a softmax layer at the end of the ResNet18 network. The softmax layer is a commonly used classifier that maps feature maps to class probability distributions, thereby classifying different behavioral patterns. During training, the parameters of the softmax layer are optimized by minimizing the cross-entropy loss function, enabling it to more accurately classify the behavioral patterns of the children being tested.

[0057] After feature extraction and classification using the ResNet18 network and softmax layer, the classification results of the child's behavioral patterns and their corresponding confidence scores were obtained. These classification results and confidence scores are of great significance to doctors and researchers, serving as a reference for early screening and diagnosis of autism.

[0058] The invention will be further illustrated below through specific experiments.

[0059] Please see Figure 5 The system takes a video recording of a child's behavior as input, processes it, and overlays predicted skeletal points and micro-expression features onto the original video surface. It outputs the classification results of the child's behavioral patterns within the video, along with the corresponding confidence scores. A graph is plotted with the timestamp on the x-axis and the confidence score on the y-axis, showing the corresponding video frame below each timestamp. The output results allow for real-time assessment of the risk level of the child's behavior, enabling targeted preventative measures.

[0060] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as ROM / RAM, disk, optical disk, etc.

[0061] The above description discloses only preferred embodiments of the present invention and should not be construed as limiting the scope of the present invention. Therefore, equivalent variations made in accordance with the claims of the present invention are still within the scope of the present invention.

[0062] It should be noted that embodiments of the present invention can be implemented using hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, with such code provided on a programmable memory or a data carrier such as an optical or electronic signal carrier.

[0063] Furthermore, although the operation of the method of the present invention is described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowcharts may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps. It should also be noted that the features and functions of two or more devices according to the present invention can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0064] While the invention has been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A method for identifying the behavior of autistic children based on micro-expression and motion behavior analysis, characterized in that, The method comprises the following steps: Step 1: obtaining a free activity video containing only one object of the child to be tested; Step 2: performing image frame preprocessing on each frame of the video, including size transformation of the image frame and normalization of the pixel value of the image frame; Step 3: based on the pre-processed video frame, using a skeleton point extraction module to detect and extract the two-dimensional space coordinates of the whole body skeleton points of the child to be tested, including face skeleton points and body skeleton points; Step 4: performing skeleton point processing on the two-dimensional space coordinates of the whole body skeleton points extracted in each frame of the video, including forming a skeleton point queue in the order of video frame time, spatial normalization of the skeleton points in the skeleton point queue, and interpolation processing of the skeleton point queue to obtain an extended skeleton point queue with longer length; Step 5: based on the extended skeleton point queue, performing classification and recognition on the child to be tested.

2. The micro-expression and motion behavior analysis based autism child behavior recognition method according to claim 1, characterized in that, The skeleton point extraction module comprises four cascaded differential convolution long short-term memory units, a Flatten layer and two cascaded fully connected layers; The four cascaded differential convolution long short-term memory units are each composed of a DC-LSTM layer, a BN layer, a ReLU layer and a Maxpool layer connected in sequence; The Flatten layer is used to flatten the spatio-temporal features extracted by the DC-LSTM layer into a one-dimensional vector, and then the two cascaded fully connected layers are used to perform nonlinear transformation and abstraction on these high-dimensional integrated spatio-temporal features; The two cascaded fully connected layers, the first layer contains 128 hidden neurons, which transforms the abstract features through nonlinear transformation; the second layer inputs the features into a skeleton point prediction head, and finally generates multiple outputs representing the coordinates of each skeleton point of the child to be tested.

3. The micro-expression and motion behavior analysis based autism child behavior recognition method of claim 1, wherein: The DC-LSTM layer is composed of an input gate i, a forgetting gate f, a cell gate g, an output gate o and a cell state C; the gates i, f and g control the update of the cell state C, and the gate o determines the proportion of the cell memory transmitted to the hidden state output H; For the input video at time t, the working principle is as follows: ； where X t represents the input video frame tensor, ΔH t-1 represents the difference between the hidden states at time t-1 and t-2, and Ct represents the storage cell tensor; and each represent a weight kernel tensor corresponding to a gate k = i, f, g, o, respectively, for performing a convolution operation, each represent a bias term corresponding to a gate k = i, f, g, o, respectively; and ⊙ represent a convolution operation and a Hadamard product, respectively; represents a Sigmoid activation function.

4. The autism child behavior recognition method based on micro-expression and motion behavior analysis according to claim 3, characterized in that: wherein the threshold value For promoting the increase of sparsity, for flexibly controlling the sparsity of the DC-LSTM module.

5. The micro-expression and motion behavior analysis-based autism child behavior recognition method according to claim 1, wherein in step 3, each bone point of the child's whole body in each frame is split and stored in a bone point queue with a fixed length of in time sequence, wherein represents a sampling window length of the input video frame sequence; the bone point queue with the length of is spatially normalized so that the coordinates of the topmost bone point remain the center of the image, and thus the overall spatial distribution of the bone point queue is closer to the center of the image; and the bone point queue is subjected to interpolation processing to obtain an extended bone point queue with a longer length.

6. The autism child behavior recognition method based on micro-expression and motion behavior analysis according to any one of claims 1-5: the specific implementation of step 4 comprises the following sub-steps: Step 4.1: extracting expression features of face skeleton points and body features of body skeleton points; Step 4.2: fusing the expression features and body features, and then inputting an autism classifier for autism classification and recognition.

7. The autism child behavior recognition method based on micro-expression and motion behavior analysis according to claim 6: in step 4.2, the autism classifier is composed of a Conv layer, a BN layer, a Conv layer and a Softmax layer connected in sequence; and finally outputs the classification result of autism and the corresponding confidence.

8. A system for recognizing behavior of autistic children based on micro-expression and motion behavior analysis, characterized in that, comprise: one or more processors; A storage device configured to store one or more programs, which when executed by one or more processors, cause the one or more processors to implement the micro-expression and motion behavior analysis based autism child behavior recognition method according to any one of claims 1 to 7.

9. An autism child behavior recognition product based on micro-expression and motion behavior analysis, comprising computer program instructions, characterized in that: The computer program instructions, when running on a computer, cause the computer to perform the micro-expression and motion behavior analysis based autism child behavior recognition method according to any one of claims 1 to 7.

10. A non-transitory computer readable storage medium embodying a computer program, comprising: The computer program instructions, when running on a computer, cause the computer to perform the micro-expression and motion behavior analysis based autism child behavior recognition method according to any one of claims 1 to 7.

Images