Data-driven methods, systems, and media for identifying autistic behaviors in children.

Abstract

This application relates to the field of data processing technology, and discloses a method, system, and medium for identifying autistic behaviors in children based on data analysis. The method includes: collecting and preprocessing multimodal behavioral data of children; extracting nonlinear features to obtain a time-series feature matrix; identifying repetitive behavioral patterns using a three-dimensional convolutional network; analyzing temporal changes in behavior using a temporal convolutional network; fusing multiple feature representations to obtain a comprehensive feature vector; and applying a multi-classifier system to identify autistic behavior types and assess severity. By extracting nonlinear time-series features, constructing a deep spatiotemporal feature extraction network, designing a feature fusion mechanism, and integrating a multi-classifier system, this invention overcomes the limitations of existing technologies involving single modality, linear features, and single algorithms, thereby improving the accuracy and interpretability of autistic behavior identification.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, system and medium for identifying autistic behaviors in children based on data analysis. Background Technology

[0002] Autism spectrum disorder (ASD) is a neurodevelopmental disorder characterized by impairments in social interaction, repetitive and stereotyped behaviors, and restricted interests. Currently, the diagnosis of autism primarily relies on professional physicians through behavioral observation and standardized assessment scales, such as the Childhood Autism Rating Scale (CARS) and the Diagnostic Observational Scale for Autism (ADOS). These traditional assessment methods heavily depend on the subjective judgment of clinical experts and typically involve short-term observations in structured environments, making it difficult to comprehensively capture children's behavioral characteristics in natural settings.

[0003] In recent years, with the development of sensing technology and artificial intelligence, researchers have begun to explore using multimodal technologies to collect behavioral data on children with autism and combine it with machine learning algorithms for automated identification. These methods include computer vision-based action recognition, wearable device-based motion pattern analysis, and audio signal-based speech communication detection, providing new technological approaches for the objective quantification of autistic behaviors. Existing wearable device technologies, such as the six-axis sensors in smartwatches, can monitor children's movement status, positional changes, and basic physiological indicators in real time, providing continuous data support for behavioral analysis. Simultaneously, multi-target video surveillance technology in classroom environments is increasingly being applied to group behavior analysis, enabling the simultaneous observation of interaction patterns and individual behavioral differences among multiple children.

[0004] However, existing technologies still have many shortcomings. First, most studies focus only on single-modal data analysis, such as analyzing only video data or only sensor data, making it difficult to comprehensively capture the multidimensional characteristics of autistic behaviors, especially failing to effectively integrate the complementary information from smartwatch sensor data and video data. Second, existing feature extraction methods mostly focus on linear statistical features, such as behavior frequency and average movement intensity, ignoring the nonlinear dynamic characteristics and complex temporal patterns that may exist in autistic behaviors. Third, existing classification models often use a single algorithm, such as using only convolutional neural networks or only support vector machines, which are difficult to adapt to the complex characteristics of different types of autistic behaviors and lack the comprehensive recognition ability for diverse behavioral patterns such as stereotyped movements, gait abnormalities, and positional preferences. Fourth, most studies only focus on the classification and identification of behaviors, lacking quantitative assessment of the severity of behaviors, making it difficult to provide refined guidance for clinical intervention. Fifth, when dealing with individual behavior identification in group environments, existing methods often cannot effectively handle complex scenarios with simultaneous monitoring of multiple targets, making it difficult to achieve large-scale screening applications in natural educational environments such as kindergartens and primary schools. Finally, existing methods typically employ simple sliding windows or statistical features when processing time-series data, failing to fully uncover long-term dependencies and state transition features in behavioral sequences. In particular, they cannot effectively analyze motion trajectory patterns and repetitive behavioral features in smartwatch sensor data. Summary of the Invention

[0005] This application provides a data analysis-based method, system, and medium for identifying autistic behaviors in children. It is used to extract nonlinear time series features, construct a deep spatiotemporal feature extraction network, design a feature fusion mechanism, and integrate a multi-classifier system. This invention can overcome the limitations of single modality, linear features, and single algorithm in the prior art, and improve the accuracy and interpretability of autistic behavior identification.

[0006] In a first aspect, this application provides a data analysis-based method for identifying autistic behaviors in children. The method includes: multimodal acquisition and preprocessing of children's behavioral data to obtain a multimodal dataset, wherein the multimodal acquisition includes acquiring six-axis sensor data, positioning data, and audio data through a smartwatch worn by the child, and simultaneously acquiring video data from multiple children through multi-target monitoring cameras deployed in the classroom; extracting nonlinear features from the multimodal dataset to obtain a behavioral time-series feature matrix; inputting the multimodal dataset into a three-dimensional convolutional neural network for processing to identify repetitive behavioral patterns and stereotyped actions, obtaining a spatial behavioral feature representation; constructing a temporal convolutional network model based on the multimodal dataset to analyze behavioral temporal changes and state transition features, obtaining a behavioral dynamic feature vector; fusing the behavioral time-series feature matrix, the spatial behavioral feature representation, and the behavioral dynamic feature vector, and obtaining a comprehensive feature vector through feature concatenation and autoencoder dimensionality reduction; and inputting the comprehensive feature vector into a multi-classifier for autism behavior type identification and severity assessment to obtain behavioral identification data.

[0007] Secondly, this application provides a data analysis-based system for recognizing autistic behaviors in children, the data analysis-based system for recognizing autistic behaviors in children comprising:

[0008] The data acquisition module is used to collect six-axis sensor data, positioning data and audio data of children through a smartwatch, as well as to collect video data of multiple children simultaneously through a multi-target monitoring camera in the classroom, to obtain a multimodal dataset;

[0009] The extraction module is used to extract nonlinear features from the multimodal dataset to obtain a behavioral time series feature matrix;

[0010] The processing module is used to input the multimodal dataset into a three-dimensional convolutional neural network for processing, identify repetitive behavior patterns and stereotyped actions, and obtain spatial behavior feature representations.

[0011] The construction module is used to build a temporal convolutional network model based on the multimodal dataset, analyze the temporal changes and state transition features of behavior, and obtain dynamic feature vectors of behavior.

[0012] The fusion module is used to fuse the behavior time series feature matrix, the spatial behavior feature representation, and the behavior dynamic feature vector, and obtain a comprehensive feature vector through feature concatenation and dimensionality reduction by an autoencoder.

[0013] The identification module is used to input the comprehensive feature vector into a multi-classifier to identify autism behavior types and assess severity, thereby obtaining behavior identification data.

[0014] Thirdly, a data analysis-based device for recognizing autistic behaviors in children is provided, comprising: a memory and at least one processor, wherein the memory stores instructions; the at least one processor invokes the instructions in the memory to cause the data analysis-based device for recognizing autistic behaviors in children to execute the aforementioned data analysis-based method for recognizing autistic behaviors in children.

[0015] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored therein, which, when executed on a computer, cause the computer to perform the aforementioned data analysis-based method for identifying autistic behaviors in children.

[0016] The technical solution provided in this application effectively solves the technical problem of incomplete information from a single data source in traditional autism diagnosis by using a multimodal data acquisition method that combines multi-sensor fusion acquisition from a smartwatch with multi-target video monitoring in a classroom. The smartwatch's six-axis sensor can continuously monitor the child's subtle changes in movement and motion patterns, the positioning module records the repetitive characteristics of spatial movement trajectories, the audio sensor captures speech communication patterns and responses to sound stimuli, and the classroom camera enables simultaneous monitoring of multiple children, breaking through the limitations of traditional methods that can only observe individually. The time-synchronous processing of multimodal data ensures the consistency and integrity of information from different sensors. Nonlinear feature extraction, through fractal analysis, long-range correlation analysis, and dynamic instability analysis, delves into the complexity and irregularity of the time series of behaviors in children with autism, overcoming the technical shortcomings of traditional linear statistical features that cannot reflect the essence of behavior. The dilated convolution design and temporal attention mechanism of the three-dimensional convolutional neural network effectively identify the spatiotemporal characteristics of repetitive behavioral patterns and stereotyped actions. The residual connections and causal convolutional structure of the temporal convolutional network accurately analyze the temporal changes and state transitions of behavior. Feature fusion compresses multidimensional features into a compact representation through autoencoder dimensionality reduction technology. The multi-classifier integration system fully leverages the complementary advantages of different algorithms.

[0017] Fractal analysis algorithms accurately quantify the chaotic characteristics of behavioral patterns in children with autism by measuring the complexity and irregularity of behavioral data, providing a mathematical basis for identifying stereotyped and repetitive behaviors. The Hearst exponent calculated by the rescaled range method accurately reflects the long-term memory characteristics of behavioral sequences, effectively distinguishing the behavioral persistence differences between children with autism and typical developmental children. The maximum Lyapunov exponent algorithm quantifies the instability of behavioral systems, providing a scientific basis for assessing behavioral predictability. The dilated 3D convolutional network achieves multi-scale spatiotemporal feature extraction through an incremental dilation rate design. The temporal attention mechanism automatically identifies key behavioral segments with repetition and regularity. The dilated convolutional structure of the residual temporal convolutional network efficiently captures multi-level temporal dependencies from the short to the long term. The causal convolutional design ensures the causal constraints of temporal prediction. The symmetric structure of the autoencoder achieves effective feature compression and denoising through unsupervised learning. The multi-classifier weighted voting mechanism combines the robustness of random forests, the generalization ability of support vector machines, the accuracy of gradient boosting trees, and the nonlinear modeling ability of neural networks. The synergistic effect of these algorithmic features significantly improves the accuracy, robustness, and interpretability of autism behavior recognition. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of an embodiment of the data analysis-based behavior recognition method for children with autism in this application.

[0020] Figure 2 This is a schematic diagram of an embodiment of a data analysis-based behavioral recognition system for children with autism in this application.

[0021] Figure 3 This is a schematic block diagram of the structure of a child autism behavior recognition device based on data analysis in an embodiment of the present invention. Detailed Implementation

[0022] This application provides a data analysis-based method, system, and medium for identifying autistic behaviors in children. The terms "first," "second," "third," "fourth," etc. (if present) in the specification, claims, and accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms "comprising" or "having" and any variations thereof are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0023] For ease of understanding, the specific process of the embodiments of this application is described below. Please refer to [link / reference]. Figure 1 One embodiment of the data analysis-based method for identifying autistic behaviors in children in this application includes:

[0024] Step S101: Multimodal acquisition and preprocessing of children's behavior data to obtain a multimodal dataset. Multimodal acquisition includes collecting six-axis sensor data, positioning data and audio data through smartwatches worn by children, and simultaneously collecting video data of multiple children through multi-target monitoring cameras deployed in the classroom.

[0025] Step S102: Extract nonlinear features from the multimodal dataset to obtain the behavioral time series feature matrix;

[0026] Step S103: Input the multimodal dataset into a three-dimensional convolutional neural network for processing, identify repetitive behavior patterns and stereotyped actions, and obtain spatial behavior feature representations;

[0027] Step S104: Construct a temporal convolutional network model based on a multimodal dataset, analyze the temporal changes and state transition features of behavior, and obtain dynamic feature vectors of behavior;

[0028] Step S105: The behavioral time series feature matrix, spatial behavioral feature representation and behavioral dynamic feature vector are fused and processed. By feature concatenation and dimensionality reduction by autoencoder, a comprehensive feature vector is obtained.

[0029] Step S106: Input the comprehensive feature vector into the multi-classifier to identify autism behavior types and assess severity, and obtain behavior identification data.

[0030] It is understood that the executing entity of this application can be a data analysis-based behavioral recognition system for children with autism, or it can be a terminal or a server; the specific implementation is not limited here. This application's embodiments use a server as an example for illustration.

[0031] Specifically, multimodal data acquisition is achieved through two core data sources: a smartwatch and a classroom multi-target monitoring camera. The smartwatch's six-axis sensor includes a three-axis accelerometer and a three-axis gyroscope. The accelerometer measures linear acceleration in the X, Y, and Z directions, while the gyroscope measures angular velocity around the three axes. The combination of these measurements calculates the child's posture and positional changes. Location data is obtained through a GPS module or indoor positioning technology to acquire the child's spatial coordinates. Audio data is collected through a built-in microphone, capturing ambient sound and the child's vocalizations. The multi-target monitoring camera uses a wide-angle lens and a high-resolution sensor to simultaneously capture images of 20 children. The camera performs real-time target detection and tracking using an image processing chip. Data preprocessing includes timestamp synchronization, aligning data from different sensors along a unified timeline to eliminate time shifts caused by sampling frequency differences, followed by data cleaning to remove outliers caused by sensor malfunctions or obstructions.

[0032] Fractal analysis methods calculate the fractal dimension of data using box counting. This method plots the time series on a two-dimensional plane, covers the curve with grids of different sizes, counts the number of grids required for coverage, and obtains the fractal dimension through linear regression. A larger value indicates a more complex behavioral pattern. Long-range correlation analysis uses the rescaled range method to calculate the Hearst exponent. First, the cumulative deviation of the time series is calculated. The series is divided into subsequences of different lengths, and the range and standard deviation of each subsequence are calculated. The Hearst exponent is obtained through linear regression on a double logarithmic scale. This exponent reflects the long-term memory characteristics of the data. Dynamic instability analysis reconstructs the phase space using time-delay embedding and calculates the maximum Lyapunov exponent, which quantifies the degree of chaos and instability of the system. These three methods describe the nonlinear characteristics of children's behavior from three dimensions: complexity, correlation, and stability, forming a behavioral time series feature matrix.

[0033] The 3D convolutional neural network (CNN) process achieves spatiotemporal feature extraction. The network first takes 16 consecutive frames of video as input, each frame being 224×224 pixels in size, forming a 16×224×224×3 four-dimensional tensor. Dilated convolution expands the receptive field by inserting holes between kernel elements; a 3×3 kernel with a dilation rate of 2 has an actual receptive field of 5×5, a dilation rate of 4 has a receptive field of 9×9, and so on. The dilation rates of the four feature extraction modules are 1, 2, 4, and 8, respectively, with the number of output channels increasing to 64, 128, 256, and 512. Each convolutional layer is followed by a batch normalization layer to standardize the feature distribution, the ReLU activation function introduces non-linearity, and a max-pooling layer reduces spatial resolution. A temporal attention mechanism calculates the similarity of feature maps at different time steps, generating a weight distribution through a softmax function, focusing on recurring behavioral patterns. Global average pooling compresses the feature map into a fixed-length vector, and a fully connected layer maps the 512-dimensional features to a 256-dimensional spatial behavioral feature representation.

[0034] The input projection layer uses one-dimensional convolution to uniformly map multi-channel data to a 64-dimensional feature space, with a kernel size of 3 and a stride of 1. Four residual temporal convolutional blocks employ dilated convolutions with dilation rates of 1, 2, 4, and 8. Each block contains two one-dimensional convolutional layers: the first reduces dimensionality, and the second restores it. Residual connections directly add the input to the output to avoid gradient vanishing. Causal convolutions use left padding to ensure the current output depends only on historical information; the padding size is equal to the dilation rate multiplied by the kernel size minus 1. Batch normalization layers standardize the output of each layer, and discard layers randomly zero out some neurons to prevent overfitting. Adaptive max pooling compresses variable-length sequences into a fixed dimension, and the fully connected layer outputs a 128-dimensional dynamic feature vector.

[0035] The 15-dimensional behavioral time-series feature matrix, 256-dimensional spatial behavioral feature representation, and 128-dimensional behavioral dynamic feature vector are normalized, mapping the values ​​to the 0-1 interval to eliminate dimensional differences. Feature concatenation connects the three vectors into a 399-dimensional joint feature vector, retaining all original information but with higher dimensionality. The symmetric autoencoder consists of an encoder and a decoder. The encoder has a hierarchy of 399-256-128-64, and the decoder has a hierarchy of 64-128-256-399. The 64-dimensional bottleneck layer is a compressed representation of the features. During training, the mean squared error loss function is used, and the network parameters are optimized through backpropagation to minimize the reconstruction error. After training, the bottleneck layer output is extracted as compressed features, and a feature selection algorithm is used to retain key dimensions based on importance scores, forming a comprehensive feature vector.

[0036] A random forest is used to construct 100 decision trees, each using a bootstrap sampling training subset. Node splitting uses the Gini coefficient to select the optimal feature, and the maximum depth is limited to 10 to prevent overfitting. A support vector machine uses a radial basis function kernel to map the data to a high-dimensional space. A penalty parameter of 10 controls the tolerance for training error, and a kernel parameter of 0.01 determines the complexity of the decision boundary. Gradient boosting decision trees use a forward stepwise algorithm, where each training iteration fits the residuals from the previous iteration to a new tree. A learning rate of 0.1 controls the contribution of each tree, resulting in a total of 100 trees. A neural network classifier contains two hidden layers with a 64-32-16-3 structure, using ReLU activation and a softmax output layer. Weighted voting determines weights based on the accuracy of each classifier on the validation set, and the predicted probabilities are weighted and summed to obtain the final classification result. A regression network outputs a severity score based on the classification result, mapping the output to a 0-10 score range using a sigmoid function.

[0037] In one specific embodiment, the process of performing step S101 may specifically include the following steps:

[0038] By collecting children's acceleration and gyroscope data through the six-axis sensor of the smartwatch, the system can identify position changes, movement patterns, behavioral trajectories, stereotyped movements and gait characteristics, thus obtaining smartwatch sensor data.

[0039] By recording children's spatial movement trajectories through the positioning module of smartwatches, and analyzing the repetition and regularity of behavioral paths, positioning trajectory data can be obtained.

[0040] By using the audio sensor of a smartwatch to collect ambient sounds and children's vocalizations, the system analyzes speech communication patterns and responses to sound stimuli to obtain audio feature data.

[0041] By deploying multi-target monitoring cameras in kindergarten or primary school classrooms to simultaneously monitor no more than 20 children, group behavior videos are collected to obtain group video data;

[0042] Time synchronization is performed on smartwatch sensor data, location trajectory data, audio feature data, and group video data. Multimodal data streams are aligned based on timestamps to obtain a time-aligned data matrix.

[0043] Data augmentation is performed on the time-aligned data matrix, and the training samples are augmented by random cropping, rotation, and mirroring transformations to obtain a multimodal dataset.

[0044] Specifically, the smartwatch's six-axis sensor data acquisition is achieved through MEMS (Micro-Electro-Mechanical Systems) sensors. A three-axis accelerometer measures linear acceleration changes in the X, Y, and Z axes, while a three-axis gyroscope measures angular velocity changes around the three axes. The raw accelerometer output data is processed by a low-pass filter to remove high-frequency noise, and then a gravity component separation algorithm is used to separate gravitational acceleration from the total acceleration, yielding pure motion acceleration. Gyroscope data is used to calculate angle changes through integration. Combined with accelerometer data, a complementary filtering algorithm is used to fuse the two sensor information, eliminating integration drift errors. Position change recognition is achieved by calculating the difference in position vectors at consecutive time points. Motion pattern recognition determines states such as stationary, walking, and running by analyzing the variation patterns of acceleration amplitude. Behavioral trajectory is generated by converting the three-axis acceleration data into a spatial coordinate sequence. Stereotyped motion recognition is achieved by detecting repetitive acceleration patterns, and gait feature extraction obtains step frequency and stride length information by analyzing the periodic changes in vertical acceleration.

[0045] The positioning module acquires the child's geographic coordinates using a GPS chip or indoor positioning technology. GPS positioning calculates location outdoors by receiving satellite signals, while indoor positioning is achieved through WiFi fingerprint recognition or Bluetooth beacon positioning. Positioning trajectory data processing first transforms the original coordinates from geographic coordinates to a local Cartesian coordinate system, then smooths the trajectory using a Kalman filter algorithm to eliminate noise caused by positioning errors. Behavioral path repetition analysis is performed by calculating the Euclidean distance between trajectory points. When a child repeatedly traverses the same area, if the distance between the new trajectory point and the historical trajectory point is less than a set threshold, the system records the number of repetitions and the duration of dwell time. Regularity analysis detects periodic patterns in the trajectory using time series analysis methods, calculating the similarity of trajectories under different time delays using an autocorrelation function; peak positions correspond to periodic behavioral characteristics. Audio sensor acquisition is achieved through a digital microphone, with a sampling frequency set to 16kHz to meet speech analysis requirements. Audio data preprocessing includes pre-emphasis filtering to enhance high-frequency components, followed by frame segmentation to divide the continuous audio signal into overlapping short frames, each 25 milliseconds long with a 10-millisecond frame shift. Each audio frame is processed using a windowing function to reduce spectral leakage, and then subjected to a Fast Fourier Transform to obtain frequency domain features. Speech communication pattern analysis distinguishes speech segments from silence segments using a speech activity detection algorithm. Speech segments obtain speech feature parameters through fundamental frequency extraction and formant analysis. Sound stimulus response analysis is achieved by calculating changes in children's vocal patterns when environmental noise changes abruptly. Sound stimulus events are marked when the ambient volume exceeds twice the standard deviation of the average, and the changes in children's vocal frequency and intensity before and after the event are analyzed.

[0046] The multi-target monitoring camera employs a wide-angle lens and a high-resolution image sensor, covering the entire classroom area. The video stream output by the camera identifies and tracks children using target detection algorithms. A deep learning-based detector locates the child's position in each frame and assigns a unique identifier. Multi-target tracking is achieved through Kalman filtering to predict target positions and a Hungarian algorithm for data association. The tracking trajectory is updated when a newly detected target matches the predicted position, and deleted when the target disappears for more than a set time. The group behavior video data includes the bounding box coordinates, tracking identifier, and timestamp information for each child, while also recording the relative positional relationships and interactions between children.

[0047] Time synchronization is achieved through a unified timestamp benchmark. The smartwatch and camera synchronize their clocks at the start of data acquisition, using a network time protocol to ensure consistent time references. Different sensors have different sampling frequencies: smartwatch sensors are typically 100Hz, positioning modules are 1Hz, audio is 16kHz, and cameras are 30fps. The system uses a resampling algorithm to unify all data onto the same time grid. During resampling, linear interpolation is used to fill in missing time points. Decimation is used for data with sampling frequencies higher than the target frequency, and interpolation is used for data with sampling frequencies lower than the target frequency. The time-aligned data matrix uses time as the row index and different modalities as the column index, forming a structured multidimensional data table.

[0048] Data augmentation enhances the diversity of training samples through geometric and temporal transformations. Random cropping extracts data segments of varying lengths along the temporal dimension, preserving behavioral integrity while increasing the sample size. Rotation transformations are applied to sensor data, transforming the coordinate systems of acceleration and gyroscope data using rotation matrices to simulate different wearing orientations. Mirror transformations are achieved by flipping data values ​​along certain axes, such as transforming left-hand movement data into right-hand movement data. Video data augmentation is achieved through image processing techniques, including brightness adjustment, contrast modification, and noise addition, enhancing the model's robustness to changes in lighting and image quality.

[0049] In one specific embodiment, the process of performing step S102 may specifically include the following steps:

[0050] Motion intensity sequences are extracted from smartwatch sensor data in a multimodal dataset, and a sequence representing the overall motion amplitude changes over time is constructed to obtain motion intensity sequence data.

[0051] The location change distance sequence and dwell time sequence are extracted from the positioning trajectory data in the multimodal dataset to reflect spatial movement characteristics and location stability, respectively, thus obtaining the location change sequence data;

[0052] Sound intensity changes and speech activity sequences are extracted from audio feature data in multimodal datasets to reflect auditory stimulus response and language communication features, thus obtaining sound activity sequence data;

[0053] Body posture stability and limb activity are calculated from group video data in a multimodal dataset to reflect the changes in body posture over time and obtain body activity sequence data.

[0054] Fractal analysis methods were applied to motion intensity sequence data, position change sequence data, sound activity sequence data, and body activity sequence data to measure the complexity and irregularity of the data, and to obtain a set of behavioral complexity indicators.

[0055] Long-range time series correlation analysis was applied to motion intensity sequence data, location change sequence data, sound activity sequence data, and physical activity sequence data to measure the long-term dependencies between data points and obtain a set of behavioral correlation indicators.

[0056] Dynamic instability analysis methods are applied to motion intensity sequence data, position change sequence data, sound activity sequence data, and physical activity sequence data to measure the instability and predictability of behavioral systems and obtain behavioral time series feature matrices.

[0057] Specifically, the motion intensity sequence extraction from the smartwatch sensor data is achieved through vector synthesis of triaxial acceleration data. First, the magnitude of the acceleration vector at each time point is calculated using the formula: the square root of the sum of the squares of the accelerations along the X, Y, and Z axes. Then, the synthesized acceleration is low-pass filtered to remove the gravity component, yielding pure motion acceleration. Motion intensity is obtained by calculating the root mean square value using a sliding window. The window length is set to 1 second, and the sliding step size is 0.1 seconds. The average square root of the squared acceleration within each window is calculated to obtain the motion intensity value for that time period. The overall motion amplitude sequence reflects the changes in the intensity of children's activities; high intensity values ​​correspond to vigorous activities such as running and jumping, while low intensity values ​​correspond to sitting still or slow movement. The fluctuating patterns of the sequence reflect the children's activity patterns and behavioral characteristics.

[0058] The location change distance sequence of the positioning trajectory data is obtained by calculating the Euclidean distance between consecutive time points. The distance between any two adjacent positioning points is equal to the square root of the sum of the squared differences in the X and Y coordinates. The dwell time sequence is calculated by detecting consecutive time periods where the location change is less than a threshold. When the location change distance at multiple consecutive time points is less than 0.5 meters, the child is considered to be in a dwelling state. The start and end times of the dwelling are recorded to calculate the dwell time. The location change sequence data includes information such as movement speed, movement direction, and dwelling distribution, reflecting the child's spatial exploration behavior and location preferences. Children with autism often exhibit characteristics of repetitive path walking and dwelling in fixed areas.

[0059] The sound intensity changes in audio feature data are obtained by calculating the short-time energy of the audio signal. The audio signal is divided into frames, and the energy value of each frame is calculated. The energy is equal to the sum of the squares of all sampling points within the frame divided by the frame length. Speech activity sequences are identified using a speech activity detection algorithm. This algorithm combines energy detection and zero-crossing rate detection to determine whether an audio frame contains speech. A frame is marked as a speech activity frame when its energy exceeds the background noise energy threshold and its zero-crossing rate is within the typical range for speech. Sound activity sequence data reflects children's vocal patterns and response characteristics to sound stimuli, including parameters such as vocal frequency, duration, and intensity changes. These parameters are closely related to children's language development level and social communication abilities.

[0060] The stability of body posture in group video data is calculated using the standard deviation of keypoint coordinates. First, a posture estimation algorithm is used to extract the coordinates of the child's skeletal keypoints, including 18 keypoints such as the head, shoulders, elbows, and wrists. Then, the standard deviation of each keypoint's position within a time window is calculated; a smaller standard deviation indicates greater posture stability. Limb activity is calculated using the rate of change of angles between keypoints, calculating joint angles such as the shoulder-elbow angle and elbow-wrist angle. The change in angles between consecutive frames is then calculated, and the activity level is equal to the weighted average of all joint angle changes. Body activity sequence data includes posture stability indicators and limb movement indicators, reflecting the child's motor coordination and control abilities.

[0061] Fractal analysis uses box counting to measure the fractal dimension of time series data. First, the time series data is plotted on a two-dimensional coordinate system, with time on the horizontal axis and numerical values ​​on the vertical axis. Then, square grids of varying side lengths are used to cover the curve, and the minimum number of grids required to cover the curve is calculated. As the grid side length decreases, the corresponding number of grids increases. A graph showing the relationship between grid side length and the number of grids is plotted on a log-log coordinate system. The slope is calculated using linear regression; the negative value of the slope is the fractal dimension. The fractal dimension reflects the complexity and irregularity of the data. A value close to 1 indicates relatively simple and regular data, while a value close to 2 indicates complex and irregular data. Behavioral data of children with autism often exhibits a high fractal dimension.

[0062] Long-range correlation analysis of time series uses the rescaled range method to calculate the Hearst exponent. First, the cumulative deviation of the time series is calculated, which is the cumulative sum of the values ​​at each point minus the series mean. Then, the cumulative deviation is divided into subsequences of different lengths, and the range and standard deviation of each subsequence are calculated. The range is the difference between the maximum and minimum values ​​of the subsequence, and the standard deviation is the standard deviation of the subsequence. The rescaled range is equal to the range divided by the standard deviation. The average of the rescaled ranges for subsequences of different lengths is calculated. The relationship between subsequence length and rescaled range is plotted on a log-log coordinate system, and the slope obtained through linear regression is the Hearst exponent. The Hearst exponent reflects the long-term memory characteristics of the data; a value greater than 0.5 indicates positive correlation, less than 0.5 indicates negative correlation, and equal to 0.5 indicates randomness.

[0063] Dynamic instability analysis employs the maximum Lyapunov exponent. First, a one-dimensional time series is reconstructed into a multi-dimensional phase space using a time-delay embedding method. By selecting appropriate embedding dimensions and time delay parameters, the original sequence is transformed into a multi-dimensional trajectory. In the reconstructed phase space, the nearest neighbor of each point is found, and the separation of these nearest neighbors over time is tracked, recording the initial distance and the distance after a certain time. The maximum Lyapunov exponent is equal to the logarithmic mean of the distance growth rate; a positive value indicates chaotic characteristics, a negative value indicates stability, and a zero value indicates a critical state. The Lyapunov exponent reflects the instability and predictability of a behavioral system; the behavior of children with autism often exhibits high instability.

[0064] In one specific embodiment, the process of executing step S103 may specifically include the following steps:

[0065] The group video data in the multimodal dataset is segmented and processed. A sliding sampling is performed with a step size of 2 frames, using 16 consecutive frames as an analysis unit to obtain a spatiotemporal data block sequence.

[0066] The spatiotemporal data block sequence is input into the input layer of the dilated 3D convolutional neural network. The input layer receives data blocks of size 16×224×224×3 to obtain the initial feature map.

[0067] The initial feature map is processed sequentially through four feature extraction modules. Each module contains two dilated 3D convolutional layers, a batch normalization layer, an activation function layer, and a pooling layer. The dilation rates are 1, 2, 4, and 8, respectively, and the number of output channels are 64, 128, 256, and 512, respectively, to obtain a multi-scale spatiotemporal feature map.

[0068] A temporal attention mechanism is applied to multi-scale spatiotemporal feature maps to calculate the correlation weights between different time steps, focusing on repetitive and regular behavioral segments to obtain weighted feature maps;

[0069] Perform global average pooling on the weighted feature map to compress the feature map into a 512-dimensional feature vector, thus obtaining the primary feature representation;

[0070] The initial feature representation is input into the fully connected layer, and the dimensionality is reduced to a 256-dimensional vector through the weight matrix mapping, thus obtaining the spatial behavior feature representation.

[0071] Specifically, the segmented processing of group video data is achieved through a sliding window technique. The original video stream is arranged into a frame sequence in chronological order, with each analysis unit containing 16 consecutive frames. The window sliding step is set to 2 frames, meaning there is a 14-frame overlap between adjacent analysis units. Sliding sampling ensures temporal continuity; the first analysis unit contains frames 1 to 16, the second contains frames 3 to 18, and so on, forming a large number of overlapping spatiotemporal data blocks. Each data block maintains temporal coherence, capturing short-term motion changes; the 16-frame length is sufficient to contain a complete basic action such as a gesture or gait cycle. The spatiotemporal data block sequence converts the 3D video data into a 4D tensor format, facilitating processing by a 3D convolutional neural network.

[0072] The input layer of the dilated 3D convolutional neural network receives standardized spatiotemporal data blocks, each with a size of 16×224×224×3, where 16 represents the number of frames in the temporal dimension, 224×224 represents the spatial resolution of each frame, and 3 represents the number of RGB color channels. The input layer preprocesses the data blocks, including normalizing pixel values ​​to the 0-1 range and data type conversion, ensuring the data format conforms to network requirements. Initial feature maps are generated through the first layer of 3D convolutional operations, where the convolutional kernel slides simultaneously in both temporal and spatial dimensions to extract local spatiotemporal features. The 3D convolutional kernel is typically 3×3×3 in size, covering both temporal and spatial neighborhoods, and compared to 2D convolution, it can capture the temporal evolution of actions.

[0073] The four feature extraction modules employ a progressive dilated convolution design, which expands the receptive field by inserting zeros between kernel elements. The first module has a dilation rate of 1, equivalent to standard convolution, with a receptive field of 3×3×3, primarily extracting fine-grained local features. The second module has a dilation rate of 2, inserting a zero between kernel elements, effectively expanding the receptive field to 5×5×5, capturing a wider range of spatiotemporal correlations. The third module has a dilation rate of 4, further expanding the receptive field to 9×9×9, and the fourth module has a dilation rate of 8, reaching a receptive field of 17×17×17. This progressively expanding receptive field design enables the network to capture multi-scale features from local details to global patterns. Each module contains two dilated 3D convolutional layers for deep feature extraction. Batch normalization layers standardize the feature distribution to accelerate convergence, the ReLU activation function introduces a non-linear transformation, and max pooling layers reduce feature map resolution and computational cost. The number of output channels increases from 64 to 512, with feature representation capabilities increasing layer by layer.

[0074] The temporal attention mechanism is implemented by calculating similarity weights between feature maps at different time steps. First, the multi-scale spatiotemporal feature map is decomposed into multiple two-dimensional feature maps along the time dimension. Then, an attention score is calculated between each pair of time steps. The attention score is calculated using a dot product operation, where the query vector and key vector are multiplied to obtain the similarity measure; higher similarity indicates a stronger correlation between the features of the two time steps. The attention weights are normalized using the softmax function to ensure that the sum of the weights for all time steps is 1. Repetitive and regular behaviors exhibit high similarity in the time dimension, and the system automatically assigns higher attention weights to highlight these key time periods. The weighted feature map is obtained by multiplying the attention weights by the original feature map, enhancing the feature representation of important time steps and suppressing noise interference from irrelevant time steps.

[0075] Global average pooling compresses a 3D weighted feature map into a 1D feature vector by averaging across all spatial and temporal locations of the feature map, producing a scalar value for each channel. The weighted feature map has dimensions T×H×W×C, where T is the temporal dimension, H and W are the spatial dimensions, and C is the number of channels (512). Global average pooling yields a 512-dimensional feature vector. Pooling eliminates spatial and temporal location information in the feature map while preserving the feature intensity contrast between channels, making the feature representation invariant to the spatial location and temporal offset of the input. The primary feature representation captures high-level semantic information in the video sequence, including abstract features such as action type, motion pattern, and spatiotemporal correlation.

[0076] The fully connected layer maps 512-dimensional primary features to a 256-dimensional spatial behavioral feature representation through a linear transformation. The weight matrix has a size of 512×256, and the bias vector has a size of 256. The linear transformation is calculated by multiplying the feature vector by the weight matrix and then adding the bias vector; mathematically, the output equals the input feature multiplied by the weight matrix plus the bias. The weight parameters of the fully connected layer are trained using a backpropagation algorithm to learn the optimal transformation that maps high-dimensional primary features to low-dimensional discriminative features. The 256-dimensional spatial behavioral feature representation is a high-level abstraction and compression of the original video data, preserving the key information needed to identify autistic behaviors.

[0077] In one specific embodiment, the process of executing step S104 may specifically include the following steps:

[0078] The smartwatch sensor data, positioning trajectory data, audio feature data, and group video data in the multimodal dataset are arranged in chronological order to obtain a time-series data sequence;

[0079] The time-series data sequence is processed through the input projection layer, and one-dimensional convolution is used to map the multi-channel input to a unified 64-dimensional feature space to obtain a standardized feature sequence.

[0080] The standardized feature sequence is sequentially input into four residual temporal convolutional blocks. Each convolutional block contains two one-dimensional convolutional layers, a batch normalization layer, an activation function layer, and a dropout layer. The dilation rates are 1, 2, 4, and 8, respectively, to obtain multi-scale temporal features.

[0081] By applying causal convolution design to multi-scale temporal features and padding to ensure that the output length is consistent with the input length, temporal correlation data can be obtained.

[0082] Adaptive max pooling is performed on temporal correlation data to compress the variable-length temporal features into a fixed-dimensional representation, resulting in a compressed feature vector.

[0083] The compressed feature vector is mapped to a 128-dimensional vector through a fully connected layer, and regularization constraints are applied to obtain the dynamic feature vector of behavior.

[0084] Specifically, the temporal arrangement of the multimodal datasets is achieved through a unified timestamp benchmark. Smartwatch sensor data includes 6 dimensions: three-axis acceleration and three-axis gyroscope data. Positioning trajectory data includes 3 dimensions: X and Y coordinates and distance traveled. Audio feature data includes 2 dimensions: sound intensity and voice activity markers. Group video data includes 2 dimensions: body posture stability and limb movement intensity. The four data sources have different sampling frequencies: 100Hz for sensor data, 1Hz for positioning data, 50Hz for audio data, and 30Hz for video data. The system resamples all data to a frequency of 10Hz, i.e., one data point every 0.1 seconds. The time-series data sequence is organized into a matrix format along the time axis, with each row representing a time point and each column representing a feature dimension, for a total of 6+3+2+2=13 dimensions, forming a T×13 time-series matrix, where T is the length of the time series.

[0085] The input projection layer uses one-dimensional convolution to map the 13-dimensional multi-channel input to a 64-dimensional unified feature space. The kernel size is set to 3, meaning that each convolution operation considers information from three consecutive time points. The one-dimensional convolution slides along the time dimension, linearly transforming the 13-dimensional feature vector within each time window, generating 64 output channels through 64 different convolution kernels. The convolution operation is essentially a linear combination and non-linear transformation of feature dimensions, with each output channel learning specific correlation patterns between different modalities. The batch normalization layer standardizes the convolution output, eliminating dimensional differences between different features, and the ReLU activation function introduces non-linear transformation capabilities. The standardized feature sequence maintains the original time series length, but the feature dimension is expanded from 13 to 64 dimensions, enhancing feature expressive power.

[0086] The four residual temporal convolutional blocks employ an increasing dilation rate design. Dilated convolutions increase the receptive field by inserting zeros between kernel elements. The first convolutional block has a dilation rate of 1, equivalent to a standard convolution, with a receptive field of 3 time steps, primarily capturing short-term temporal dependencies. The second convolutional block has a dilation rate of 2, expanding the receptive field to 5 time steps; the third convolutional block has a dilation rate of 4, expanding the receptive field to 9 time steps; and the fourth convolutional block has a dilation rate of 8, reaching a receptive field of 17 time steps. This exponentially increasing dilation rate design enables the network to efficiently capture multi-scale temporal patterns, from short-term behavioral changes to long-term behavioral trends. Each convolutional block contains two one-dimensional convolutional layers: the first layer is used for feature dimensionality reduction, and the second layer is used for feature recovery. Residual connections directly add the input to the output, avoiding the gradient vanishing problem common in deep networks. Batch normalization layers standardize the distribution of intermediate features, the ReLU activation function provides a non-linear transformation, and dropout layers randomly set some neuron outputs to zero to prevent overfitting.

[0087] Causal convolutional design ensures that the model uses only historical information for prediction, conforming to the causal constraints of time series. Causality is achieved through asymmetric padding, where zero values ​​are padded to the left of the input sequence. The padding length is equal to the dilation rate multiplied by the kernel size minus 1. For example, a convolutional layer with a dilation rate of 4 and a kernel size of 3 will have 4 × (3 - 1) = 8 zero values ​​padded to the left. The padding operation ensures that the output length after convolution is the same as the input length, preventing the sequence length from shortening during network propagation. Temporal correlation data retains complete temporal dimension information; the feature vector at each time point integrates multimodal information within the historical time window, reflecting behavioral correlation patterns at different time scales.

[0088] Adaptive max pooling handles variable-length temporal features, compressing input sequences of varying lengths into a fixed-dimensional output. The pooling operation divides the temporal features into a fixed number of segments along the time dimension, performing a maximum value operation within each segment to select the most salient feature for activation. The number of segments is typically set to 8 or 16 to ensure a fixed output dimension and sufficient temporal information. Max pooling preserves peak features within each time segment, which often correspond to important behavioral events or state transitions. The dimension of the compressed feature vector is the number of segments multiplied by the number of feature channels; for example, 8 time segments and 64 feature channels produce a 512-dimensional compressed vector.

[0089] The fully connected layer maps the compressed feature vectors to 128-dimensional dynamic feature vectors through linear transformation, and the weight matrix learns the optimal feature combination. Regularization constraints include L2 weight decay and dropout regularization. L2 regularization, by adding a penalty term to the sum of squared weights in the loss function, encourages the model to learn smaller weight values, improving generalization ability. Dropout regularization randomly sets the output of some neurons to zero during training, forcing the network to learn more robust feature representations. The regularization coefficients are determined to be optimal through cross-validation, balancing model complexity and generalization performance.

[0090] In one specific embodiment, the process of executing step S105 may specifically include the following steps:

[0091] The behavioral time series feature matrix, spatial behavioral feature representation, and behavioral dynamic feature vector are normalized to obtain a standardized feature set;

[0092] Perform a feature concatenation operation on the standardized feature set to concatenate the 15-dimensional behavioral time series feature matrix, the 256-dimensional spatial behavioral feature representation, and the 128-dimensional behavioral dynamic feature vector into a 399-dimensional joint feature, resulting in a feature concatenation vector.

[0093] A symmetric autoencoder network is constructed by concatenating feature vectors. The autoencoder consists of an encoder and a decoder, resulting in an encoding-decoding model.

[0094] The feature concatenation vector is input into the encoding and decoding model, trained through unsupervised learning, and the network parameters are optimized using the mean squared error loss function to obtain the trained autoencoder.

[0095] Extract the 64-dimensional representation of the bottleneck layer from the trained autoencoder to obtain a compact feature representation, thus obtaining a compressed feature representation;

[0096] A feature selection algorithm is applied to the compressed feature representation, and key dimensions are retained based on feature importance scores to obtain a comprehensive feature vector.

[0097] Specifically, normalization uses a min-max scaling method to map different features to a uniform numerical range, eliminating the impact of dimensional differences on subsequent processing. The behavioral time series feature matrix contains 15 nonlinear indicators, namely the fractal dimension, Hearst exponent, and Lyapunov exponent for four types of sequences: motion intensity, position change, sound activity, and physical activity. These indicators have significantly different numerical ranges: fractal dimension typically ranges from 1.0 to 2.0, Hearst exponent from 0.0 to 1.0, and Lyapunov exponent from -0.5 to 0.5. The normalization formula is: the standardized value equals the original value minus the minimum value, then divided by the maximum value minus the minimum value, mapping all feature values ​​to the 0-1 interval. The 256-dimensional vector representing spatial behavioral features is processed using the ReLU activation function, with a numerical range from 0 to positive infinity. Normalization is achieved by dividing by the maximum value of the vector. The 128-dimensional values ​​of the behavioral dynamic feature vector are linearly transformed by a fully connected layer and standardized using the Z-score method. The standardized value is calculated as the original value minus the mean and then divided by the standard deviation, making the feature distribution close to the standard normal distribution.

[0098] The feature concatenation operation joins three types of feature vectors into a high-dimensional joint representation in a fixed order: the 15 dimensions of the behavioral time-series feature matrix first, the 256 dimensions of the spatial behavioral feature representation in the middle, and the 128 dimensions of the behavioral dynamic feature vector last, forming a 399-dimensional feature concatenation vector. The concatenation process maintains the internal structural integrity of each feature type. The first 15 dimensions correspond to the nonlinear time-series analysis results, the middle 256 dimensions correspond to the spatial patterns extracted by the 3D convolutional network, and the last 128 dimensions correspond to the dynamic changes captured by the temporal convolutional network. The feature concatenation vector integrates the temporal, spatial, and dynamic dimensions of multimodal data, providing a comprehensive feature description for autism behavior recognition.

[0099] The symmetric autoencoder network employs an hourglass-shaped structure. The encoder consists of four fully connected layers with decreasing numbers of neurons (399-256-128-64) to progressively compress and abstract features. The decoder is a mirror image of the encoder, with increasing numbers of neurons (64-128-256-399) to reconstruct the compressed features back to their original dimensions. Each fully connected layer is followed by a batch normalization layer to standardize the feature distribution. The ReLU activation function introduces non-linear transformation capabilities, and a dropout layer randomly zeros the neuron outputs with a probability of 0.2 to prevent overfitting. The bottleneck layer, set to 64 dimensions, is the information compression center of the entire network, forcing the network to learn the most important feature representations. The symmetric design of the encoder-decoder model ensures a balance between feature compression and reconstruction, avoiding excessive information loss or retaining redundant information.

[0100] Unsupervised learning training does not require external labels; it uses only the input data itself as a supervision signal. The training objective is to make the autoencoder's output as close to the input as possible, learning the intrinsic representation of the data by minimizing the reconstruction error. The mean squared error loss function calculates the mean of the squared differences between the original input and the reconstructed output; mathematically, the loss value equals the sum of the squares of the differences between the input and output vectors divided by the vector dimension. The backpropagation algorithm calculates the gradient of the loss function with respect to the network parameters, and the Adam optimizer updates the weight parameters based on the gradient. The learning rate is set to 0.001, and the training iterations are 100 epochs. The batch size is set to 32, and the average gradient of 32 samples is used for each update to improve training stability and convergence speed. An early stopping strategy monitors the loss change on the validation set; training stops when there is no improvement for 10 consecutive epochs to avoid overfitting.

[0101] The 64-dimensional bottleneck layer represents the compressed data learned by the autoencoder, containing the core information of the original 399-dimensional features. The extraction process splits the trained autoencoder into an encoder part, and the input feature concatenation vector is forward-propagated through the encoder to obtain the bottleneck layer output. Each dimension of the 64-dimensional compressed feature representation is a non-linear combination of the original multi-dimensional features, and the optimal feature combination is automatically learned through network training. Compared to the original features, the compressed features have better discriminative and generalizable properties, removing redundant information and noise interference while retaining the key patterns for recognizing autistic behaviors.

[0102] The feature selection algorithm employs a variance-based feature importance assessment method, calculating the variance of each compressed feature dimension across all samples. A larger variance indicates stronger discriminative power for that dimension. The variance calculation formula is: feature variance equals the sum of the squares of all sample values ​​for that dimension divided by the sample size minus the square of the mean. The feature importance score is obtained by normalizing the variance values, dividing the variance values ​​of all dimensions by the largest variance value to obtain an importance score ranging from 0 to 1. The feature selection threshold is set to 0.3, retaining dimensions with importance scores greater than 0.3, typically retaining 40-50 key dimensions to form a comprehensive feature vector. The feature selection process further reduces the feature dimensions, improving computational efficiency while maintaining the discriminative power of the features.

[0103] In one specific embodiment, the process of executing step S106 may specifically include the following steps:

[0104] A random forest classifier is constructed based on the comprehensive feature vector. The number of trees is set to 100 and the maximum depth is 10. The Gini coefficient is used as the splitting criterion to obtain the first classifier model.

[0105] A support vector machine classifier is constructed based on the comprehensive feature vectors. A radial basis function kernel function is used, with a penalty parameter of 10 and a kernel parameter of 0.01, to obtain the second classifier model.

[0106] A gradient boosting decision tree classifier is constructed based on the comprehensive feature vectors. The number of trees is set to 100, the learning rate is 0.1, and the maximum depth is 5, resulting in a third classifier model.

[0107] A neural network classifier is constructed based on comprehensive feature vectors, containing two hidden layers with a structure of 64-32-16-3. An activation function and an output layer are used to obtain a fourth classifier model.

[0108] Weighted voting is performed on the prediction results of the first classifier model, the second classifier model, the third classifier model, and the fourth classifier model. The weights are determined based on the performance of each classifier on the validation set to obtain the behavior type discrimination result.

[0109] A regression network is constructed based on the behavior type discrimination results to output autism behavior severity scores. The scores are then mapped to standardized scale intervals to obtain behavior identification data.

[0110] Specifically, the random forest classifier achieves classification prediction by integrating multiple decision trees. Each decision tree uses a bootstrap sampling method to randomly select samples from the training data. Repeated selection is allowed during the sampling process, and typically the training set size of each tree is equal to the original training set. 100 decision trees are built in parallel. When splitting a node, each tree randomly selects a subset of features from all dimensions of the integrated feature vector. The size of the subset is typically set to the square root of the total number of features. The Gini coefficient is used as a splitting criterion to measure the impurity of a node. The formula is: Gini coefficient = 1 minus the sum of the squares of the probabilities of each class; the smaller the value, the purer the node. The decision tree recursively splits, selecting the features and thresholds that maximize the reduction of the Gini coefficient, until the maximum depth of 10 is reached or the number of samples in the node is less than the minimum number of split samples. After each tree is trained independently, the random forest makes predictions through majority voting. The predictions from all 100 trees are aggregated, and the class with the most votes is selected as the final prediction.

[0111] Support Vector Machine (SVM) classifiers separate samples of different classes by finding the optimal separating hyperplane. The radial basis function (RBF) maps the low-dimensional feature space to a high-dimensional space, enhancing the classification ability of linearly inseparable data. The mathematical expression of the RBF is that the kernel value equals an exponential function, with the kernel parameter having a negative exponent multiplied by the sum of the squares of the differences between the two sample vectors. The kernel parameter 0.01 controls the bandwidth of the RBF; a smaller value indicates a sharper function and a more complex decision boundary. The penalty parameter 10 controls the model's tolerance for training error; a larger value indicates a more severe penalty for misclassification, and the model tends to generate more complex decision boundaries to reduce training error. SVM finds the optimal hyperplane by solving a quadratic programming problem. The optimization objective is to maximize the classification margin while minimizing the classification error. The Lagrange multiplier method transforms the constrained optimization problem into a dual problem.

[0112] The gradient boosting decision tree classifier employs a forward stepwise addition model, iteratively training weak classifiers and combining them into a strong classifier. The first decision tree is trained using the original training data, and subsequent trees are trained on the prediction residuals of the previous model, gradually reducing the model's prediction error. A learning rate of 0.1 controls the contribution of each tree to the final model; a smaller learning rate requires more trees to achieve the same fitting effect, but usually results in better generalization ability. The maximum depth of the decision trees is limited to 5 to prevent individual trees from overfitting the training data and to maintain the characteristics of weak classifiers. During the iterative training of 100 trees, the prediction error of the current model is calculated in each round, a new decision tree is constructed to fit the error distribution, and then the new tree is added to the ensemble model with the learning rate as weight. The core idea of ​​gradient boosting is to use gradient descent to find the optimal model in the function space, with each new tree optimizing the model's prediction along the negative gradient direction of the loss function.

[0113] The neural network classifier employs a multilayer perceptron architecture, comprising an input layer, two hidden layers, and an output layer. The input layer receives all dimensions of the synthesized feature vector. The first hidden layer contains 64 neurons, the second hidden layer contains 32 neurons, and the output layer contains 16 intermediate neurons and 3 final output neurons corresponding to the three behavior types. The hidden layers use the ReLU activation function, calculated as the activation value equal to the maximum of the input value and 0, introducing non-linear transformation capability while avoiding the gradient vanishing problem. The output layer uses the softmax activation function, transforming the network output into a probability distribution. The softmax function calculates the exponent value of each class divided by the sum of the exponent values ​​of all classes. Network training uses backpropagation and a stochastic gradient descent optimizer, with a learning rate of 0.001, a batch size of 32, and 200 training iterations. The cross-entropy loss function measures the difference between the predicted probability distribution and the true label distribution, driving the network parameters to be updated in a direction that reduces classification error.

[0114] The weighted voting mechanism assigns voting weights to each classifier based on their performance on the validation set, with higher-performing classifiers receiving higher weights. Validation set performance is evaluated comprehensively using accuracy, precision, recall, and F1 score. The weight calculation formula is that the classifier's weight equals its F1 score divided by the sum of the F1 scores of all classifiers. The weighted voting process multiplies the predicted probability of each classifier by its corresponding weight and then sums them to obtain the final class probability distribution. For example, the weights for Random Forest, Support Vector Machine, Gradient Boosting Tree, and Neural Network are 0.3, 0.25, and 0.2 respectively, resulting in a final predicted probability that is the weighted average of the four models' predicted probabilities. The behavior type discrimination result selects the class with the highest probability as the predicted label, while retaining the probability value as a confidence index.

[0115] The regression network estimates the severity score of autistic behaviors based on classification results. The network structure includes an input layer that receives classification probabilities and some raw features, two hidden layers containing 32 and 16 neurons respectively, and an output layer that produces a single numerical score. Input features include predicted probabilities of the three behavior classes, classification confidence, and key raw features such as the frequency of repetitive behaviors and the number of social interactions. The regression network is trained using a mean squared error loss function, aiming to make the predicted values ​​as close as possible to the severity scores assessed by experts. The score mapping uses a linear transformation to map the continuous numerical values ​​of the network output to a standardized scale range, typically 0-10, where 0-3 indicates mild autism, 4-6 indicates moderate autism, and 7-10 indicates severe autistic behavioral manifestations.

[0116] The above describes the data analysis-based behavioral recognition method for children with autism in the embodiments of this application. The following describes the data analysis-based behavioral recognition system for children with autism in the embodiments of this application. Please refer to [link / reference]. Figure 2 One embodiment of the data analysis-based behavioral recognition system for children with autism in this application includes:

[0117] The acquisition module 201 is used to collect and preprocess children's behavioral data in a multimodal manner to obtain a multimodal dataset. The multimodal acquisition includes collecting six-axis sensor data, positioning data and audio data through a smartwatch worn by the children, as well as simultaneously collecting video data of multiple children through a multi-target monitoring camera deployed in the classroom.

[0118] Extraction module 202 is used to extract nonlinear features from multimodal datasets to obtain behavioral time series feature matrices;

[0119] Processing module 203 is used to input multimodal datasets into a three-dimensional convolutional neural network for processing, identify repetitive behavior patterns and stereotyped actions, and obtain spatial behavior feature representations;

[0120] Module 204 is used to build a temporal convolutional network model based on a multimodal dataset, analyze the temporal changes and state transition features of behavior, and obtain dynamic feature vectors of behavior.

[0121] The fusion module 205 is used to fuse the behavioral time series feature matrix, the spatial behavioral feature representation, and the behavioral dynamic feature vector, and obtain a comprehensive feature vector through feature concatenation and dimensionality reduction by an autoencoder.

[0122] The identification module 206 is used to input the comprehensive feature vector into the multi-classifier to identify the autism behavior type and assess the severity, thereby obtaining behavior identification data.

[0123] above Figure 2The data analysis-based behavioral recognition system for children with autism in this invention will be described in detail from the perspective of modular functional entities. The data analysis-based behavioral recognition device for children with autism in this invention will be described in detail from the perspective of hardware processing.

[0124] Figure 3 This is a schematic diagram of a data analysis-based behavioral recognition device for children with autism, provided in an embodiment of the present invention. The data analysis-based behavioral recognition device 300 for children with autism can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 may be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module may include a series of instruction operations on the data analysis-based behavioral recognition device 300 for children with autism. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the data analysis-based behavioral recognition device 300 to implement the steps of the aforementioned data analysis-based behavioral recognition method for children with autism.

[0125] The data analysis-based behavioral recognition device for children with autism 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will understand that... Figure 3 The illustrated structure of the data analysis-based behavioral recognition device for children with autism does not constitute a limitation on the data analysis-based behavioral recognition device for children with autism provided by the present invention. It may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0126] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when the instructions are executed on a computer, cause the computer to perform the steps of the data analysis-based method for identifying autistic behaviors in children.

[0127] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0128] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a data analysis-based child autism behavior recognition device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0129] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying autistic behaviors in children based on data analysis, characterized in that, The method includes: Multimodal acquisition and preprocessing of children's behavioral data are performed to obtain a multimodal dataset. The multimodal acquisition includes collecting six-axis sensor data, positioning data, and audio data through smartwatches worn by children, as well as simultaneously collecting video data of multiple children through multi-target monitoring cameras deployed in the classroom. Nonlinear features are extracted from the multimodal dataset to obtain a behavioral time series feature matrix; The multimodal dataset is input into a three-dimensional convolutional neural network for processing to identify repetitive behavior patterns and stereotyped actions, thereby obtaining spatial behavior feature representations. Based on the multimodal dataset, a temporal convolutional network model is constructed to analyze the temporal changes and state transition features of behavior, thereby obtaining a dynamic feature vector of behavior. The behavioral time series feature matrix, the spatial behavioral feature representation, and the behavioral dynamic feature vector are fused together, and a comprehensive feature vector is obtained through feature concatenation and dimensionality reduction by an autoencoder. The comprehensive feature vector is input into a multi-classifier to identify autistic behavior types and assess severity, thus obtaining behavior identification data.

2. The method for identifying autistic behaviors in children based on data analysis according to claim 1, characterized in that, The process of collecting and preprocessing children's behavioral data in a multimodal manner yields a multimodal dataset, including: By collecting children's acceleration and gyroscope data through the six-axis sensor of the smartwatch, the system can identify position changes, movement patterns, behavioral trajectories, stereotyped movements and gait characteristics, thus obtaining smartwatch sensor data. By recording children's spatial movement trajectories through the positioning module of smartwatches, and analyzing the repetition and regularity of behavioral paths, positioning trajectory data can be obtained. By using the audio sensor of a smartwatch to collect ambient sounds and children's vocalizations, the system analyzes speech communication patterns and responses to sound stimuli to obtain audio feature data. By deploying multi-target monitoring cameras in kindergarten or primary school classrooms to simultaneously monitor no more than 20 children, group behavior videos are collected to obtain group video data; Time synchronization is performed on the smartwatch sensor data, the positioning trajectory data, the audio feature data, and the group video data. The multimodal data streams are aligned based on timestamps to obtain a time-aligned data matrix. Data augmentation is performed on the time-aligned data matrix to expand the training samples through random pruning, rotation, and mirror transformation, thus obtaining the multimodal dataset.

3. The method for identifying autistic behaviors in children based on data analysis according to claim 1, characterized in that, The step of extracting nonlinear features from the multimodal dataset to obtain a behavioral time series feature matrix includes: Motion intensity sequences are extracted from smartwatch sensor data in the multimodal dataset, and a sequence representing the overall motion amplitude changes over time is constructed to obtain motion intensity sequence data; The location change distance sequence and dwell time sequence are extracted from the positioning trajectory data in the multimodal dataset to reflect spatial movement characteristics and location stability, respectively, thus obtaining the location change sequence data; Sound intensity changes and speech activity sequences are extracted from the audio feature data in the multimodal dataset to reflect auditory stimulus response and language communication characteristics, thus obtaining sound activity sequence data; The stability of body posture and the degree of limb movement are calculated from the group video data in the multimodal dataset to reflect the changes in body posture over time, and body activity sequence data are obtained. Fractal analysis is applied to the motion intensity sequence data, the position change sequence data, the sound activity sequence data, and the body activity sequence data to measure the complexity and irregularity of the data, thereby obtaining a set of behavioral complexity indicators. The long-term correlation analysis method of time series is applied to the motion intensity sequence data, the position change sequence data, the sound activity sequence data, and the body activity sequence data to measure the long-term dependence between data points and obtain a set of behavioral correlation indicators. By applying dynamic instability analysis methods to the motion intensity sequence data, the position change sequence data, the sound activity sequence data, and the body activity sequence data, the instability and predictability of the behavioral system are measured, and the behavioral time series feature matrix is ​​obtained.

4. The method for identifying autistic behaviors in children based on data analysis according to claim 1, characterized in that, The process of inputting the multimodal dataset into a three-dimensional convolutional neural network for processing to identify repetitive behavioral patterns and stereotyped actions, and obtaining spatial behavioral feature representations, includes: The group video data in the multimodal dataset is segmented and processed by using 16 consecutive frames as an analysis unit and sliding sampling with a step size of 2 frames to obtain a spatiotemporal data block sequence. The spatiotemporal data block sequence is input into the input layer of the dilated three-dimensional convolutional neural network. The input layer receives data blocks of size 16×224×224×3 to obtain the initial feature mapping. The initial feature map is processed sequentially through four feature extraction modules. Each module contains two dilated 3D convolutional layers, a batch normalization layer, an activation function layer, and a pooling layer. The dilation rates are 1, 2, 4, and 8, respectively, and the number of output channels are 64, 128, 256, and 512, respectively, to obtain a multi-scale spatiotemporal feature map. A temporal attention mechanism is applied to the multi-scale spatiotemporal feature map to calculate the correlation weights between different time steps, focusing on repetitive and regular behavioral segments to obtain a weighted feature map; Perform a global average pooling operation on the weighted feature map to compress the feature map into a 512-dimensional feature vector, thus obtaining a primary feature representation; The primary feature representation is input into the fully connected layer and reduced to a 256-dimensional vector through a weight matrix mapping to obtain the spatial behavior feature representation.

5. The method for identifying autistic behaviors in children based on data analysis according to claim 1, characterized in that, The process involves constructing a temporal convolutional network model based on the multimodal dataset, analyzing temporal changes and state transition features of behavior, and obtaining dynamic feature vectors of behavior, including: The smartwatch sensor data, positioning trajectory data, audio feature data, and group video data in the multimodal dataset are arranged in chronological order to obtain a time-series data sequence; The time-series data sequence is processed through an input projection layer, and a one-dimensional convolution is used to map the multi-channel input to a unified 64-dimensional feature space to obtain a standardized feature sequence. The standardized feature sequence is sequentially input into four residual temporal convolutional blocks. Each convolutional block contains two one-dimensional convolutional layers, a batch normalization layer, an activation function layer, and a dropout layer. The dilation rates are 1, 2, 4, and 8, respectively, to obtain multi-scale temporal features. Causal convolution design is applied to the multi-scale temporal features, and padding is used to ensure that the output length is consistent with the input length to obtain temporal correlation data; Based on the aforementioned temporal correlation data, an adaptive max pooling operation is performed to compress the variable-length temporal features into a fixed-dimensional representation, resulting in a compressed feature vector. The compressed feature vector is mapped to a 128-dimensional vector through a fully connected layer, and regularization constraints are applied to obtain the dynamic feature vector of the behavior.

6. The method for identifying autistic behaviors in children based on data analysis according to claim 1, characterized in that, The process of fusing the behavioral time-series feature matrix, the spatial behavioral feature representation, and the behavioral dynamic feature vector, and obtaining a comprehensive feature vector through feature concatenation and dimensionality reduction by an autoencoder, includes: The behavioral time series feature matrix, the spatial behavioral feature representation, and the behavioral dynamic feature vector are normalized to obtain a standardized feature set; Perform a feature concatenation operation on the standardized feature set to concatenate the 15-dimensional behavioral time series feature matrix, the 256-dimensional spatial behavioral feature representation, and the 128-dimensional behavioral dynamic feature vector into a 399-dimensional joint feature, resulting in a feature concatenation vector. A symmetric autoencoder network is constructed from the concatenated feature vectors. The autoencoder consists of an encoder and a decoder, resulting in an encoding-decoding model. The feature concatenation vector is input into the encoding and decoding model, trained by unsupervised learning, and the network parameters are optimized using the mean squared error loss function to obtain the trained autoencoder. Extract the 64-dimensional representation of the bottleneck layer from the trained autoencoder to obtain a compact feature representation, thus obtaining a compressed feature representation; A feature selection algorithm is applied to the compressed feature representation, and key dimensions are retained based on feature importance scores to obtain the comprehensive feature vector.

7. The method for identifying autistic behaviors in children based on data analysis according to claim 1, characterized in that, The process involves inputting the comprehensive feature vector into a multi-classifier for autism behavior type identification and severity assessment to obtain behavior identification data, including: A random forest classifier is constructed based on the comprehensive feature vector. The number of trees is set to 100, the maximum depth is 10, and the Gini coefficient is used as the splitting criterion to obtain the first classifier model. A support vector machine classifier is constructed based on the comprehensive feature vectors. A radial basis function kernel function is used, with a penalty parameter of 10 and a kernel parameter of 0.01, to obtain the second classifier model. A gradient boosting decision tree classifier is constructed based on the comprehensive feature vectors. The number of trees is set to 100, the learning rate is 0.1, and the maximum depth is 5, resulting in a third classifier model. A neural network classifier is constructed based on the comprehensive feature vector, which contains two hidden layers and has a structure of 64-32-16-3. An activation function and an output layer are used to obtain the fourth classifier model. Weighted voting is performed on the prediction results of the first classifier model, the second classifier model, the third classifier model, and the fourth classifier model. The weights are determined based on the performance of each classifier on the validation set to obtain the behavior type discrimination result. Based on the behavior type discrimination results, a regression network is constructed to output an autism behavior severity score. The score is then mapped to a standardized scale interval to obtain the behavior recognition data.

8. A data analysis-based system for recognizing autistic behaviors in children, characterized in that, For implementing the data analysis-based method for identifying autistic behaviors in children as described in any one of claims 1-7, the data analysis-based system for identifying autistic behaviors in children comprises: The acquisition module is used to collect and preprocess children's behavioral data in a multimodal manner to obtain a multimodal dataset. The multimodal acquisition includes collecting six-axis sensor data, positioning data and audio data through a smartwatch worn by the children, and simultaneously collecting video data of multiple children through multi-target monitoring cameras deployed in the classroom. The extraction module is used to extract nonlinear features from the multimodal dataset to obtain a behavioral time series feature matrix; The processing module is used to input the multimodal dataset into a three-dimensional convolutional neural network for processing, identify repetitive behavior patterns and stereotyped actions, and obtain spatial behavior feature representations. The construction module is used to build a temporal convolutional network model based on the multimodal dataset, analyze the temporal changes and state transition features of behavior, and obtain dynamic feature vectors of behavior. The fusion module is used to fuse the behavior time series feature matrix, the spatial behavior feature representation, and the behavior dynamic feature vector, and obtain a comprehensive feature vector through feature concatenation and dimensionality reduction by an autoencoder. The identification module is used to input the comprehensive feature vector into a multi-classifier to identify autism behavior types and assess severity, thereby obtaining behavior identification data.

9. A data analysis-based device for recognizing autistic behaviors in children, characterized in that, The method includes a memory and a processor, the memory storing a computer program that can run on the processor, and the processor executing the computer program to implement the data analysis-based method for identifying autistic behaviors in children as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is run by a processor, the processor performs the data analysis-based method for identifying autistic behaviors in children as described in any one of claims 1 to 7.

Images