A method and system for evaluating children's interpersonal coordination behavior based on skeleton point extraction

Abstract

The application discloses a child interpersonal coordination behavior evaluation method and system based on skeleton point extraction, and relates to the technical fields of behavior recognition, deep learning and medical auxiliary evaluation. The method collects a video of a subject and a testee cooperating in movement, and obtains a standardized video through cutting, frame synchronization calibration and data enhancement. A key point coordinate is obtained through a pre-trained human skeleton point extraction model, and a skeleton point sequence is obtained through confidence filtering, interpolation completion, normalization and Gaussian filter denoising. A coordination score is quantified from four dimensions of posture similarity, simultaneous movement, rhythm similarity and coordination / smoothness, a comprehensive score is obtained through weighted fusion, and a result is output according to a preset threshold. The system comprises corresponding functional units. The application realizes objective and automatic evaluation, has low computing power requirement and can be deployed in a light weight mode, and is suitable for child interpersonal coordination capability evaluation in primary and family scenes.

Description

Technical Field

[0001] This invention relates to the fields of behavior recognition, deep learning, and medical auxiliary assessment technology, specifically to a method and system for assessing children's interpersonal collaborative behavior based on skeletal point extraction. Technical Background

[0002] Children’s interpersonal collaborative behavior is a core indicator reflecting children’s neurodevelopment level and social interaction ability. Early, accurate and convenient behavioral assessment is a key prerequisite for early intervention of children’s developmental abnormalities. It has a wide range of applications in scenarios such as children’s developmental behavior assessment, primary healthcare screening, and daily developmental monitoring at home. Currently, assessment technologies for children's interpersonal coordination behavior suffer from the following core shortcomings: First, traditional assessments heavily rely on manual scales and subjective observation by professionals, resulting in significant human bias. Furthermore, the process is time-consuming and highly dependent on professional medical resources, making it difficult to meet the large-scale, routine assessment needs of grassroots and family settings. Second, existing automated behavior assessment technologies often focus on single-action recognition, failing to adequately consider the multi-dimensional characteristics of interpersonal coordination behavior. They lack systematic quantification of core coordination indicators such as posture matching, action synchronization, rhythm consistency, and motor coordination, resulting in a single assessment dimension and an inability to comprehensively reflect children's interpersonal coordination abilities. Third, existing computer vision-based posture analysis models are generally computationally complex and bulky, requiring high-performance GPUs to operate, making lightweight deployment on common mobile devices like phones and tablets impossible, thus limiting the technology's widespread application. Fourth, the industry lacks integrated, end-to-end assessment systems, failing to achieve an automated closed loop of "video capture—feature extraction—multi-dimensional quantitative scoring—report generation," resulting in cumbersome operation procedures and high technical requirements for operators. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for assessing children's interpersonal collaborative behavior based on skeletal point extraction, so as to solve the technical problems of existing assessment methods being highly subjective, having a single assessment dimension, requiring high model computing power, and being difficult to automate screening.

[0004] To achieve the above objectives, this invention provides a method for assessing interpersonal cooperative behavior in children with autism based on skeletal point extraction, comprising the following steps:

[0005] Step 1: Coordinated Motion Video Acquisition and Preprocessing. Coordinated motion videos of the experimenter and participants were acquired. The experimenter refers to the facilitator of the coordinated interaction, and the participants refer to the children undergoing the assessment. The videos were cropped to retain the core motor areas of the experimenter and participants. Frame timestamp synchronization calibration was used to eliminate shooting delays, and data augmentation techniques were employed to improve data robustness, resulting in standardized video data.

[0006] Step 2: Human skeleton point extraction and post-processing. Standardized video data is input into a pre-trained human skeleton point extraction model to extract the coordinates of key points of the subjects and subjects; the extracted key point data is then subjected to confidence filtering, missing value completion, coordinate normalization, and motion denoising to obtain a temporally continuous and noise-controlled skeleton point sequence.

[0007] Step 3: Quantitative scoring of multi-dimensional interpersonal collaborative behavior. Based on the preprocessed skeletal point sequence, the quantitative scores of collaborative movement between the experimenter and the subject are calculated in four dimensions: posture similarity, simultaneous movement, rhythm similarity, and coordination / smoothness. The maximum score for each dimension is 10 points.

[0008] Step 4: Comprehensive Assessment and Result Output. The weighted and fused quantitative scores of the four dimensions of coordinated movement are used to calculate the comprehensive assessment score. Based on the preset assessment threshold, the assessment results of the child's interpersonal coordinated behavior are output, and a visual assessment report is generated.

[0009] Preferably, the preprocessing in step 1 specifically includes: cropping irrelevant areas to preserve the motion areas of the subject and the experimenter; eliminating shooting delay through frame timestamp synchronization calibration; and performing data augmentation using random flipping, brightness ±10% adjustment, and ±15° rotation.

[0010] Preferably, the extraction and data post-processing in step 2 specifically includes: filtering key points with visibility confidence scores below 0.7, using linear interpolation to complete missing data; using Min-Max normalization to map coordinates to the [0,1] interval; and using a 5th-order Gaussian filter to remove motion noise (standard deviation σ=1.5).

[0011] As a preferred embodiment, the four-dimensional score calculation method in step 3 is as follows:

[0012] (1) Postural similarity score: Ten key limb segments, such as shoulder-elbow, elbow-wrist, hip-knee, and knee-ankle, were selected. Spatial vectors of the corresponding limb segments for the experimenter and the subject were constructed. The cosine similarity of the corresponding vectors was calculated. The average value of the similarity of all limb segments was taken and normalized to obtain a postural similarity score of 0-10. The calculation formula is as follows:

[0013]

[0014] In the formula, M represents the number of critical limb segments. Let be the cosine similarity of the spatial vectors of the corresponding limb segments in the m-th group.

[0015] (2) Simultaneous motion score: The velocity mutation rate of skeletal points was set to ≥0.3 as the threshold for dividing the action into sub-stages. Based on the velocity mutation rate, the complete action was divided into multiple consecutive action sub-stages, and the start time difference between the experimenter and the subject for the corresponding action sub-stage was calculated. A time difference matrix is ​​constructed, and an exponential decay model is introduced to quantify the time deviation, yielding simultaneous motion scores ranging from 0 to 10 points. The calculation formula is as follows:

[0016]

[0017] In the formula, k is the attenuation coefficient, with a value of 0.8. This represents the time difference between the corresponding action sub-stages, in seconds.

[0018] (3) Rhythm similarity score: Extract the power spectral density (focusing on the frequency range of 0-5Hz) and joint angle change rate of the skeletal point motion velocity sequence, construct the temporal feature vector, use the dynamic time warping (DTW) algorithm to calculate the DTW distance between the temporal feature vectors of the experimenter and the subject, normalize the DTW distance to the [0,1] interval, and map to obtain a rhythm similarity score of 0-10.

[0019] (4) Coordination / Smoothness Score: Calculate the standard deviation of acceleration of the subject's motion trajectory. The mean Euclidean distance d between the experimenter and the subject at corresponding joint points; and the standard deviation of the maximum acceleration. Maximum Euclidean distance After normalizing the two indicators, a weighted fusion is performed to obtain a coordination / smoothness score ranging from 0 to 10. The calculation formula is as follows:

[0020]

[0021] In the formula, 'a' is the weighting coefficient, with a value of 0.5.

[0022] Preferably, in step 4, the quantitative scores of the four dimensions of cooperative motion are used to calculate the comprehensive evaluation score using an equal-weighted fusion method. The calculation formula is as follows:

[0023]

[0024] Preferably, the present invention provides a method and system for assessing children's interpersonal collaborative behavior based on skeletal point extraction, for implementing the above-mentioned assessment method, comprising a data acquisition unit, a preprocessing unit, a skeletal point extraction unit, a multi-dimensional quantification unit, a comprehensive assessment unit, and an output unit connected in sequence via communication:

[0025] The data acquisition unit is used to acquire videos of the coordinated movements of the experimenter and the subject, and supports video acquisition from various devices such as mobile phones, tablets, and cameras;

[0026] The preprocessing unit is used to crop, synchronize and calibrate frame timestamps, and enhance data of the acquired video, and output standardized video data.

[0027] The skeleton point extraction unit integrates a pre-trained human skeleton point extraction model, which is used to extract the coordinates of human key points of the subject and the experimenter frame by frame, and complete data filtering, missing value completion, coordinate normalization and motion denoising processing, and output a temporally continuous skeleton point sequence.

[0028] The dimensional quantification unit is used to calculate the collaborative motion quantification scores of the experimenter and the subject in four dimensions: posture similarity, simultaneous movement, rhythm similarity, and coordination / smoothness, based on the skeletal point sequence.

[0029] The comprehensive evaluation unit is used to weight and fuse the quantitative scores of the four dimensions of collaborative movement to calculate the comprehensive evaluation score, and generate a level evaluation conclusion of interpersonal collaborative behavior based on the preset level evaluation threshold.

[0030] The output unit is used to generate and output a visual evaluation report that includes score curves for each dimension, comprehensive evaluation conclusions, and grade classification explanations.

[0031] In the above technical solution, the present invention provides a method and system for assessing interpersonal collaborative behavior in children with autism based on skeletal point extraction, which has the following beneficial effects: It achieves automated scoring based on human skeletal point data through multi-dimensional quantitative algorithms, eliminating bias caused by subjective human assessment, and ensuring good reproducibility of assessment results; it constructs a four-dimensional quantitative assessment system encompassing posture similarity, simultaneous movement, rhythm similarity, and coordination / smoothness, covering the core characteristics of interpersonal collaborative behavior and overcoming the shortcomings of existing technologies with their single assessment dimension; it uses a lightweight YOLOv11 model to extract skeletal points, with an extraction frame rate ≥25 frames / second, requiring low computing power and running smoothly on mobile devices such as phones and tablets, without requiring high-performance GPU equipment, thus meeting the widespread needs of grassroots and home scenarios; it establishes an automated closed loop of "video acquisition—feature extraction—multi-dimensional quantitative scoring—report generation," with a single 2-minute video assessment taking ≤30 seconds, a low operational threshold, and enabling large-scale routine assessment. Attached Figure Description

[0032] Figure 1 This is a diagram illustrating the effect of video data preprocessing on the experimenter and participants in this invention patent.

[0033] Figure 2 This invention patent involves extracting skeletal points from the experimenter and the subject;

[0034] Figure 3 This invention patent provides a quantitative framework for the four dimensions of coordinated movement in children. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0036] This embodiment is implemented using the Python language and open-source libraries such as OpenCV, SciPy, and PyTorch.

[0037] Specific Implementation of the Method for Assessing Children's Interpersonal Collaborative Behavior

[0038] Step 1: Coordinated Motion Video Acquisition and Preprocessing. Collect coordinated motion videos of the experimenter (facilitator) and subjects (children aged 3-8 years) using a mobile phone, tablet, or ordinary camera device. Video parameters are set as follows: frame rate 30 frames / second, resolution 1920×1080, each video segment 2 minutes long, and interactive actions covering core coordinated movements such as synchronized clapping, mirroring, interactive pointing, and coordinated limb imitation. Preprocess the acquired videos using the OpenCV library, such as... Figure 1 First, the video was cropped to a resolution of 1280×720, and irrelevant background areas were removed, while the core motion areas of the subjects and participants were retained. Frame timestamp alignment calibration was used to eliminate frame delay caused by dual-device or single-device shooting. Data augmentation was performed using random flipping, brightness adjustment of ±10%, and angle rotation of ±15° to improve data robustness and obtain standardized video data.

[0039] Step 2: As Figure 2Human skeleton point extraction and post-processing were performed by calling the Python API of the pre-trained YOLOv11-nano lightweight human pose estimation model, with a model confidence threshold set to 0.7. The pre-processed standardized video was input frame by frame into the model, and the two-dimensional coordinates and visibility confidence of 17 standard human key points (including nose, eyes, ears, shoulders, elbows, wrists, hips, knees, and ankles) of the subjects and examiners were extracted simultaneously. The extracted raw data was stored in npz format for easy subsequent processing. Post-processing of the extracted keypoint data was performed using Pandas and SciPy libraries: First, invalid keypoints with a visibility confidence score below 0.7 were filtered out. For missing keypoint data in consecutive frames, linear interpolation was used to complete them. Min-Max normalization was used to map all keypoint coordinates to the [0,1] interval, eliminating systematic errors caused by differences in shooting distance and human body size. Finally, a 5th-order Gaussian filter (standard deviation σ=1.5) was used to smooth and denoise the motion trajectory, removing high-frequency jitter noise, resulting in a temporally continuous and noise-controlled skeleton point sequence. Testing showed that the skeleton point extraction accuracy in this embodiment was ≥92%.

[0040] Step 3: As Figure 3Multidimensional interpersonal collaborative behavior quantitative scoring: Based on the preprocessed skeletal point sequence, the collaborative movement quantitative scores of four dimensions are calculated respectively. The full score of each dimension is 10 points. The specific calculation method is as follows: (1) Posture similarity score: Select 10 key limb segments such as shoulder-elbow, elbow-wrist, hip-knee, knee-ankle, shoulder-hip, left hip-right hip, etc., construct spatial vectors for each limb segment, calculate the cosine similarity between the experimenter and the subject for the corresponding limb segment vectors, take the average value of all limb segment similarities, and map to obtain a posture similarity score of 0-10 points. (2) Simultaneous movement score: Calculate the movement speed sequence of skeletal points, set the speed mutation rate ≥0.3 as the threshold for dividing the action sub-stage, divide the complete collaborative movement into multiple continuous action sub-stages; calculate the starting time difference Δt between the experimenter and the subject for the corresponding action sub-stages, construct the time difference matrix, introduce the exponential decay model to quantify the time deviation, take the decay coefficient k as 0.8, and obtain a simultaneous movement score of 0-10 points. (3) Rhythm similarity score: Extract the power spectral density (focusing on the 0-5Hz frequency range) and joint angle change rate of the skeletal point motion velocity sequence to construct a multi-dimensional temporal feature vector; use the Dynamic Time Warping (DTW) algorithm called by the SciPy library to calculate the DTW distance between the two sets of temporal feature vectors of the experimenter and the subject, normalize the DTW distance to the [0,1] interval, and map to obtain a rhythm similarity score of 0-10. (4) Coordination / smoothness score: Calculate the acceleration standard deviation σa of the subject's motion trajectory, and the mean Euclidean distance d of the corresponding joint point motion trajectories of the experimenter and the subject; set the maximum acceleration standard deviation. Maximum Euclidean distance After normalizing the two indicators, they are merged with equal weights, and the weight coefficient 'a' is set to 0.5 to obtain a coordination / smoothness score of 0-10.

[0041] Step 4: Comprehensive Assessment and Result Output. A weighted fusion method is used to calculate the comprehensive assessment score, with each of the four dimensions having a weight of 0.25. Based on 54 self-collected valid sample data (28 children with developmental deviations and 26 children with typical development; all data confidentiality agreements were signed with parents), ROC curve analysis was performed to set the optimal assessment thresholds: a comprehensive assessment score ≥7 indicates excellent coordination ability, 4-6 indicates average coordination ability, and ≤3 indicates weak coordination ability. Finally, a visual assessment report is generated, including score curves for each dimension, the comprehensive assessment conclusion, and explanations of the level classification. Testing in this embodiment showed that the complete assessment of a single 2-minute video segment took ≤30 seconds, and the comprehensive assessment accuracy was ≥88%.

[0042] Those skilled in the art will understand that the system module division in the above embodiments is only a logical functional division. In practical applications, the functions of multiple units can be integrated into a single physical processor through computer program instructions.

[0043] The foregoing has only described certain exemplary embodiments of the present invention by way of illustration. Undoubtedly, those skilled in the art can modify the described embodiments in various ways without departing from the spirit and scope of the present invention. Therefore, the foregoing drawings and descriptions are illustrative in nature and should not be construed as limiting the scope of protection of the claims of the present invention.

[0044] This technology is only used for the assessment of interpersonal and social behaviors in children with autism and does not replace professional diagnosis by doctors. Data collection has been approved by the ethics committee and written informed consent has been obtained from guardians. All children's data has been anonymized.

Claims

1. A method and system for assessing children's interpersonal collaborative behavior based on skeletal point extraction, characterized by comprising the following steps: 1.1 Step 1: Collect videos of the coordinated movements of the experimenter and the subject, and perform preprocessing such as cropping, frame timestamp synchronization calibration and data augmentation on the videos to obtain standardized video data; 1.2 Step 2: Input standardized video data into the pre-trained human skeleton point extraction model, and simultaneously extract the coordinates of human key points of the subject and the experimenter. After data filtering, missing value completion, coordinate normalization and motion denoising, a temporally continuous skeleton point sequence is obtained. 1.3 Step 3: Based on the skeletal point sequence, calculate the quantitative scores of collaborative movement between the experimenter and the subject in four dimensions: posture similarity, simultaneous movement, rhythm similarity, and coordination / smoothness. 1.4 Step 4: Weight and fuse the quantitative scores of the four dimensions of collaborative movement to calculate the comprehensive evaluation score, and output the level evaluation result of interpersonal collaborative behavior according to the preset level evaluation threshold.

2. The method for assessing children's interpersonal cooperative behavior according to claim 1, characterized in that, In step 1, the data enhancement includes at least one of random flipping, brightness adjustment, and angle rotation.

3. The method for assessing children's interpersonal cooperative behavior according to claim 1, characterized in that, In step 2, the extracted human body key points are 17 key points covering the head, torso, limbs, etc., and each key point contains coordinate information and visibility confidence.

4. The method for assessing children's interpersonal cooperative behavior according to claim 3, characterized in that, In step 2, the data filtering is to filter key points with a visibility confidence level lower than 0.7; the missing value completion uses linear interpolation; the coordinate normalization uses Min-Max normalization to map the coordinates to the [0,1] interval; and the motion denoising uses a 5th-order Gaussian filter.

5. The method for assessing children's interpersonal cooperative behavior according to claim 1, characterized in that, In step 3, the posture similarity score is calculated by constructing spatial vectors of key limb segments between the experimenter and the subject, calculating the cosine similarity of the corresponding vectors, and taking the average value.

6. The method for assessing children's interpersonal cooperative behavior according to claim 1, characterized in that, In step 3, the simultaneous motion score is calculated by dividing the action into sub-stages based on the velocity mutation rate of skeletal points, calculating the time difference between the corresponding action sub-stages of the experimenter and the subject, and quantifying the deviation by combining an exponential decay model.

7. The method for assessing children's interpersonal cooperative behavior according to claim 1, characterized in that, In step 3, the rhythm similarity score is calculated by extracting the temporal feature vectors of skeletal point movements and using the Dynamic Time Warping (DTW) algorithm to calculate the similarity between the feature vectors of the experimenter and the subject.

8. The method for assessing children's interpersonal cooperative behavior according to claim 1, characterized in that, In step 3, the coordination / smoothness score is obtained by normalizing and weighting the two indicators, which are calculated by the standard deviation of the acceleration of the subject's motion trajectory and the mean of the Euclidean distance between the corresponding joint points of the experimenter and the subject.

9. The method for assessing children's interpersonal cooperative behavior according to claim 1, characterized in that, In step 4, the quantitative scores of the four dimensions of coordinated movement are used to calculate the comprehensive evaluation score using an equal-weighted fusion method; the level evaluation threshold is as follows: a comprehensive evaluation score ≥7 indicates excellent coordinated ability, 4-6 indicates average coordinated ability, and ≤3 indicates weak coordinated ability.

Images