Abnormal limb behavior detection method for autism spectrum disorder based on posture tracking
Children's posture tracking was performed through YOLOv3, HRNet and OSNet, and combined with random forest algorithms, the left and right shoulder, elbow and wrist movement trajectories of children with autism spectrum disorder were detected, solving the lack of professional evaluation and limbs of diagnosis of autism spectrum disorder in the prior art. The problem of insufficient behavioral analysis improves diagnostic accuracy and supports early detection and treatment.
Patent Information
- Application Number
- CN202210014731.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-01-07
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2042-01-07
AI Technical Summary
The existing diagnostic methods for autism spectrum disorders have problems such as insufficient professional assessment staff, complex system, strong subjectivity, and insufficient analysis of children's limb behavior, which is difficult to meet the needs of early diagnosis.
The object detection algorithm YOLOv3, human posture estimation method HRNet and pedestrian recognition network OSNet were used to track children's postures, detect the movement trajectories of left and right shoulders, elbows, and wrists. Combining the mean value of movement speed and standard deviation characteristics, a random forest algorithm was used for comprehensive judgment.
It improves the accuracy of autism spectrum disorder detection, achieves a comprehensive analysis of children's limb behavior, solves the problem of small sample size, and supports early detection and early treatment.
Smart Images

Figure CN114358194B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of video behavior analysis, and in particular relates to a method for detecting abnormal limb behavior of autism spectrum disorder based on posture tracking. Background Art
[0002] Autism spectrum disorder (ASD), also known as autism, is a relatively serious developmental disorder. Autism spectrum disorder patients mainly have three core symptoms: social interaction disorder, communication disorder, narrow interests and stereotyped repetitive behaviors. According to statistics from the World Health Organization, one in every 160 children in the world has an autism spectrum disorder. In China, the number of children with autism spectrum disorder increases by 200,000 each year. So far, the number of children with autism spectrum disorder in China has far exceeded 10 million. At present, clinical diagnosis of autism spectrum disorder is generally between 3 and 4 years old. Therefore, through video behavior analysis and deep learning technology, the detection of abnormal limb behavior in children with autism spectrum disorder in the early stage is studied to help doctors make auxiliary judgments and achieve "early detection and early treatment", so as to take different treatment methods for early intervention and alleviate the symptoms of children with autism spectrum disorder, which has important clinical and social significance.
[0003] At present, there are three types of methods used to diagnose children with autism spectrum disorders. The first method is the traditional medical method. The medical diagnosis of children with autism spectrum disorders mainly relies on parents filling out questionnaires and taking different assessment scales to evaluate children for comprehensive diagnosis. Among them, ADOS-2 is the most authoritative assessment method at present. Different levels of assessment scales are adopted according to age and language level. By observing children's performance in games and the use of materials, children with autism spectrum disorders are evaluated and divided into three levels: mild, moderate and severe. Professionally trained assessment physicians are required to judge the level of children, but the number of professional assessors is small, which makes it difficult to meet the needs of early diagnosis of patients with autism spectrum disorders. In addition, the system is complex, time-consuming and subjective. The second method is acceleration measurement, which requires children to wear multiple sensor devices to extract parameters, but this method will cause discomfort and rejection to children, thereby affecting the measurement results of children. The third method is to use computer vision technology, but this method mostly focuses on eye tracking and facial recognition, ignoring the analysis of children's physical behavior. Children with autism spectrum disorders sometimes show subtle or obvious behavioral confusion, so physical behavior analysis of children is conducive to the early detection of autism spectrum disorders. Summary of the invention
[0004] In order to solve the above-mentioned problems existing in the prior art, the purpose of the present invention is to provide a method for detecting abnormal limb behavior in autism spectrum disorder based on posture tracking, which detects the movement trajectories of the left and right shoulders, elbows, and wrists of children, obtains more comprehensive information than single joint detection, and extracts two types of features related to movement speed for comprehensive judgment, thereby improving the accuracy of autism spectrum disorder detection.
[0005] The present invention uses the target detection algorithm YOLOv3, the human posture estimation method HRNet and the pedestrian re-identification network OSNet to track the human posture of children, calculates the mean speed and the standard deviation of the motion trajectory of the left and right shoulders, elbows and wrists of the children, uses the random forest algorithm to perform leave-one-out cross-validation on the obtained features, and makes a comprehensive judgment on the classification results to determine whether the child is normal, including the following steps:
[0006] 1.1 Child video collection: Collect surveillance videos of doctors diagnosing children and obtain video data of children’s physical behavior during the interaction between children and doctors, including the following steps:
[0007] 1.1.1 For the collected surveillance videos, screen out the videos of children with the ADOS-2 assessment scale and the assessment levels of T, 1, and 2, and roughly capture the surveillance videos after the doctor makes the same body movement;
[0008] 1.1.2 Preprocess the children's video obtained in step 1.1.1, convert it into frame images, select 750 consecutive frame images without noise interference, and uniformly name them;
[0009] 1.1.3 Convert the 750 frames of images obtained in step 1.1.2 into a 30-second children's video with a sampling frame rate of 25 frames per second;
[0010] 1.2 Tracking the movement trajectory of children: For the child video obtained in step 1.1.3, the movement trajectory of the child's left and right shoulders, elbows, and wrists is obtained using the human posture tracking algorithm, including the following steps:
[0011] 1.2.1 Human frame detection: Perform reasonable frame extraction operations on the preprocessed frame images obtained in step 1.1.2 to create a human frame detection data set. A total of 1223 frame images are collected, and then manually annotated to train the YOLOv3 network. The child video obtained in step 1.1.3 is input into the retrained YOLOv3 network to obtain the detection frames of the child, doctor, and parent in the video, and then crop them to obtain sub-images;
[0012] 1.2.2 Human joint point detection: For the sub-image obtained in step 1.2.1, use the HRNet human posture estimation method to detect human joint points and obtain the position coordinates and confidence of each joint point;
[0013] 1.2.3 Multi-target tracking: For the sub-image obtained in step 1.2.1, use the OSNet person re-identification network to extract the apparent features of the sub-image, calculate the probability that different people belong to the same target, perform data association, and assign a digital ID to each person in the video frame image;
[0014] 1.2.4 Through steps 1.2.1, 1.2.2 and 1.2.3, the ID information of the child in the video can be obtained, and the movement trajectory of the left and right shoulder, elbow and wrist joints of the child can be tracked;
[0015] 1.3 Analyze the motion trajectory: Calculate the motion velocity of the left and right shoulder, elbow, and wrist joints of the child obtained by tracking in step 1.2.4, and extract the mean value and standard deviation of the motion velocity as feature vectors, including the following steps:
[0016] 1.3.1 Screen the motion trajectories of the left and right shoulder, elbow, and wrist joints of children, and remove samples with less than 700 frames and obvious human posture tracking errors;
[0017] 1.3.2 For the samples obtained in step 1.3.1, extract the position information of the x-axis and y-axis coordinate changes of the six joints of the left and right shoulders, elbows, and wrists, and calculate the movement speed;
[0018] 1.3.3 For the movement speed obtained in step 1.3.2, calculate the mean movement speed of the six joints of the left and right shoulders, elbows, and wrists, and obtain the fused speed mean feature vector;
[0019] 1.3.4 For the motion speed obtained in step 1.3.2, calculate the standard deviation of the motion speed of the six joints of the left and right shoulders, elbows, and wrists, and obtain the fused speed standard deviation feature vector;
[0020] 1.4 The fused velocity mean feature vector obtained in step 1.3.3 and the fused velocity standard deviation feature vector obtained in step 1.3.4 are trained using random forest, including the following steps:
[0021] 1.4.1 The samples with mild diagnosis results on the doctor's evaluation scale are regarded as normal samples, and the samples with moderate and severe diagnosis results on the doctor's evaluation scale are regarded as abnormal samples. The normal sample label is set to 1, and the abnormal sample label is set to -1;
[0022] 1.4.2 Leave-one-out cross validation is used during training, that is, when the total number of samples is N, each sample will be used as a validation set, and the remaining N-1 samples will be used as training sets;
[0023] 1.4.3 Random forest is a classifier composed of multiple decision trees. The construction of random forest requires two aspects: random selection of data and random selection of features. The final classification result output by random forest is determined by the mode of the classification results output by each decision tree.
[0024] 1.4.4 For the fused velocity mean feature vector obtained in step 1.3.3 and the fused velocity standard deviation feature vector obtained in step 1.3.4, construct sample sets respectively, set 2 / 3 of the total number of samples of the original sample set to be replaced each time as the sub-sample set, and obtain two optimal random forest models by adjusting the number of decision trees contained in the random forest, the number of randomly selected features and the maximum depth of each decision tree;
[0025] 1.5 Comprehensive judgment of abnormal limb behavior in autism spectrum disorder: Based on the two best random forest models obtained in step 1.4.4, make a comprehensive judgment on the classification results and stipulate a discrimination rule: if the classification results obtained by the two best random forest models are both -1, the child is judged to have autism spectrum disorder; if at least one of the classification results obtained by the two best random forest models is 1, the child is judged to be normal.
[0026] The calculated motion speed described in step 1.3.2 is calculated using the following formula:
[0027] v=s / t
[0028] Among them: s is the distance, t is the time;
[0029] According to the position information of the x- and y-axis coordinate changes of the six joints of the left and right shoulders, elbows, and wrists obtained in step 1.3.2, the movement of children is a non-uniform curved movement. The frame rate of the children's video is 25 frames / s. To obtain the speed between two adjacent frames, the time interval is set to 1 for the convenience of calculation, and the motion trajectory between two adjacent frames is quantized into an approximate straight line. The speed calculation formula obtained by the Pythagorean theorem is:
[0030]
[0031] Wherein: subscript t+1 represents the t+1th frame information, and subscript t represents the tth frame information.
[0032] The beneficial effects of the present invention are:
[0033] 1. The present invention adopts a method for detecting abnormal limb behavior in autism spectrum disorder based on posture tracking. First, the collected monitoring video of the child receiving diagnosis is preprocessed, and the child's posture is tracked using the target detection algorithm YOLOv3, the human posture estimation method HRNet and the pedestrian re-identification network OSNet. For the target detection algorithm YOLOv3, the preprocessed video frame image is reasonably extracted to construct a human detection frame data set with a data volume of 1223. YOLOv3 is retrained to improve the effect of human posture tracking. The motion trajectory obtained by human posture tracking is screened to remove samples with a frame number of less than 700 frames and obviously inaccurate human posture tracking. The mean and standard deviation of the motion speed of the samples that meet the conditions are calculated, and the obtained features are trained with random forest to obtain two optimal random forest models. Finally, the classification results are comprehensively judged to obtain the best recognition result.
[0034] 2. In addition to the existing eye tracking and facial recognition methods, the present invention uses limb behavior detection to assist in the judgment of autism spectrum disorder. It combines the information of six body joints of the left and right shoulders, elbows, and wrists to obtain more comprehensive information than single joint information. At the same time, the obtained features are cross-validated with a random forest, which solves the problem of small sample number, and makes a comprehensive judgment on the classification results to improve the accuracy. It assists in judging whether a child suffers from autism spectrum disorder, which can achieve "early detection and early treatment" and is of great significance to alleviating autism spectrum disorder. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] Figure 1 A flowchart of a method for detecting abnormal limb behavior in autism spectrum disorder based on posture tracking;
[0036] Figure 2 This is the effect diagram of human posture tracking in single frame images;
[0037] Figure 3 It is a trajectory image of a child's left shoulder movement speed detection;
[0038] Figure 4 Flowchart for determining whether a child has autism spectrum disorder. DETAILED DESCRIPTION
[0039] The implementation process of the present invention is further described below in conjunction with the accompanying drawings.
[0040] like Figure 1 As shown, the present invention provides a method for detecting abnormal limb behavior in autism spectrum disorder based on posture tracking, comprising the following steps:
[0041] 1.1 Child video collection: Collect surveillance videos of doctors diagnosing children and obtain video data of children’s physical behavior during the interaction between children and doctors, including the following steps:
[0042] 1.1.1 The collected surveillance videos were screened for children with an ADOS-2 assessment scale and assessment levels of T, 1, and 2. Most of them were children with an assessment scale level of 1, aged 31 months and above, who did not often use phrases. After the doctor performed the same physical activity, the surveillance video was roughly intercepted, and the time was controlled between 31s and 35s;
[0043] 1.1.2 Preprocess the children's video obtained in step 1.1.1, convert it into frame images, select 750 consecutive frame images without noise interference, and uniformly name them;
[0044] 1.1.3 Convert the 750 frames of images obtained in step 1.1.2 into a 30-second children's video with a sampling frame rate of 25 frames per second;
[0045] 1.2 Tracking the movement trajectory of children: For the child video obtained in step 1.1.3, the movement trajectory of the child's left and right shoulders, elbows, and wrists is obtained using the human posture tracking algorithm, including the following steps:
[0046] 1.2.1 Human frame detection: For the children's video obtained in step 1.1.3, use YOLOv3 to perform human frame detection, obtain the detection frame of all people in the video frame image, crop it, and obtain a sub-image. The scene contains children, doctors and parents, so it is necessary to adopt human posture estimation and multi-target tracking algorithms to obtain the human posture of children. In addition, the scene is small, and it is easy for children to be too close to doctors or parents, resulting in inaccurate multi-target tracking. In order to improve the accuracy of human posture tracking, the pre-processed frame image obtained in step 1.1.2 is reasonably extracted to produce a human frame detection data set. The number of frame images collected is 1223, and then manually labeled, the human detection frame is framed as the true value, and the YOLOv3 network is trained to improve the detection effect of the human detection frame, thereby improving the effect of human posture tracking;
[0047] 1.2.2 Human joint point detection: For the sub-graph obtained in step 1.2.1, the HRNet human posture estimation method is used to detect human joint points. HRNet connects the high-resolution to low-resolution sub-networks in parallel, rather than serially, to maintain high-resolution features, and obtains more accurate position information by repeating multi-scale information fusion. The present invention directly uses the trained weights to obtain the position coordinates and confidence of the 17 joint points of each person in the video frame image;
[0048] 1.2.3 Multi-target tracking: The workflow of the multi-target tracking algorithm includes five steps, namely, the original frame of a given video; running a human detector to obtain a human detection frame; for each detected human frame, calculating different features, usually motion features and appearance features; calculating similarity; performing association, and assigning a digital ID to each person. The present invention uses the OSNet pedestrian re-identification network to extract the appearance features of the sub-graph obtained in step 1.2.1, and assigns a digital ID to each person in the video frame image;
[0049] 1.2.4 Through steps 1.2.1, 1.2.2 and 1.2.3, the ID information of the child in the video can be obtained, and the movement trajectory of the left and right shoulder, elbow and wrist joints of the child can be tracked;
[0050] 1.3 Analyze the motion trajectory: Calculate the motion velocity of the left and right shoulder, elbow, and wrist joints of the child obtained by tracking in step 1.2.4, and extract the mean value and standard deviation of the motion velocity as feature vectors, including the following steps:
[0051] 1.3.1 Screen the motion trajectories of the left and right shoulder, elbow, and wrist joints of children, and remove samples with less than 700 frames and obvious human posture tracking errors;
[0052] 1.3.2 For the samples obtained in step 1.3.1, extract the position information of the x-axis and y-axis coordinate changes of the six joints of the left and right shoulders, elbows, and wrists, and calculate the movement speed using the following formula:
[0053] v=s / t
[0054] Among them: s is the distance, t is the time;
[0055] According to the position information of the x- and y-axis coordinate changes of the six joints of the left and right shoulders, elbows, and wrists, it can be known that the movement of children is a non-uniform curved movement. The frame rate of the children's video is 25 frames / s. To obtain the speed between two adjacent frames, the time interval is set to 1 for the convenience of calculation, and the motion trajectory between two adjacent frames is quantized into an approximate straight line. The speed calculation formula obtained by the Pythagorean theorem is as follows:
[0056]
[0057] Where: subscript t+1 represents the t+1th frame information, subscript t represents the tth frame information;
[0058] 1.3.3 For the movement speed obtained in step 1.3.2, calculate the mean movement speed of the six joints of the left and right shoulders, elbows, and wrists, and obtain the fused speed mean feature vector;
[0059] 1.3.4 For the motion speed obtained in step 1.3.2, calculate the standard deviation of the motion speed of the six joints of the left and right shoulders, elbows, and wrists, and obtain the fused speed standard deviation feature vector;
[0060] 1.4 The fused velocity mean feature vector obtained in step 1.3.3 and the fused velocity standard deviation feature vector obtained in step 1.3.4 are trained using random forest, including the following steps:
[0061] 1.4.1 The samples with mild diagnosis results on the doctor's evaluation scale are regarded as normal samples, and the samples with moderate and severe diagnosis results on the doctor's evaluation scale are regarded as abnormal samples. The normal sample label is set to 1, and the abnormal sample label is set to -1;
[0062] 1.4.2 Leave-one-out cross validation is used during random forest training, that is, when the total number of samples is N, each sample will be used as a validation set, and the remaining N-1 samples will be used as training sets, which effectively solves the problem of small number of samples;
[0063] 1.4.3 Random forest is a classifier composed of multiple decision trees, and each decision tree is not related to each other. The final classification result of random forest is determined by the mode of classification results output by each decision tree. The construction of random forest needs to consist of two aspects: random selection of data and random selection of features. The random selection of data refers to randomly selecting n samples from the original sample set with replacement as sub-sample sets; the random selection of features refers to randomly selecting d features from all features as the splitting features of each decision tree, where the elements in each sub-sample set can be repeated, and the elements in the same sub-sample set can also be repeated. The number of sub-sample sets finally constructed is the same as the number of decision trees contained in the random forest;
[0064] 1.4.4 For the fused velocity mean feature vector obtained in step 1.3.3 and the fused velocity standard deviation feature vector obtained in step 1.3.4, construct sample sets respectively, set 2 / 3 of the total number of samples of the original sample set to be replaced each time as the sub-sample set, and obtain two optimal random forest models by adjusting the number of decision trees contained in the random forest, the number of randomly selected features and the maximum depth of each decision tree;
[0065] 1.5 Comprehensive judgment of abnormal limb behavior in autism spectrum disorder: Based on the two best random forest models obtained in step 1.4.4, make a comprehensive judgment on the classification results and stipulate a discrimination rule: if the classification results obtained by the two best random forest models are both -1, the child is judged to have autism spectrum disorder; if at least one of the classification results obtained by the two best random forest models is 1, the child is judged to be normal.
[0066] The performance analysis of the abnormal limb behavior detection method for autism spectrum disorder based on posture tracking provided by the present invention is as follows: three metrics for judging the performance of the method in this paper are defined: sensitivity (SE), specificity (SP), and accuracy (AC). The specific formula is as follows:
[0067]
[0068]
[0069]
[0070] Among them: Definition True Positive (TP): Children with ASD are judged as children with ASD. False Positive (FP): Normal children are judged as children with ASD. True Negative (TN): Normal children are judged as normal children. False Negative (FN): Children with ASD are judged as normal children. The results are shown in the following table:
[0071]
[0072] The present invention uses the speed mean and speed standard deviation characteristics for comprehensive judgment. As shown in Table 1, the obtained sensitivity (SE) is 61.54%, the specificity (SP) is 65.22%, and the accuracy (AC) is 62.90%.
Claims
1. A method for detecting abnormal limb behavior in autism spectrum disorder based on posture tracking, characterized in that it comprises the following steps: 1.1 Child video collection: Collect surveillance videos of doctors diagnosing children and obtain video data of children’s physical behavior during the interaction between children and doctors, including the following steps: 1.1.1 For the collected surveillance videos, screen out the videos of children with the ADOS-2 assessment scale and the assessment levels of T, 1, and 2, and roughly capture the surveillance videos after the doctor makes the same body movement; 1.1.2 Preprocess the children's video obtained in step 1.1.1, convert it into frame images, select 750 consecutive frame images without noise interference, and uniformly name them; 1.1.3 Convert the 750 frames of images obtained in step 1.1.2 into a 30-second children's video with a sampling frame rate of 25 frames per second; 1.2 Tracking the movement trajectory of children: For the child video obtained in step 1.1.3, the movement trajectory of the child's left and right shoulders, elbows, and wrists is obtained using the human posture tracking algorithm, including the following steps: 1.2.1 Human frame detection: Perform reasonable frame extraction operations on the preprocessed frame images obtained in step 1.1.2 to create a human frame detection data set. A total of 1223 frame images are collected, and then manually annotated to train the YOLOv3 network. The child video obtained in step 1.1.3 is input into the retrained YOLOv3 network to obtain the detection frames of the child, doctor, and parent in the video, and then crop them to obtain sub-images; 1.2.2 Human joint point detection: For the sub-image obtained in step 1.2.1, use the HRNet human posture estimation method to detect human joint points and obtain the position coordinates and confidence of each joint point; 1.2.3 Multi-target tracking: For the sub-image obtained in step 1.2.1, use the OSNet person re-identification network to extract the apparent features of the sub-image, calculate the probability that different people belong to the same target, perform data association, and assign a digital ID to each person in the video frame image; 1.2.4 Through steps 1.2.1, 1.2.2 and 1.2.3, the ID information of the child in the video can be obtained, and the movement trajectory of the left and right shoulder, elbow and wrist joints of the child can be tracked; 1.3 Analyze the motion trajectory: Calculate the motion velocity of the left and right shoulder, elbow, and wrist joints of the child obtained by tracking in step 1.2.4, and extract the mean value and standard deviation of the motion velocity as feature vectors, including the following steps: 1.3.1 Screen the motion trajectories of the left and right shoulder, elbow, and wrist joints of children, and remove samples with less than 700 frames and obvious human posture tracking errors; 1.3.2 For the samples obtained in step 1.3.1, extract the position information of the x-axis and y-axis coordinate changes of the six joints of the left and right shoulders, elbows, and wrists, and calculate the movement speed; 1.3.3 For the movement speed obtained in step 1.3.2, calculate the mean movement speed of the six joints of the left and right shoulders, elbows, and wrists, and obtain the fused speed mean feature vector; 1.3.4 For the motion speed obtained in step 1.3.2, calculate the standard deviation of the motion speed of the six joints of the left and right shoulders, elbows, and wrists, and obtain the fused speed standard deviation feature vector; 1.4 The fused velocity mean feature vector obtained in step 1.3.3 and the fused velocity standard deviation feature vector obtained in step 1.3.4 are trained using random forest, including the following steps: 1.4.1 The samples with mild diagnosis results on the doctor's evaluation scale are regarded as normal samples, and the samples with moderate and severe diagnosis results on the doctor's evaluation scale are regarded as abnormal samples. The normal sample label is set to 1, and the abnormal sample label is set to -1; 1.4.2 Leave-one-out cross validation is used during training, that is, when the total number of samples is N, each sample will be used as a validation set, and the remaining N-1 samples will be used as training sets; 1.4.3 Random forest is a classifier composed of multiple decision trees. The construction of random forest requires two aspects: random selection of data and random selection of features. The final classification result output by random forest is determined by the mode of the classification results output by each decision tree. 1.4.4 For the fused velocity mean feature vector obtained in step 1.3.3 and the fused velocity standard deviation feature vector obtained in step 1.3.4, construct sample sets respectively, set 2 / 3 of the total number of samples of the original sample set to be replaced each time as the sub-sample set, and obtain two optimal random forest models by adjusting the number of decision trees contained in the random forest, the number of randomly selected features and the maximum depth of each decision tree; 1.5 Comprehensive judgment of abnormal limb behavior in autism spectrum disorder: Based on the two best random forest models obtained in step 1.4.4, make a comprehensive judgment on the classification results and stipulate a discrimination rule: if the classification results obtained by the two best random forest models are both -1, the child is judged to have autism spectrum disorder; if at least one of the classification results obtained by the two best random forest models is 1, the child is judged to be normal.
2. The method for detecting abnormal limb behavior in autism spectrum disorder based on posture tracking according to claim 1, It is characterized in that The calculated motion speed described in step 1.3.2 is calculated using the following formula: v=s / t Among them: s is the distance, t is the time; According to the position information of the x- and y-axis coordinate changes of the six joints of the left and right shoulders, elbows, and wrists obtained in step 1.3.2, the movement of children is a non-uniform curved movement. The frame rate of the children's video is 25 frames / s. To obtain the speed between two adjacent frames, the time interval is set to 1 for the convenience of calculation, and the motion trajectory between two adjacent frames is quantized into an approximate straight line. The speed calculation formula obtained by the Pythagorean theorem is: Wherein: subscript t+1 represents the t+1th frame information, and subscript t represents the tth frame information.
