Development coordination obstacle single-hand fine motion evaluation method based on visual behavior analysis

Abstract

The invention discloses a development coordination obstacle single-hand fine motion evaluation method based on visual behavior analysis. The problems that an existing development coordination obstacle single-hand fine motion evaluation method is insufficient in objectivity, poor in accuracy and low in generalization are mainly solved. The implementation scheme is as follows: constructing a standard nail insertion task video data set and dividing the standard nail insertion task video data set into a training data set and a test data set; the method comprises the following steps: constructing a first-stage repeated action positioning network F1 for positioning repeated actions in a video; constructing a time sequence area network F2 of a second stage, wherein the time sequence area network F2 is used for evaluating the positioned action fragments; respectively training the two networks F1 and F2 through gradient descent by using a training data set; and respectively inputting a test data set into the two trained networks to obtain an evaluation result. According to the evaluation method based on vision, fine-grained evaluation of single-hand fine actions is achieved, the evaluation accuracy and efficiency are effectively improved, and the method can be used for large-scale screening of the developmental coordination disorder DCD.

Description

Technical Field

[0001] The present invention belongs to the technical field of pattern recognition and intelligent systems, and mainly relates to an evaluation method for fine motor skills of one hand in developmental coordination disorder (DCD), which can be used for large-scale screening of DCD, and is helpful for early auxiliary diagnosis and intervention of DCD to avoid further deterioration of symptoms. Background Art

[0002] Developmental coordination disorder, also known as motor coordination disorder, is a developmental disorder that affects the development of children's motor skills. The prevalence rate in preschool and school-age children is approximately between 2% and 10%. Children with the disorder usually show clumsy movements, uncoordinated hands and feet, and are unable to complete specific motor tasks as smoothly as their peers. If not diagnosed and intervened in time, the symptoms of most children with the disorder will continue into adolescence or adulthood, seriously affecting the patient's social, academic and emotional development, and bringing some secondary health problems, such as obesity and depression.

[0003] At present, the clinical diagnosis of DCD is made by doctors based on diagnostic reference books. Doctors score the performance of the test subjects by observing the accuracy, speed and fluency of the test subjects in completing specific tasks. Lower scores indicate possible motor coordination problems. At present, the mainstream diagnostic reference books include Movement Assessment Battery for Children-Second Edition (Movement ABC-2), Bruininks-Oseretsky Test of Motor Proficiency-Second Edition (BOT-2), Developmental Coordination Disorder Questionnaire (DCDQ), etc. Among them, Movement ABC-2 is the most widely used standardized DCD diagnostic assessment tool and is considered the "gold standard" for determining children's motor performance. According to the Movement ABC-2 reference book, fine motor skill assessment tasks include one-handed operation tasks, two-handed coordination tasks, and handwriting trajectory drawing tasks. Among them, one-handed operation is the basis of other fine movements. The development of children's fine movements usually starts with simple one-handed movements, and then gradually develops more complex two-handed movements and trajectory drawing skills. Therefore, the practice and mastery of one-handed fine movements is the key to the development of fine motor skills.

[0004] The main function of one-handed manipulation tasks is to assess the fine motor skills and hand-eye coordination ability of children or individuals to complete specific tasks with one hand. These tasks can help understand the individual's hand dexterity, strength control and accuracy. For example, in the nail-insertion task specified in the MovementABC-2 reference book, the subject needs to insert the yellow nails into the blue board one by one, and the assessor records the total time to complete the insertion of all 12 nails to assess the skill level. Each subject undergoes two rounds of formal testing for each hand, and there is a round of pre-practice limited to inserting three nails. The task is designed to carefully assess the subject's one-handed fine motor skills and the efficiency of their execution. Generally speaking, the process of a complete one-handed manipulation task includes repetitive movements. By repeatedly completing the same movement, the randomness that may occur in a single operation can be avoided, and the memory ability of the individual's hand muscles and the ability to maintain consistency of movement over a certain period of time can be accurately assessed.

[0005] In the prior art, there are usually two methods for evaluating single-hand fine movements: a method based on manual evaluation and a method based on computer vision.

[0006] The manual evaluation method is for medical professionals or physicians to score the performance of the test subjects through visual observation and based on diagnostic reference books and their own experience. This method requires one-to-one recording, which is labor-intensive, inefficient, and highly subjective. It is easy to cause supervisor fatigue, which may lead to inconsistent judgment standards and excessive scoring errors, resulting in inaccurate evaluation results. At the same time, since the data of the test subjects are all recorded on paper, there is a risk of missing data.

[0007] With the development of modern computer technology and artificial intelligence technology, computer vision-based methods have been widely used. This type of method first collects information such as videos and images of the fine movements of one hand through non-contact data collection, and then designs intelligent algorithms to intelligently process and analyze the image information, thereby realizing the evaluation of fine movements of one hand. Li et al. proposed a method based on template matching in the article "Automated Fine Motor Evaluation for Developmental Coordination Disorder". For the task of inserting nails, it extracts the appearance of nails as a template image, matches it in the nail insertion video, obtains the number of nails in each frame of the video, and obtains the total time to complete the nail insertion task in the video through a filtering strategy, and then obtains the score of the nail insertion task according to the standards of the MovementABC-2 reference book. However, the generalization of this method is insufficient and can only be used for nail insertion tasks, and cannot be extended to other evaluation tasks of fine movements of one hand.

[0008] Cui et al. proposed a method for evaluating fine one-handed movements based on skeletal data in the article "Study on Assessment Methods of Developmental Coordination Disorder in Children". The method first uses a posture estimation algorithm to extract the overall skeletal data of the human body, then extracts the hand skeletal coordinate data, and directly evaluates fine one-handed movements through a three-dimensional convolutional neural network and a long short-term memory network. However, this method ignores the repeatability of fine one-handed movements and directly evaluates the complete movement execution process, lacking processing of fine movement details. In addition, since this method only uses skeletal data and lacks natural image information, it is difficult to extract the joint features of the hand and the object for one-handed movements that interact with objects, resulting in low evaluation accuracy. Summary of the invention

[0009] The purpose of the present invention is to address the deficiencies of the above-mentioned prior art and propose a method for evaluating fine motor skills of one hand in developmental coordination disorder based on repeated action positioning and temporal area enhancement, so as to improve the objectivity, generalization and accuracy of fine motor skills evaluation of one hand.

[0010] To achieve the above object, the technical solution of the present invention includes the following:

[0011] 1. A method for evaluating fine motor skills of one hand in developmental coordination disorder based on visual behavior analysis, characterized in that it comprises the following steps:

[0012] (1) Construct a single-hand fine motor assessment task dataset D based on the developmental coordination disorder assessment tool book MovementABC-2 peg :

[0013] Data collection. Children aged 7 to 10 were recruited for data collection. Videos of the children performing the nail-insertion task in the one-hand fine motor assessment task were recorded with a camera, and the videos were annotated according to the MovementABC-2 reference book.

[0014] Data partition. peg The data set is divided into training set D in a ratio of 8:2 t and the test set D v ;

[0015] (2) Constructing a single-hand fine motor assessment network F for developmental coordination disorders, including a repetitive action localization network F1 and a spatiotemporal area enhancement network F2;

[0016] The repeated action localization network F1 is used to locate the start and end time regions of repeated actions in the input video V in the first stage, and obtain an action localization set S consisting of the start frame and end frame positions of each repeated action:

[0017] S={[B1,E1],[B2,E2],…[B n ,E n ],…,[B N ,E N ]}

[0018] Among them, B n and E n are the start frame and end frame of the nth action in the input video V, respectively, and the value of n ranges from 1 to N;

[0019] The spatiotemporal region enhancement network F2 is used to perform the video segment set V s Each video segment v n The included single-handed action skills are evaluated to obtain the repeated action evaluation result set R s :

[0020] R s ={r1,r2,…r n ,…,r N};

[0021] Among them, V s ={v1,v2,…v n ,…,v N} is the set of video clips obtained by segmenting the result set S of the repeated action localization stage, v n is the nth video segment after segmentation, r n is the evaluation result of the nth video clip, its value is 0: non-standard action, or 1: standard action, the size of n is from 1 to N;

[0022] (3) Using training set D t The repeated action localization network F1 and the spatiotemporal region enhancement network F2 in the single-hand fine action evaluation network are iteratively trained by the gradient descent method until their respective loss functions converge, and the trained single-hand fine action evaluation network F is obtained;

[0023] (4) Test data set D v Input into the trained single-hand fine movement evaluation network to obtain the estimation result corresponding to the test data.

[0024] 2. A developmental coordination disorder single-hand fine motor assessment system based on visual behavior analysis, characterized in that it is based on the developmental coordination disorder assessment tool book MovementABC-2, and includes: an information entry unit, a data acquisition unit, an automatic assessment unit, a result feedback unit and a display unit;

[0025] The information input unit is used to input the basic information of the test subject, including name, number, birthday, height, weight, dominant hand and gender, and store the information persistently in the database;

[0026] The data acquisition unit is used to automatically call the Azure Kinect camera to collect the process of the test subject performing the one-handed task and save it to a designated folder;

[0027] The automatic evaluation unit is used to automatically call the evaluation algorithm to perform a two-stage evaluation on the video data:

[0028] In the first stage, repeated action positioning is performed on the video data. In the second stage, the video is segmented into video segments according to the repeated positioning results. The video segments are then evaluated to obtain a set containing the evaluation results of each video segment.

[0029] The evaluation result of the complete video is obtained according to whether the result set contains non-standard actions;

[0030] If the result set does not contain non-standard movements, the evaluation result of the complete video is qualified and recorded as 1. The video is scored according to the completion time of the single-handed task in the video and the standards of the MovementABC-2 reference book.

[0031] Otherwise, it is considered unqualified and recorded as 0.

[0032] Save the evaluation results into the database;

[0033] The result feedback unit is used to obtain the video evaluation results recorded in the database and feed them back to the test subject;

[0034] The display unit is used to display the results of each functional unit.

[0035] Compared with the prior art, the present invention has the following advantages:

[0036] First, the objectivity of fine-grained evaluation of single-hand fine movements is improved:

[0037] The present invention uses an automated evaluation method based on computer vision, inputs a single-hand fine movement video to be evaluated, and outputs an evaluation result of the video after automated evaluation. No human participation in the evaluation is required, thus eliminating the problems of subjectivity and inaccurate scoring caused by methods based on manual evaluation.

[0038] Second, the accuracy and generalization of fine-grained evaluation of single-hand fine movements are improved:

[0039] The present invention uses the repetitive nature of single-hand fine movements to propose a fine-grained evaluation method for single-hand fine movements in developmental coordination disorders, that is, in the first stage, repeated movements are located to find the time interval of each movement. In the second stage, the complete video is first decomposed into segments containing only a single movement based on the positioning results, and then each action segment obtained by repeated movement location is used as a sample for separate evaluation, and the spatiotemporal features are enhanced to obtain more detailed features of hand changes. Through these two stages, more accurate evaluation results can be obtained. In addition, this method can also be extended to other single-hand fine movement evaluation tasks with repetitive properties, improving the generalization of single-hand fine movement evaluation.

[0040] Third, the efficiency of single-hand fine motor assessment is improved:

[0041] The present invention constructs a one-hand fine motor assessment system for developmental coordination disorder based on visual behavior analysis, realizes automated basic information collection, video data collection and movement assessment, improves the efficiency of one-hand fine motor assessment, and in addition, by storing the collected basic information and videos in a database, data traceability is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a flow chart for realizing the method of the present invention;

[0043] Figure 2 It is a structural diagram of the first-stage action localization network F1 constructed in the method of the present invention;

[0044] Figure 3 It is a structural diagram of the second-stage sequential regional enhancement network F2 constructed in the method of the present invention;

[0045] Figure 4 It is a structural block diagram of the system of the present invention;

[0046] Figure 5 It is an information input interface diagram in the system of the present invention;

[0047] Figure 6 It is an evaluation result feedback interface diagram in the system of the present invention;

[0048] Figure 7 It is a correlation analysis diagram of the system score of the present invention. DETAILED DESCRIPTION

[0049] The embodiments and effects of the present invention are further described in detail below with reference to the accompanying drawings.

[0050] Example 1, two-stage method for evaluating fine motor skills in one hand for developmental coordination disorder

[0051] Reference Figure 1 The implementation steps of this example include the following:

[0052] Step 1. Construct a single-hand fine motor assessment task dataset D based on the developmental coordination disorder assessment tool book MovementABC-2 peg .

[0053] 1.1) Data collection:

[0054] The subjects were screened to exclude individuals with physical disabilities, mental and emotional disorders, autism or neurological diseases. Finally, 110 subjects aged 7 to 10 were recruited to collect data on the nail-inserting task. All subjects had not received any training in related fine motor skills before data collection.

[0055] Before the video collection begins, each person has a chance to demonstrate and practice the action. After the demonstration and practice, the formal collection begins, that is, the task execution process is recorded through Azure Kinect.

[0056] Informed consent from all subjects and ethical approval from the ethics committee were obtained before data collection.

[0057] 1.2) Data labeling:

[0058] Since the video acquisition process is based on a single person doing a task, an original acquired video includes multiple repeated actions. The evaluation process needs to convert the original video into multiple videos containing a single repeated action. Therefore, the simulation dataset provides the start frame, end frame, whether the action is standard, and action score annotations of a single action based on the Movement ABC-2 tool book, and converts the original video into multiple video clips containing a single repeated action based on the start frame and end frame labels.

[0059] 1.3) Statistics:

[0060] In this example, a total of 435 original nail-insertion videos are collected, but not limited to, and the dataset is denoted as D peg , D peg Each video in the video records the whole process of the subject completing a specific task;

[0061] Randomly transform the data set D peg The training and test sets are selected in a ratio of 8:2. The sizes of the training and test sets in this example are 306 and 82 respectively. The ratio of non-standard to standard actions in all video clips is 214:5014.

[0062] Step 2. Construct a developmental coordination disorder single-hand fine motor assessment network F, which includes a repetitive action localization network F1 and a spatiotemporal area enhancement network F2.

[0063] 2.1) Establish a repeated action localization network F1, which is used to locate the time region of repeated actions in the input video V in the first stage, and obtain the action localization set S composed of the positions of the start frame and the end frame of each repeated action:

[0064] Reference Figure 2 , the implementation of this step includes the following:

[0065] 2.1.1) Establish a data preprocessing module consisting of a posture estimation model and a target detection model in parallel to preprocess the complete nail-insertion video: that is, the posture estimation model is used to detect the complete nail-insertion video and obtain the hand trajectory data D h , the target detection model is used to detect the complete nail insertion video and obtain the object trajectory density data D s , and then D h and D s The preprocessed data D is obtained by splicing in the channel dimension l ;

[0066] The posture estimation model is used to detect the complete nail insertion video and obtain the hand trajectory data D h , that is, use Blazepose to detect the hand position in the video and obtain the key point coordinate position D of the hand position z , D z ∈R T×42×2 , where 42 is the number of key points, and 2 indicates that the dimensions detected by the posture estimation model include the x-axis and the y-axis; since the y-axis has relatively less hand change information than the x-axis, only the x-axis data is retained to obtain the hand trajectory data D h , D z ∈R T×42 ;

[0067] The target detection model is used to detect the complete nail insertion video and obtain the object trajectory density data D s That is, use YOLOv5 to detect the object in the video - the nail, and obtain its quantity curve P that changes over time. Since the curve contains a lot of noise, it needs to be filtered and denoised before the curve analysis, which is implemented as follows:

[0068] First, create a list C to record all the positions where the number of objects in sequence P changes;

[0069] Next, find the first occurrence of each object number and store these locations in list L. p middle;

[0070] Next, initialize another list L rep, used to store the starting time point of the repeated action. Since the starting point of the first repeated action cannot be directly determined from the sequence, the first frame of the video is used as the starting point of the first repeated action by default;

[0071] Next, list L p The first element of minus 2 frames is marked as the end point of the first repetition and is represented by L p The first element of is used as the start of the second repetition action;

[0072] Next, loop through L p For each element of the list, add L p The value of L plus 2 rep list, and L p The last element of is also added to L rep List;

[0073] Finally, according to L rep The start and end time of the action in the list is obtained to obtain the filtered curve P, and then the curve P is differentiated and smoothed to obtain the object trajectory density data D s , D s ∈R T×42 ;

[0074] 2.1.2) Establish a localization module based on LSTM and Transformer to process sequence data containing repeated patterns:

[0075] First, at the first time step, the preprocessed data And the randomly initialized hidden state h1 is input to the LSTM block, outputting the hidden state h2 of the second time step and the data of the second time step

[0076] Secondly, h2 and As the input of the second time step LSTM block, output the hidden state h3 of the third time step and the data of the third time step

[0077] Next, the hidden state and data output in the previous time step are continuously used as the input of the current time step, and the hidden state and data of the current time step are output until the Tth time step is reached, and the hidden state h of the Tth time step is output. T and the data of the Tth time step

[0078]

[0079] Next, obtain the hidden state output of each time step of the LSTM block to form a sequence data H;

[0080] H={h1,h2,…h t …,h T}, H∈R T×d ,

[0081] Each h t ∈R d , d is the hidden state dimension of the LSTM layer, and the size of t ranges from 1 to T;

[0082] Next, the sequence data H is fed into the Transformer block to obtain the aggregated feature F t ;

[0083] F t = Transformer(H), where F t ∈R T×d ;

[0084] Finally, the aggregate feature F t Through the linear layer and Softmax activation function, the density map DM is obtained;

[0085] DM=Softmax(Linear(F t )),

[0086] Among them, DM∈R T , the density map DM represents the probability of action occurring in the time process;

[0087] 2.1.3) Establish a post-processing module for performing peak detection and differential processing on the action probability density map DM to obtain a result set S of video action localization;

[0088] The peak detection process uses the peak detection method to locate the start frame and the end frame of the repetitive action, that is, the density map DM is first inverted to convert the trough into the peak to obtain the reverse density map DM. c ; Then create the video action positioning result set S and traverse the reverse density map DM c The peak of the wave is added to the result set S as the end frame of the action, and the time frame position corresponding to the peak plus 2 frames is used as the start frame of the next repeated action, and added to the result set S; at this time, the result set S contains all the start frames and end frames except the start frame of the first repeated action and the end frame of the last action;

[0089] The difference processing is used to apply the difference method to identify the reverse density map DM c , fill in the missing start frame of the first repeated action and the end frame of the last action in the result set S, that is, detect from the reverse density map DM cThe first significant change point before the first peak is taken as the starting frame of the first repeated action and added to the result set S, and the detection from the last peak to the reverse density map DM c The last significant change point at the end is taken as the end frame of the last action and added to the result set S, thereby obtaining a result set S that completely contains the start and end frames of all repeated actions;

[0090] 2.2) Establish a spatiotemporal region enhancement network F2 to perform the second phase on the video clip set V s Each video segment v of the repeated action n The execution effect of the single-hand action is evaluated to obtain the repeated action evaluation result set R s ;

[0091] Reference Figure 3 , the implementation of this step includes the following:

[0092] 2.2.1) According to the repeated action location result set S, the complete action video is divided into video segments to obtain the video segment set V s ;

[0093] V s ={v1,v2,…v n ,…,v N},

[0094] Among them, v n is the nth video segment after segmentation;

[0095] Create a set of repeated action evaluation results R s , and store the evaluation results of each video segment in R s middle;

[0096] 2.2.2) Establish a feature extraction module consisting of a posture estimation model and a video feature extractor in parallel to extract the position features of the hand area and the video clip features. That is, use the posture estimation model Blazepose (B p ) Get the video clip v n The hand position coordinate data F p , and use Video Swin Transformer (VST) as the video feature extractor to extract the video segment v n Features F h ;

[0097] F h =VST(V n ),

[0098] F s =B p (V n ),

[0099] 2.2.3) Establish a spatiotemporal feature enhancement module consisting of a position matching block and a self-attention block to enhance the video segment feature F h The hand region feature in the video clip feature F h and the position feature F of the hand region p Input to the spatiotemporal feature enhancement module SRE, the position matching block is based on the position feature F of the hand area p Locate the video segment feature F h The self-attention block performs feature enhancement aggregation in the hand area to highlight the feature representation of the hand action, and obtains the hand feature F after spatiotemporal region enhancement. h E , the features of the enhanced spatiotemporal region are similar to the features of the original video clip F h Fusion is performed to obtain the fused feature F f :

[0100] F f =SRE(F h ,F p )+F h ;

[0101] 2.2.4) Establish a feature aggregation module to aggregate the fused features F f Aggregation is performed, and the evaluation result r of the nth video clip in the video is obtained through a three-dimensional convolutional neural network, a spatiotemporal pooling layer, and a linear layer. n :

[0102] R n =Linear(Pooling(3DCNN(F f ))),

[0103] Among them, 3DCNN is a three-dimensional convolutional neural network, Pooling is a spatiotemporal pooling layer, and Linear is a linear layer;

[0104] 2.2.5) The evaluation result r of the nth video segment n Add to result set R s middle;

[0105] 2.2.6) For the video clip set V s Evaluate all the video clips in the result set R to obtain the corresponding evaluation results, and add these evaluation results to the result set R s middle,

[0106] R s ={r1,r2,…r n ,…,r N};

[0107] Among them, r n is the evaluation result of the nth video clip. If the result set R s If the video does not contain non-standard actions, the evaluation result of the complete video is qualified; otherwise, it is unqualified.

[0108] Step 3. Use the training set D t The repeated action localization network F1 and the spatiotemporal region enhancement network F2 in the single-hand fine action evaluation network are iteratively trained by the gradient descent method until their respective loss functions converge, and the trained single-hand fine action evaluation network F is obtained;

[0109] 3.1) Iterative training of the repeated action localization network F1:

[0110] 3.1.1) The mean square loss function MSE is selected as the loss function Loss1 of the repeated action localization network F1;

[0111] 3.1.2) Randomly select from the training set D t Select x training sample videos and input them into the data preprocessing module for preprocessing to obtain the preprocessed data D l ;

[0112] 3.1.3) D l Input to the LSTM and Transformer-based positioning module for feature fusion, and output the action probability density map DM;

[0113] 3.1.4) Calculate the value of the loss function Loss1 according to the action probability density map DM and the labeled value, and use the gradient descent algorithm to update the network parameters;

[0114] 3.1.5) Repeat steps 3.1.2) to 3.1.4) until the loss function Loss1 converges, and obtain the trained repeated action localization network F1;

[0115] 3.2) Iterative training of the spatiotemporal region enhancement network F2 is implemented as follows:

[0116] 3.2.1) Select the weighted cross entropy loss function Weighted CrossEntropyLoss as the loss function Loss2 of the spatiotemporal region enhancement network F2;

[0117] 3.2.2) Randomly select from the training set D t Select y training sample videos and decompose them into video segments according to the start and end frames of the repeated actions in the annotations;

[0118] 3.2.3) Inputting the video clip into the feature extraction module to obtain the hand area position information and video clip features in the video clip;

[0119] 3.2.4) Inputting the hand region position information and the video segment features into the spatiotemporal feature enhancement module to obtain the enhanced video segment features;

[0120] 3.2.5) Inputting the enhanced video segment features into the feature aggregation module to obtain the video segment evaluation results;

[0121] 3.2.6) Calculate the value of the loss function Loss2 based on the video clip evaluation results and the annotation values, and use the gradient descent algorithm to update the network parameters;

[0122] 3.2.7) Repeat steps 3.2.2) to 3.2.6) until the loss function Loss2 converges to obtain the trained spatiotemporal region enhancement network F2;

[0123] 3.3) The trained repetitive action localization network F1 and the trained spatiotemporal region enhancement network F2 are cascaded to obtain the trained developmental coordination disorder single-hand fine action assessment network F.

[0124] Step 4. Test dataset D v Input into the trained single-hand fine movement evaluation network to obtain the estimation result corresponding to the test data.

[0125] 4.1) Sequentially from the test dataset D v Take c test sample videos in batches, input the test sample videos into the trained single-hand fine movement evaluation network, and obtain the estimation results corresponding to c test samples. When the evaluation result is 1, it means that the action is qualified, and when the evaluation result is 0, it means that the action is unqualified.

[0126] 4.2) Repeat step 4.1) until the test set D v The traversal is complete and the estimation results corresponding to all test data are obtained.

[0127] Example 2. Two-stage evaluation system for fine motor skills in one hand for developmental coordination disorder

[0128] Reference Figure 4 The system of this example comprises: an information input unit (1), a data acquisition unit (2), an automatic evaluation unit (3), a result feedback unit (4) and a display unit (5), wherein:

[0129] The information input unit (1) is used to complete the basic information input of the testee and save the basic information of the testee into the local database. That is, the testee needs to fill in all the information required by the system, including: name, number, gender, height, date of birth, weight and dominant hand. Otherwise, the information saving fails, and the system will calculate the real age of the testee according to the collection date. In addition, the subsequent fine movement videos will be classified and managed according to the number of the testee;

[0130] A data acquisition unit (2) is used to record the entire process of the subject performing a single-hand fine movement, and save the recorded video to a folder named with the corresponding number of the subject;

[0131] The automatic evaluation unit (3) is used to automatically call the trained developmental coordination disorder single-hand fine movement evaluation network F to evaluate the single-hand fine movement video of the test subject:

[0132] First, the first stage of repeated action positioning is performed to obtain the action probability density map DM and repeated action positioning result set S of the single-hand fine action video;

[0133] Secondly, the second stage of spatiotemporal region enhancement is performed based on the result set S. That is, the video is first divided into video segments according to the start and end frames of the repeated actions in the result set S, and then all video segments are subjected to feature extraction, spatiotemporal feature enhancement, and feature aggregation to obtain the evaluation result set R. s ;

[0134] Finally, the action probability density map DM, the repeated action positioning result set S and the evaluation result set R are combined. s All are saved in the database;

[0135] The result feedback unit (4) is used to feed back the evaluation results in the database to the test subject, which is implemented as follows:

[0136] Obtain the evaluation results of the test subjects in the database, including the action probability density map DM, the repeated action location result set S and the evaluation result set R s ;

[0137] According to R s Whether it contains non-standard actions to obtain the evaluation results of the complete video: when the result set R s If there are no non-standard actions in the video, the evaluation result R is shown as qualified, and the completion time of the one-handed task in the video is calculated according to the

[0138] The scoring results are obtained by the standard of the MovementABC-2 reference book. When the result set R s When non-standard actions are included, the evaluation result R is displayed as unqualified, and the location of the non-standard actions in the result set S is obtained;

[0139] Feedback the action probability density map DM, repeated action location result set S, the evaluation result R of the complete video, and the location of the non-standard action in the result set S to the test subject;

[0140] The display unit (5) is used to display the information input unit interface and the result feedback unit interface, wherein:

[0141] The information entry unit interface is as follows Figure 5 The interface shows the basic information that the testee needs to enter and the "OK" button. After entering all the information, click the "OK" button to submit. If the submission is successful, a prompt will appear at the bottom of the interface.

[0142] The words "Entry successful" are displayed;

[0143] The result feedback unit interface is as follows Figure 6 , the feedback results obtained by the feedback unit and the statistical feedback results are displayed, which include the action probability density map DM, the repeated action positioning result set S, and the evaluation result R of the complete video. In addition, if the result R is qualified, the score of the complete video is displayed, otherwise the positioning of the non-standard action in the result set S is displayed.

[0144] The effect of the present invention can be further illustrated by the following simulation.

[0145] 1. Simulation conditions

[0146] The experimental simulation of the repeated action localization network F1 in the first stage is on the nail insertion dataset D peg The network F1 training uses the nail insertion dataset D peg The training set D t , the network test uses the nail insertion dataset D peg The test set D in v The Adam optimizer was used, the mean square loss function (MSE) was used as the loss function, and the basic learning rate was 5×10 -4 The number of network training rounds is set to 100 rounds, the early stop is set to 10 rounds, and the batch size is set to 1. In both training and testing processes, the number of LSTM blocks in the network is set to 3, the input feature dimension is 84, the hidden layer size is set to twice the input feature dimension, that is, 168, the Transformer block input and output dimensions are set to 168, the number of layers is set to 4, the attention head is set to 16, the number of special diagnoses per attention head is set to 16, the feedforward neural network layer dimension is set to 16, and the dropout rate is set to 0.2.

[0147] The experimental simulation of the spatiotemporal region enhancement network F2 in the second stage is on the nail insertion dataset D peg The network F2 training uses the nail insertion dataset Dpeg The training set D t , the network test uses the nail insertion dataset D peg The test set D v The training process uses the Adam optimizer with a basic learning rate of 8×10 -7 The model was trained for 50 rounds, with early stopping set to 8 rounds and batch size set to 4. The input video was downsampled to 64 frames at equal intervals, and the weighted cross entropy loss function WeightedCrossEntropyLoss was used as the loss function.

[0148] 2. Simulation content

[0149] Simulation 1: Under the simulation conditions of the first stage above, on the test set D v The above uses different networks to perform repeated action positioning and calculates the mAP evaluation index of each under the specified tIOU. The results are shown in Table 1:

[0150] Table 1. Experimental results of action localization comparison methods

[0151]

[0152] In Table 1: YOLOv5+post-processing is a network consisting of a YOLOv5 model and a post-processing module;

[0153] Only the object trajectory density F t It is a network composed of a pre-processing module, an action localization module and a post-processing module. The pre-processing module of the network uses a target detection algorithm to obtain an object trajectory density map;

[0154] Only hand trajectory network F h The network is composed of a preprocessing module, an action localization module and a post-processing module. The preprocessing module of the network uses a posture estimation model to obtain a hand trajectory map.

[0155] It can be seen from Table 1 that, except under the condition of tIOU=0.9, the mAP index of the repeated action localization network F1 in the present invention is lower than that of the YOLOv5+post-processed network. Under other tIOU conditions, the mAP index of the F1 network in the present invention is the highest, indicating that the repeated action localization network F1 in the present invention can effectively complete the localization of repeated actions.

[0156] Simulation 2. Under the simulation conditions of the second stage above, on the test set D vThe effectiveness of the spatiotemporal region enhancement network F2 proposed in the present invention in evaluating single-hand fine movements is verified. Accuracy and F1-Score are used as evaluation indicators, and the existing network Video Swin Transformer and TSA-Net models that perform well in the movement evaluation task are simulated and compared. The results are shown in Table 2 below.

[0157] Table 2. Experimental results of action evaluation comparison methods

[0158] method Accuracy F1-Score VideoSwinTransformer 93.43 50.74 TSA-Net 95.82 41.79 <![CDATA[The Timing Region Enhancement Network F2 in the present invention]]> 96.71 63.73

[0159] As can be seen from Table 2, the temporal region enhancement network F2 in the present invention is superior to Video SwinTransformer and TSA-Net in all indicators. Among them, the temporal region enhancement network F2 in the present invention reaches 63.73% in the F1-Score indicator, which is 21.94% higher than TSA-Net and 12.99% higher than Video Swin Transformer. The results show that the temporal region enhancement network F2 in the present invention can effectively evaluate the fine movements of one hand.

[0160] Simulation 3. In the test set D v The significance analysis was performed on the data to verify the significant correlation between the automated scoring of the result feedback unit of the system of the present invention and the true value of the score label. The results are shown in Table 3. The accuracy of the automated scoring was further verified by curve fitting of the automated scoring and the score label value. The results are shown in Table 3. Figure 7 As shown, the horizontal axis is the label value and the vertical axis is the automatic scoring value.

[0161] Table 3. Significance analysis of automated scoring and true value of score labels

[0162] Mean error ± standard deviation P-Value 1.37±2.41 <![CDATA[7.01×10 -7 ]]>

[0163] As can be seen from Table 3, the average error value between the automated scoring of the result feedback unit of the system of the present invention and the true value of the score label is low. The results of the F test on these two variables show that at the significance level of p<0.001, the automated scoring results of the system are significantly correlated with the true value of the score label.

[0164] from Figure 7 It can be seen that the slope of the fitting curve formed by the automatic scoring and the true value of the score label is close to 45° and the intercept is close to 0, indicating that the automatic scoring value has a good prediction result for the true value of the label, verifying the accuracy of the automatic scoring of the present invention.

[0165] The above results show that the present invention can effectively improve the accuracy of single-hand fine motor assessment through a two-stage single-hand fine motor assessment method for developmental coordination disorder.

[0166] The above description is only a specific example of the present invention and does not constitute any limitation to the present invention. It is obvious that for professionals in this field, after understanding the content and principles of the present invention, they may make various modifications and changes in form and details without departing from the principles and structures of the present invention. However, these modifications and changes based on the ideas of the present invention are still within the scope of protection of the claims of the present invention.

[0167] It should be noted that the step numbers in the specification and claims of the present invention are only for a clear description of the implementation scheme of the present invention and for ease of understanding, and the order of the step numbers is not limited.

Claims

1. A method for evaluating fine motor skills of one hand in developmental coordination disorder based on visual behavior analysis, characterized in that: The steps include: (1) Construct a single-hand fine motor assessment task dataset D based on the developmental coordination disorder assessment tool book MovementABC-2 peg : Children aged 7 to 10 were recruited for data collection. Videos of the children performing the nail-insertion task in the one-hand fine motor assessment task were recorded with a camera, and the videos were annotated according to the MovementABC-2 reference book. D peg The data set is divided into training set D in a ratio of 8:2 t and the test set D v ; (2) Constructing a single-hand fine motor assessment network F for developmental coordination disorders, including a repetitive action localization network F1 and a spatiotemporal area enhancement network F2; The repeated action localization network F1 is used to locate the start and end time regions of repeated actions in the input video V in the first stage, and obtain an action localization set S consisting of the start frame and end frame positions of each repeated action: S={[B1,E1],[B2,E2],…[B n ,E n ],…,[B N ,E N ]} Among them, B n and E n are the start frame and end frame of the nth action in the input video V, respectively, and the value of n ranges from 1 to N; The spatiotemporal region enhancement network F2 is used to perform the video segment set V s Each video segment v n The included single-handed action skills are evaluated to obtain the repeated action evaluation result set R s : R s ＝{r1,r2,…r n ,…,r N}； Among them, V s ={v1,v2,…v n ,…,v N } is the set of video clips obtained by segmenting the result set S of the repeated action localization stage, v n is the nth video segment after segmentation, r n is the evaluation result of the nth video clip, its value is 0: non-standard action, or 1: standard action, the size of n is from 1 to N; (3) Using training set D t The repeated action localization network F1 and the spatiotemporal region enhancement network F2 in the single-hand fine action evaluation network are iteratively trained by the gradient descent method until their respective loss functions converge, and the trained single-hand fine action evaluation network F is obtained; (4) Test data set D v Input into the trained single-hand fine movement evaluation network to obtain the estimation result corresponding to the test data.

2. The method according to claim 1, characterized in that: In step (1), according to the MovementABC-2 reference book, D peg The video in the dataset is labeled by first marking the positions of the start and end frames of the repeated actions in the complete video, and then marking the completion status of each action, that is, marking non-standard actions as 0 and standard actions as 1.

3. The method according to claim 1, characterized in that The repeated action localization network F in step (2) 1， include: The data preprocessing module is used to preprocess the complete nail insertion video. It is composed of a posture estimation model and a target detection model in parallel. The posture estimation model is used to obtain the hand trajectory data D h , the target detection model is used to detect the complete nail insertion video and obtain the object trajectory density data D s , D h and D s Stitching in channel dimension Get Preprocessed data D l ; The positioning module based on LSTM and Transformer, which is composed of LSTM blocks, Transformer blocks and linear layers in series, is used to perform localization on the preprocessed data D l Perform feature fusion and output the action probability density map DM; A post-processing module is used to post-process the action probability density map DM using a peak detection algorithm and a difference method to obtain an action location set S consisting of the start frame and end frame positions of each repeated action; The data preprocessing module, positioning module, and post-processing module are sequentially cascaded to form a repeated action positioning network F1, whose input is a complete nail-inserting video, and whose output is an action positioning set S consisting of repeated action positions in the video.

4. The method according to claim 1, characterized in that The spatiotemporal region enhancement network F2 of step (2) comprises: A feature extraction module, which is composed of a posture estimation model and a video encoder model connected in series. The posture estimation model is used to obtain the position information of the hand area, and the video encoder model is used to obtain the features of the video clip; The spatiotemporal feature enhancement module consists of a position matching block and a self-attention block, which is used to enhance the information of the hand area of ​​the video clip feature and obtain the video clip features after feature enhancement; The feature aggregation module, which consists of a three-dimensional convolution block, a pooling layer, and a linear layer, is used to aggregate the enhanced video clip features and obtain the evaluation result set R of the video clip. s ; The feature extraction module, spatiotemporal feature enhancement module, and feature aggregation module are sequentially cascaded to form a spatiotemporal region enhancement network F2, whose input is the video clip segmented by the action location set S, and the output is the repeated action evaluation result set R s .

5. A method according to any one of claims 1 to 3, characterized in that Step (3) uses the training set D t The repeated action localization network F1 is trained by gradient descent method as follows: (3a) Randomly select from the training set D t Select x training sample videos and input them into the data preprocessing module for preprocessing to obtain the preprocessed data D l ; (3b) D l Input to the LSTM and Transformer-based positioning module for feature fusion, and output the action probability density map DM; (3c) Select MSE as the loss function Loss of the repeated action localization network F1, calculate the loss function value according to the action probability density map DM, and use the gradient descent algorithm to update the network parameters; (3d) Repeat steps (3a) to (3c) until the loss function Loss converges, and obtain the trained repeated action localization network F1.

6. A method according to any one of claims 1 to 4, characterised in that Step (3) uses the training set D t The spatiotemporal region enhancement network F2 is trained by the gradient descent method as follows: (3e) Randomly select from the training set D t Select y training sample videos and decompose them into video segments according to the start and end frames of the repeated actions in the annotations; (3f) inputting the video clip into a feature extraction module to obtain the hand region position information and video clip features in the video clip; (3g) inputting the hand region position information and the video segment features into a spatiotemporal feature enhancement module to obtain the video segment features after feature enhancement; (3h) Inputting the enhanced video segment features into the feature aggregation module to obtain the video segment evaluation results; (3i) Select WeightedLoss as the loss function Loss of the spatiotemporal region enhancement network F2, calculate the loss function value according to the video clip evaluation results, and use the gradient descent algorithm to update the network parameters; (3j) Repeat steps (3e) to (3i) until the loss function Loss converges to obtain the trained spatiotemporal region enhancement network F2.

7. The method according to claim 1, characterized in that Step (4) The test data set D v Input it into the trained single-hand fine movement evaluation network to obtain the estimation result corresponding to the test data, which is implemented as follows: (4a) Sequentially from the test dataset D v Take c test sample videos in batches, input the test sample videos into the trained single-hand fine movement evaluation network, and obtain the estimation results corresponding to the c test samples; (4b) Repeat step (4a) until the test set D v Be traversed completely and obtain the estimates corresponding to all test data result.

8. The method according to claim 3, characterized in that The target detection model detects the complete nail insertion video and obtains the object trajectory density data D s , implemented as follows First, the existing target detection algorithm YOLOv5 is used to obtain the number curve P of the object in the video—the nails changing over time; Next, the curve P is filtered, that is, the position where each object quantity first appears is found through the curve P, and the position is used as the starting position of the next action, and the position minus two frames is used as the end position of the action; Then, according to the start and end time of the action, the filtered curve P is obtained, and then the curve P is differentiated and smoothed to obtain the object trajectory density data D s .

9. The method according to claim 4, characterized in that The spatiotemporal feature enhancement module enhances the information of the hand area of ​​the video clip feature by first using the position matching block therein to locate the position R of the hand area in the video clip feature, and then using the self-attention block therein to enhance the hand feature in position R.

10. A system for evaluating fine motor skills of one hand in developmental coordination disorder based on visual behavior analysis, characterized in that: It is based on the developmental coordination disorder assessment tool book MovementABC-2, including: information entry unit, data collection unit, automatic evaluation unit, result feedback unit and display unit; The information input unit is used to input the basic information of the test subject, including name, number, birthday, height, weight, dominant hand and gender, and store the information persistently in the database; The data acquisition unit is used to automatically call the Azure Kinect camera to collect the process of the test subject performing the one-handed task and save it to a designated folder; The automatic evaluation unit is used to automatically call the evaluation algorithm to perform a two-stage evaluation on the video data: In the first stage, repeated action positioning is performed on the video data. In the second stage, the video is segmented into video segments according to the repeated positioning results. The video segments are then evaluated to obtain a set containing the evaluation results of each video segment. The evaluation result of the complete video is obtained according to whether the result set contains non-standard actions; If the result set does not contain non-standard movements, the evaluation result of the complete video is qualified and recorded as 1. The scoring result is obtained according to the completion time of the task in the video and the standards of the MovementABC-2 reference book; Otherwise, it is considered unqualified and recorded as 0; Save the evaluation results into the database; The result feedback unit is used to obtain the video evaluation results recorded in the database and feed them back to the test subject; The display unit is used to display the results of each functional unit.

Images