Developmental coordination disorder fine motor assessment method based on visual behavior analysis

Abstract

The application discloses a development coordination disorder single-hand fine action evaluation method based on visual behavior analysis, and mainly solves the problems of insufficient objectivity, poor accuracy and low generalization of the existing development coordination disorder single-hand fine action evaluation method. The implementation scheme is as follows: a standard plug-in sub task video data set is constructed and is divided into a training data set and a test data set; a repeated action positioning network F1 of the first stage is constructed and is used for positioning the repeated action in the video; a time sequence region network F2 of the second stage is constructed and is used for evaluating the positioned action segment; the training data set is used to train the two networks F1 and F2 through gradient descent respectively; and the test data set is input into the two trained networks to obtain an evaluation result. Through the visual-based evaluation method, the fine-grained evaluation of the single-hand fine action is realized, the evaluation accuracy and efficiency are effectively improved, and the method can be used for large-scale screening of the development coordination disorder (DCD).

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of pattern recognition and intelligent systems, and mainly relates to a method for evaluating single-hand fine motor skills of developmental coordination disorder (DCD), which can be used for large-scale screening of DCD, helps early auxiliary diagnosis and intervention of DCD, and avoids further deterioration of symptoms. BACKGROUND

[0002] Developmental coordination disorder, also known as motor coordination disorder, is a type of developmental disorder that affects the development of children's motor skills. The prevalence of this disorder among preschool and school-age children is approximately 2% to 10%. Children with this disorder often exhibit clumsy movements, poor coordination between hands and feet, and difficulty completing specific motor tasks as smoothly as their peers. If not diagnosed and intervened in a timely manner, the symptoms of most affected children will persist into adolescence or adulthood, severely affecting their social, academic, and emotional development, and causing some secondary health problems such as obesity and depression.

[0003] Currently, the clinical diagnosis of DCD is made by a doctor who observes the accuracy, speed, and fluency of a test subject in completing specific tasks and scores the subject's performance based on a diagnostic toolbook. A lower score indicates a possible motor coordination problem. Currently, the most popular diagnostic toolbooks include Movement Assessment Battery for Children-Second Edition (Movement ABC-2), Bruininks-Oseretsky Test of Motor Proficiency-Second Edition (BOT-2), Developmental Coordination Disorder Questionnaire (DCDQ), etc. Among them, Movement ABC-2 is the most widely used standardized DCD diagnostic evaluation tool and is considered the "gold standard" for assessing children's motor performance. According to the Movement ABC-2 toolbook, fine motor skill assessment tasks include single-hand operation tasks, two-hand coordination tasks, and handwriting trajectory drawing tasks. Among them, single-hand operation is the basis for other fine motor actions, and the development of children's fine motor actions usually starts with simple single-hand actions, and then gradually develops more complex two-hand actions and trajectory drawing skills. Therefore, the practice and mastery of single-hand fine motor actions are crucial for the development of fine motor skills.

[0004] The main role of the single-hand operation task is to evaluate the fine motor skills and hand-eye coordination of children or individuals using a single hand to complete specific tasks. These tasks can help understand the individual's hand flexibility, strength control, and precision. For example, in the peg insertion task specified in the Movement ABC-2 toolkit, the subject needs to insert yellow pegs into a blue board one by one, and the evaluator records the total time to complete the insertion of all 12 pegs to assess the skill level. Each subject performs two rounds of formal testing for each hand, with a pre-exercise of inserting three pegs. The task aims to evaluate the subject's single-hand fine motor skills and their execution efficiency. Generally, the complete single-hand operation task process includes repetitive actions, which can avoid the randomness of single operation and accurately assess the individual's hand muscle memory and ability to maintain consistent action over time.

[0005] In the prior art, there are usually two methods for evaluating single-hand fine motor skills: manual evaluation-based methods and computer vision-based methods.

[0006] The manual evaluation-based method is performed by medical professionals or therapists who observe the testee's performance with their naked eyes and score based on diagnostic toolbooks and their own experience. This method requires one-on-one recording, consumes manpower, is inefficient, and is highly subjective, which can lead to inconsistent judgment standards, resulting in large scoring errors and inaccurate evaluation results. In addition, since the testee's data is recorded by paper and pen, there is a risk of data loss.

[0007] With the development of modern computer technology and artificial intelligence technology, computer vision-based methods have been widely applied. This method first collects video, image, and other information of the single-hand fine motor action process through non-contact data collection, then designs intelligent algorithms to intelligently process and analyze the image information, thereby realizing the evaluation of single-hand fine motor skills. Li et al. proposed a template matching-based method in the article "Automated Fine Motor Evaluation for Developmental Coordination Disorder". For the peg insertion task, the appearance of the peg is extracted as a template, which is matched in the peg insertion video to obtain the number of pegs in each frame of the video. Through filtering strategies, the total time of completing the peg insertion task in the video is obtained, and then the score of the peg insertion task is obtained according to the standard of the Movement ABC-2 toolkit. However, this method lacks generalizability and can only be used for peg insertion tasks, and cannot be extended to other single-hand fine motor skill evaluation tasks.

[0008] Cui et al. proposed an evaluation method of single-hand fine motor action based on skeletal data in the article "Study on Assessment Methods of Developmental Coordination Disorder in Children". First, the overall skeletal data of the human body is extracted using a pose estimation algorithm, and then the hand skeletal coordinate data is extracted. The three-dimensional convolutional neural network and the long short-term memory network are directly used to evaluate the single-hand fine motor action. However, this method ignores the repetitive nature of single-hand fine motor action and directly evaluates the complete action execution process, lacking processing of fine motor action details. In addition, since this method only uses skeletal data, it lacks natural image information, so it is difficult to extract joint features of hand and object for single-hand action interacting with objects, resulting in low evaluation accuracy. SUMMARY

[0009] The present application aims to overcome the shortcomings of the prior art and provide a developmental coordination disorder single-hand fine motor action evaluation method based on repetitive action positioning and timing region enhancement to improve the objectivity, generalization and accuracy of single-hand fine motor action evaluation.

[0010] To achieve the above-mentioned purpose, the technical scheme of the present application includes the following:

[0011] 1. A developmental coordination disorder single-hand fine motor action evaluation method based on visual behavior analysis, characterized in that it comprises the following steps:

[0012] (1) Construct a single-hand fine motor action evaluation task dataset D based on the MovementABC-2 developmental coordination disorder evaluation toolbook peg :

[0013] Data collection. Recruit children aged 7 to 10 years old for data collection. Record the video of the children performing the peg insertion task in the single-hand fine motor action evaluation task through the camera, and label the video according to the MovementABC-2 toolbook.

[0014] Data division. Divide the dataset D peg into a training set D t and a test set D v in a ratio of 8:2.

[0015] (2) Construct a developmental coordination disorder single-hand fine motor action evaluation network F including a repetitive action positioning network F1 and a spatio-temporal region enhancement network F2;

[0016] The repetitive action positioning network F1 is used to position the start and end time regions of repetitive actions in the input video V in the first stage, and obtain an action positioning set S composed of the start frame and end frame positions of each repetitive action:

[0017] S = {[B1, E1], [B2, E2], … [B n , E n ], …, [B N , E N ]}

[0018] wherein B n n and E n n are the start frame and end frame of the nth action in the input video V, n ranges from 1 to N;

[0019] the spatio-temporal region enhancement network F2, configured to evaluate the single-hand action skills contained in each video segment v s in the video segment set V n in the second stage, to obtain a repeated action evaluation result set R s :

[0020] R s = {r1, r2, … r n ,…, r N};

[0021] wherein V s = {v1, v2, … v n ,…, v N} is the video segment set segmented according to the result set S of the repeated action positioning stage, v n n is the nth video segment after segmentation, r n n is the evaluation result of the nth video segment, whose value is 0: non-standard action, or 1: standard action, n ranges from 1 to N;

[0022] (3) using the training set D t to respectively perform iterative training on the repeated action positioning network F1 and the spatio-temporal region enhancement network F2 in the single-hand fine action evaluation network through gradient descent method, until the respective loss functions converge, to obtain the trained single-hand fine action evaluation network F;

[0023] (4) inputting the test data set D v to the trained single-hand fine action evaluation network to obtain the estimation result corresponding to the test data.

[0024] 2. A development coordination disorder single-hand fine action evaluation system based on visual behavior analysis, characterized in that it is based on the Movement ABC-2 development coordination disorder evaluation toolbook, and comprises an information input unit, a data acquisition unit, an automatic evaluation unit, a result feedback unit and a display unit.

[0025] The information input unit is used for inputting basic information of the testee, including name, number, birthday, height, weight, dominant hand and gender, and persistently saving the information in a database.

[0026] The data collection unit is used for automatically calling an AzureKinect camera, collecting the process of the testee performing a single-hand task, and saving to a designated folder.

[0027] The automatic evaluation unit is used for automatically calling an evaluation algorithm to perform two-stage evaluation on the video data.

[0028] In the first stage, the video data is subjected to repetitive action positioning, in the second stage, the video is segmented into video clips according to the repetitive positioning result, and the video clips are evaluated to obtain a set containing the evaluation result of each video clip.

[0029] According to whether the result set contains non-standard actions, the evaluation result of the complete video is obtained.

[0030] If the result set does not contain non-standard actions, the evaluation result of the complete video is qualified, recorded as 1, and according to the completion time of the single-hand task in the video, the score is scored according to the standard of the Movement ABC-2 tool book,

[0031] Otherwise, it is unqualified, recorded as 0,

[0032] The evaluation result is saved to the database.

[0033] The result feedback unit is used for obtaining the video evaluation result recorded in the database and feeding back to the testee.

[0034] The display unit is used for displaying the results of each functional unit.

[0035] Compared with the prior art, the present application has the following advantages:

[0036] First, the objectivity of fine-grained evaluation of single-hand fine motor action is improved:

[0037] The present application uses an automatic evaluation method based on computer vision, inputs the single-hand fine motor action video to be evaluated, automatically evaluates the video, and outputs the evaluation result of the video, without the need for manual evaluation, eliminating the subjective and inaccurate problems caused by the manual evaluation method.

[0038] Second, the accuracy and generalization of fine-grained evaluation of single-hand fine motor action are improved:

[0039] The application utilizes the property of single-hand fine action repeatability, and proposes a fine-grained evaluation method for developing coordination disorder single-hand fine action, namely, in the first stage, the repeated action positioning is performed, and the time interval of each action is found, in the second stage, the complete video is first decomposed into a segment containing only a single action according to the positioning result, then each action segment obtained by the repeated action positioning is taken as a sample for separate evaluation, and the spatiotemporal features are enhanced to obtain more detailed features of hand changes, and through the two stages, more accurate evaluation results can be obtained. In addition, the method can also be extended to other single-hand fine action evaluation tasks with repeatability, and the generalization of single-hand fine action evaluation is improved.

[0040] Third, the efficiency of single-hand fine action evaluation is improved:

[0041] The application constructs a development coordination disorder single-hand fine action evaluation system based on visual behavior analysis, realizes automatic basic information collection, video data collection and action evaluation, improves the efficiency of single-hand fine action evaluation, and in addition, by saving the collected basic information and video in the database, the traceability of data is realized. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 is the implementation flowchart of the method of the application;

[0043] Figure 2 is the first-stage action positioning network F1 structure diagram constructed in the method of the application;

[0044] Figure 3 is the second-stage time sequence region enhancement network F2 structure diagram constructed in the method of the application;

[0045] Figure 4 is the structure block diagram of the system of the application;

[0046] Figure 5 is the information entry interface diagram in the system of the application;

[0047] Figure 6 is the evaluation result feedback interface diagram in the system of the application;

[0048] Figure 7 is the correlation analysis diagram of the score of the system of the application. DETAILED DESCRIPTION

[0049] The embodiments and effects of the application will be further described in detail below with reference to the drawings.

[0050] Embodiment 1, two-stage development coordination disorder single-hand fine action evaluation method

[0051] Reference Figure 1 The implementation steps of the present example include the following:

[0052] Step 1. Constructing the single-hand fine motor assessment task dataset D based on the Movement ABC-2 developmental coordination disorder assessment tool peg .

[0053] 1.1) Data collection:

[0054] The subjects were screened to exclude individuals with physical disabilities, mental and emotional disorders, autism or nervous system diseases, and finally 110 subjects aged 7 to 10 years were recruited for data collection of the plug-in task, all of whom had not received training related to fine motor skills before data collection;

[0055] Each person had the opportunity to demonstrate and practice the action once before the video collection started, and after the demonstration and practice, the formal collection began, i.e. the task execution process was recorded by Azure Kinect.

[0056] In this example, the informed consent of all subjects and the ethical approval of the ethics committee were obtained before data collection.

[0057] 1.2) Data label annotation:

[0058] Since the video collection process is based on a single person performing a task at a time, an original collection video includes multiple repeated actions, and the evaluation process needs to convert the original video into multiple videos containing a single repeated action, so the simulation dataset provides the start frame, end frame, action standard or not, and action score annotation of the single action according to the Movement ABC-2 tool, and the original video is converted into multiple video clips containing a single repeated action according to the start frame and end frame labels.

[0059] 1.3) Data statistics:

[0060] This example collected a total of 435 original plug-in videos, not limited to, and the dataset is denoted as D peg , each video in D peg records the entire process of the subject completing a specific task;

[0061] The dataset D peg is randomly selected according to an 8:2 ratio for training and testing sets, and the training set and testing set sizes of this example are 306 and 82 respectively, and the ratio of non-standard to standard actions in all video clips is 214:5014.

[0062] Step 2. Constructing the developmental coordination disorder single-hand fine motor assessment network F including the repeated action positioning network F1 and the spatio-temporal region enhancement network F2.

[0063] 2.1) Establish a repetitive action positioning network F1 for positioning the time region of repetitive actions in the input video V in the first stage, and obtain the action positioning set S composed of the position of the start frame and the end frame of each repetitive action:

[0064] Referring to Figure 2 , the implementation of this step includes the following:

[0065] 2.1.1) Establish a data preprocessing module composed of a pose estimation model and a target detection model in parallel, for preprocessing the complete nail insertion video: the pose estimation model is used to detect the complete nail insertion video, and obtain hand trajectory data D h , the target detection model is used to detect the complete nail insertion video, and obtain object trajectory density data D s , and then D h and D s are spliced in the channel dimension to obtain the preprocessed data D l ;

[0066] The pose estimation model is used to detect the complete nail insertion video, and obtain hand trajectory data D h , that is, Blazepose is used to detect the hand position in the video, and obtain the key point coordinate position D z of the hand position, D z ∈R T×42×2 , where 42 is the number of key points, and 2 represents the dimension of the pose estimation model detection, including x-axis and y-axis; since the hand change information of the y-axis is relatively less than that of the x-axis, only the data of the x-axis is retained to obtain the hand trajectory data D h , D z ∈R T×42 ;

[0067] The target detection model is used to detect the complete nail insertion video, and obtain object trajectory density data D s , that is, YOLOv5 is used to detect the object—nail in the video, and obtain the number curve P of the object changing with time. Since the curve contains a lot of noise, it needs to be filtered and denoised before curve analysis, which is implemented as follows:

[0068] First, create a list C to record all positions of the number of objects in sequence P;

[0069] Second, find the position where each object number first appears, and store these positions in list L p ;

[0070] Next, initialize another list L repThe starting point of the repetitive motion is stored, and the first frame of the video is used as the starting point of the first repetitive motion by default because the starting point of the first repetitive motion cannot be directly determined from the sequence;

[0071] Next, the first element of the list L p is subtracted by 2 frames, marked as the end point of the first repetitive motion, and the first element of L p is used as the start of the second repetitive motion;

[0072] Next, each element of the L p list is processed in a loop, and the value of L p and its value plus 2 are added to the L rep list, and the last element of L p is also added to the L rep list;

[0073] Finally, according to the start and end time of the motion in the L rep list, the filtered curve P is obtained, and the curve P is differentiated and smoothed, and the object trajectory density data D s is obtained, D s ∈R T×42 ;

[0074] 2.1.2) Establish a positioning module based on LSTM and Transformer to process sequence data containing repetitive patterns:

[0075] First, input the preprocessed data and the randomly initialized hidden state h1 to the LSTM block at the first time step, output the hidden state h2 at the second time step and the data

[0076] Second, input h2 and to the second LSTM block as input, output the hidden state h3 at the third time step and the data

[0077] Next, the hidden state and data output at the previous time step are input as the input of the current time step, and the hidden state and data of the current time step are output until the Tth time step is reached, and the hidden state h T and the data

[0078]

[0079] Next, the hidden state output by the LSTM block at each time step is obtained to form a sequence data H;

[0080] H = {h1, h2,... h t , T}, H e R T×d ,

[0081] where each h t e R d , d is the hidden state dimension of the LSTM layer, and the size of t is from 1 to T;

[0082] Next, the sequence data H is sent into the Transformer block to obtain the aggregated feature F t ;

[0083] F t = Transformer (H), where F t e R T×d ;

[0084] Finally, the aggregated feature F t is passed through a linear layer and a Softmax activation function to obtain the density map DM;

[0085] DM = Softmax (Linear (F t )),

[0086] where DM e R T , and the density map DM represents the probability of action occurrence in the time course;

[0087] 2.1.3) Establish a post-processing module for peak detection and differential processing of the action probability density map DM to obtain a result set S of video action localization;

[0088] The peak detection processing is to use a peak detection method to locate the start frame and end frame of the repeated action, that is, first perform an inversion operation on the density map DM to convert the troughs to peaks to obtain an inverted density map DM c ; then create a video action localization result set S, traverse the peaks of the inverted density map DM c , add the peaks as the end frames of the action to the result set S, and add the time frame position corresponding to the peaks plus 2 frames as the start frame of the next repeated action to the result set S; at this time, the result set S contains all the start frames and end frames except the start frame of the first repeated action and the end frame of the last action;

[0089] The differential processing is used to apply a differential method to identify the inverted density map DM c , and complete the missing start frame of the first repeated action and the end frame of the last action in the result set S, that is, the detection is from the inverted density map DM cThe first significant change point before the first peak as the start frame of the first repetitive action and join the result set S, and detect from the last peak to the reverse density map DM c The last significant change point as the end frame of the last action and join the result set S, thus obtaining the result set S containing all the start and end frames of the repetitive actions;

[0090] 2.2) Establish a spatio-temporal region enhancement network F2 for the second stage to video segment set V s each repetitive action video segment v n The execution effect of the single-hand action is evaluated to obtain the repetitive action evaluation result set R s ;

[0091] Referring Figure 3 , the implementation of this step includes the following:

[0092] 2.2.1) According to the repetitive action positioning result set S, the complete action video is segmented into video segments to obtain the video segment set V s ;

[0093] V s ={v1,v2,…v n ,…,v N},

[0094] wherein v n is the nth video segment after segmentation;

[0095] Create a repetitive action evaluation result set R s , and store the evaluation result of each video segment in R s ;

[0096] 2.2.2) Establish a feature extraction module composed of a pose estimation model and a video feature extractor in parallel, for extracting the position features of the hand region and the video segment features, i.e. using the pose estimation model Blazepose(B p ) to obtain the hand position coordinate data F n in the video segment v p , and using Video Swin Transformer(VST) as a video feature extractor to extract the features F n of the video segment v h ;

[0097] F h =VST(V n ),

[0098] F s =B p (V n ),

[0099] 2.2.3) Establish a spatio-temporal feature enhancement module composed of a position matching block and a self-attention block, which is used to enhance the hand region feature in the video segment feature F h h p The video segment feature F p h and the position feature F h of the hand region are input into the spatio-temporal feature enhancement module SRE, and the position matching block locates the corresponding hand region position in the video segment feature F E h The self-attention block performs feature enhancement aggregation within the hand region to highlight the feature representation of the hand action, and obtains the hand feature F f after spatio-temporal region enhancement.

[0100] F f = SRE(F h , F p ) + F h ;

[0101] 2.2.4) Establish a feature aggregation module to aggregate the fused feature F f , and obtain the evaluation result r n of the nth video segment in the video through a three-dimensional convolutional neural network, a spatio-temporal pooling layer and a linear layer.

[0102] R n = Linear(Pooling(3DCNN(F f )),

[0103] wherein, 3DCNN is a three-dimensional convolutional neural network, Pooling is a spatio-temporal pooling layer, and Linear is a linear layer.

[0104] 2.2.5) Add the evaluation result r n of the nth video segment to the result set R s .

[0105] 2.2.6) Evaluate all video segments in the video segment set V s to obtain corresponding evaluation results, and add these evaluation results to the result set R s ,

[0106] R s = {r1, r2, … r n , …, r N}. ​​​

[0107] wherein, r n is the evaluation result of the nth video segment, if the result set R s does not contain non-standard actions, the evaluation result of the complete video is qualified; otherwise, it is unqualified.

[0108] Step 3. Using the training set D t The iterative training of the repeated action positioning network F1 and the spatio-temporal region enhancement network F2 in the single-hand fine evaluation network is carried out by the gradient descent method until the respective loss functions converge, and the trained single-hand fine action evaluation network F is obtained.

[0109] 3.1) Iterative training of the repeated action positioning network F1:

[0110] 3.1.1) Select the mean square loss function MSE as the loss function Loss1 of the repeated action positioning network F1;

[0111] 3.1.2) Randomly select x training sample videos from the training set D t , input them into the data preprocessing module for preprocessing, and obtain the preprocessed data D l ;

[0112] 3.1.3) Input D l to the positioning module based on LSTM and Transformer for feature fusion, and output the action probability density map DM;

[0113] 3.1.4) Calculate the value of the loss function Loss1 according to the action probability density map DM and the labeled value, and update the network parameters using the gradient descent algorithm;

[0114] 3.1.5) Repeat steps 3.1.2) to 3.1.4) until the loss function Loss1 converges, and obtain the trained repeated action positioning network F1;

[0115] 3.2) Iterative training of the spatio-temporal region enhancement network F2, including:

[0116] 3.2.1) Select the weighted cross-entropy loss function Weighted CrossEntropyLoss as the loss function Loss2 of the spatio-temporal region enhancement network F2;

[0117] 3.2.2) Randomly select y training sample videos from the training set D t , and decompose them into video segments according to the start frame and end frame of the repeated action in the label;

[0118] 3.2.3) Input the video segments into the feature extraction module to obtain the hand region position information and video segment features in the video segments;

[0119] 3.2.4) inputting the hand region position information and the video segment feature into the spatio-temporal feature enhancement module to obtain the video segment feature after feature enhancement;

[0120] 3.2.5) inputting the enhanced video segment feature into the feature aggregation module to obtain the video segment evaluation result;

[0121] 3.2.6) calculating the value of the loss function Loss2 according to the video segment evaluation result and the labeled value, and updating the network parameters using the gradient descent algorithm;

[0122] 3.2.7) repeating steps 3.2.2) to 3.2.6) until the loss function Loss2 converges, and obtaining the trained spatio-temporal region enhancement network F2;

[0123] 3.3) concatenating the trained repetitive motion positioning network F1 and the trained spatio-temporal region enhancement network F2 to obtain the trained developmental coordination disorder single-hand fine motion evaluation network F.

[0124] Step 4. inputting the test data set D v to the trained single-hand fine motion evaluation network to obtain the estimation result corresponding to the test data.

[0125] 4.1) sequentially batch taking c test sample videos from the test data set D v , inputting the test sample videos into the trained single-hand fine motion evaluation network to obtain the estimation result corresponding to the c test sample videos, wherein when the evaluation result is 1, it indicates that the motion is qualified, and when the evaluation result is 0, it indicates that the motion is unqualified motion;

[0126] 4.2) repeating step 4.1) until the test set D v is traversed completely to obtain the estimation result corresponding to all test data.

[0127] Example Two. Developmental Coordination Disorder Single-Hand Fine Motion Evaluation System Based on Two-Stage

[0128] With reference to Figure 4 , the system of the present example comprises an information input unit (1), a data acquisition unit (2), an automatic evaluation unit (3), a result feedback unit (4) and a display unit (5), wherein:

[0129] The information entry unit (1) is used to complete the basic information entry of the test subject and save the basic information of the test subject to the local database. That is, the test subject needs to fill in all the information required by the system into the system, including: name, number, gender, height, date of birth, weight and dominant hand. Otherwise, the information will fail to be saved. The system will calculate the test subject's real age based on the collection date. In addition, the subsequent fine motor video will be classified and managed according to the test subject's number.

[0130] The data acquisition unit (2) is used to record the complete process of the test subject performing fine motor skills with one hand and save the recorded video to a folder named after the corresponding number of the test subject.

[0131] The automatic evaluation unit (3) is used to automatically call the trained developmental coordination disorder single-hand fine motor assessment network F to evaluate the single-hand fine motor video of the test subject:

[0132] First, the repetitive motion localization in the first stage is carried out to obtain the motion probability density map DM and the repetitive motion localization result set S of the single-hand fine motion video.

[0133] Secondly, a second stage of spatiotemporal region enhancement is performed based on the result set S. This involves first segmenting the video into video segments based on the start and end frames of the repeated actions in the result set S, and then performing feature extraction, spatiotemporal feature enhancement, and feature aggregation on all video segments to obtain the evaluation result set R. s ;

[0134] Finally, the action probability density map DM, the repetitive action localization result set S, and the evaluation result set R are combined. s All are saved to the database;

[0135] The result feedback unit (4) is used to feed back the evaluation results from the database to the test subjects. Its implementation is as follows:

[0136] Obtain the evaluation results of test subjects from the database, including the action probability density map DM, the repetitive action localization result set S, and the evaluation result set R. s ;

[0137] According to R s The evaluation results for obtaining the complete video depend on whether it contains non-standard movements: when the result set R s If the video does not contain any non-standard movements, the evaluation result R is displayed as qualified, and the score is determined based on the completion time of the one-handed task in the video.

[0138] The standard for the MovementABC-2 reference book yields the scoring results when the result set R s When non-standard actions are included, the evaluation result R is displayed as unqualified, and the location of the non-standard action in the result set S is obtained;

[0139] The motion probability density map DM, the repetitive motion localization result set S, the evaluation result R of the complete video, and the localization of non-standard motions in the result set S are fed back to the test subject.

[0140] Display unit (5) is used to display the information input unit interface and the result feedback unit interface, wherein:

[0141] Information entry unit interface as follows Figure 5 The screen displays the basic information the test taker needs to enter, along with an "OK" button. After entering all the information, click the "OK" button to submit. If the submission is successful, a notification will appear at the bottom of the screen.

[0142] The message "Entry successful" will be displayed.

[0143] The result feedback unit interface is as follows: Figure 6 The display shows the feedback results obtained and statistically analyzed by the feedback unit, including the motion probability density map DM, the repetitive motion localization result set S, and the evaluation result R of the complete video. In addition, if the result R is qualified, the score of the complete video is displayed; otherwise, the localization of non-standard motions in the result set S is displayed.

[0144] The effects of this invention can be further illustrated by the following simulation.

[0145] 1. Simulation conditions

[0146] The experimental simulation of the repetitive action localization network F1 in the first stage was performed on the pin-insertion dataset D. peg The network F1 training is conducted using the pinned sub-dataset D. peg The training set D t Network testing uses the pinned sub-dataset D peg Test set D in v The Adam optimizer is used, with the Mean Squared Loss (MSE) function as the loss function, and a base learning rate of 5 × 10⁻⁶. -4 The network training epochs were set to 100 epochs, with 10 epochs for early stopping, and a batch size of 1. During both training and testing, the LSTM blocks in the network were configured with 3 layers, an input feature dimension of 84, a hidden layer size of twice the input feature dimension (168), a Transformer block with an input / output dimension of 168, 4 layers, 16 attention heads, and 16 features per attention head. The feedforward layer dimension was set to 16, and the dropout rate was set to 0.2.

[0147] Experimental simulations of the spatiotemporal region augmentation network F2 in the second stage were performed on the pinned dataset D. peg The network F2 training was conducted using the pinned sub-dataset D.peg the training set D in t , the network test uses the plug-in data set D peg the test set D in v . The training process uses the Adam optimizer, and the basic learning rate is 8x10 -7 . The model is trained for 50 rounds, and the early stop setting is 8 rounds, and the batch size is set to 4. The input video is down-sampled to 64 frames at equal intervals, and the weighted cross-entropy loss function WeightedCrossEntropyLoss is used as the loss function.

[0148] 2. Simulation content

[0149] Simulation 1: Under the simulation conditions of the first stage described above, repeated action positioning using different networks is performed on the test set D v , and the mAP evaluation index under the specified tIOU is calculated, and the results are shown in Table 1:

[0150] Table 1. Experimental results of action positioning comparison methods

[0151]

[0152] In Table 1: YOLOv5+post-processing is a network composed of a YOLOv5 model and a post-processing module;

[0153] Only object trajectory density F t is a network composed of a preprocessing module, an action positioning module and a post-processing module, and the preprocessing module of the network uses a target detection algorithm to obtain an object trajectory density map;

[0154] Only hand trajectory network F h is a network composed of a preprocessing module, an action positioning module and a post-processing module, and the preprocessing module of the network uses a pose estimation model to obtain a hand trajectory map.

[0155] From Table 1, it can be seen that, except under the condition of tIOU=0.9, the mAP index of the repeated action positioning network F1 in the application is lower than that of the network YOLOv5+post-processing, and under other tIOU, the mAP index of the F1 network in the application is the highest, indicating that the repeated action positioning network F1 in the application can effectively complete the positioning of repeated actions.

[0156] Simulation 2. Under the simulation conditions of the second stage described above, repeated action positioning using different networks is performed on the test set D vTo verify the effectiveness of the spatiotemporal region enhancement network F2 proposed in the application in single-hand fine motion evaluation, accuracy and F1-Score are used as evaluation indexes, and Video Swin Transformer and TSA-Net models which perform better in motion evaluation tasks are selected for comparison test, and the results are shown in Table 2.

[0157] Table 2. Experimental results of comparative methods of motion evaluation

[0158] Method Accuracy F1-Score VideoSwinTransformer 93.43 50.74 TSA-Net 95.82 41.79 The timing region enhancement network F2 in the present invention 96.71 63.73

[0159] As shown in Table 2, the timing region enhancement network F2 in the application is better than Video Swin Transformer and TSA-Net in all indexes, wherein the timing region enhancement network F2 in the application reaches 63.73% in the F1-Score index, which is improved by 21.94% compared with TSA-Net and 12.99% compared with Video Swin Transformer, and the results show that the timing region enhancement network F2 in the application can effectively evaluate single-hand fine motion.

[0160] Simulation 3. In the test set D v The results of significant analysis of the automatic scoring of the system result feedback unit and the score label true value are shown in Table 3. Curve fitting of the automatic scoring and the score label value further verifies the accuracy of the automatic scoring, and the results are shown in Figure 7 , wherein the abscissa is the label value, and the ordinate is the automatic scoring value.

[0161] Table 3. Significant analysis of automatic scoring and score label true value

[0162] Mean error ± standard deviation P-Value 1.37±2.41 7.01 x 10 -7 ]]

[0163] As shown in Table 3, the average error value of the automatic scoring of the system result feedback unit and the score label true value is low, and the F test result of the two variables shows that at the significant level of p<0.001, the automatic scoring result of the system is significantly related to the score label true value.

[0164] As shown in Table 3, the average error value of the automatic scoring of the system result feedback unit and the score label true value is low, and the F test result of the two variables shows that at the significant level of p<0.001, the automatic scoring result of the system is significantly related to the score label true value. Figure 7 As shown in Table 3, the average error value of the automatic scoring of the system result feedback unit and the score label true value is low, and the F test result of the two variables shows that at the significant level of p<0.001, the automatic scoring result of the system is significantly related to the score label true value.

[0165] The above results show that the two-stage developmental coordination disorder single-hand fine motion evaluation method of the application can effectively improve the accuracy of single-hand fine motion evaluation.

[0166] The above description is only one specific example of the present application and does not constitute any limitation to the present application. It is obvious to those skilled in the art that, after understanding the content and principles of the present application, various modifications and changes in form and details can be made without departing from the principles and structures of the present application, and these modifications and changes based on the idea of the present application are still within the protection scope of the claims of the present application.

[0167] It should be noted that the step numbers in the present application specification and claims are only for clearly describing the embodiments of the present application and facilitating understanding, and the sequence of the numbers is not limited.

Claims

1. A method for assessing fine motor skills in one hand in developmental coordination disorders based on visual-behavioral analysis, characterized in that, Includes the following steps: (1) Construct a single-hand fine motor assessment task dataset D based on the developmental coordination disorder assessment tool book Movement ABC-2. peg : Recruit children aged 7 to 10 for data collection. Record videos of children performing a nail-inserting task in a single-hand fine motor skills assessment task using a camera, and annotate the videos according to the Movement ABC-2 reference book. D peg The dataset is divided into training set D in an 8:2 ratio. t and test set D v ; (2) Construct a single-hand fine motor assessment network F for developmental coordination disorders, including a repetitive motion localization network F1 and a spatiotemporal region enhancement network F2; The repetitive action localization network F1 includes: The data preprocessing module is used to preprocess the complete nail insertion video. It consists of a pose estimation model and an object detection model connected in parallel. The pose estimation model is used to obtain hand trajectory data D. h This object detection model is used to detect objects in a complete video of nail insertion and obtain object trajectory density data D. s D h and D s splicing in the channel dimension Obtain Preprocessed data D l ; The localization module, based on LSTM and Transformer, consists of LSTM blocks, Transformer blocks, and linear layers connected in series. It is used to process the preprocessed data D. l Perform feature fusion and output the action probability density map DM. The post-processing module is used to post-process the motion probability density map DM using peak detection algorithm and differential method to obtain the motion localization set S consisting of the start frame and end frame positions of each repeated motion; The data preprocessing module, the positioning module, and the postprocessing module are cascaded in sequence to form a repetitive action positioning network F1; Its input is a complete video of nail insertion, and its output is a set of motion localizations S consisting of the positions of repeated actions in the video; The repetitive motion localization network F1 is used in the first stage to locate the start and end time regions of repetitive motions in the input video V, obtaining a motion localization set S consisting of the start and end frame positions of each repetitive motion: ； Among them, B n and E n These are the start and end frames of the nth action in the input video V, where n ranges from 1 to N. The spatiotemporal region enhancement network F2 is used in the second stage to process the video clip set. Each video clip v n The included single-handed motor skills were evaluated to obtain a set of repetitive motor skill evaluation results R. s : ； in, v is a set of video segments obtained by segmenting based on the result set S from the repetitive action localization stage. n For the nth video segment after segmentation, r n This represents the evaluation result of the nth video segment, with a value of 0 indicating non-standard action or 1 indicating standard action, where n ranges from 1 to N. (3) Using training set D t The repetitive motion localization network F1 and the spatiotemporal region enhancement network F2 in the single-hand fine motion evaluation network are iteratively trained using the gradient descent method until their respective loss functions converge, thus obtaining the trained single-hand fine motion evaluation network F. (4) Test dataset D v The data is input into a pre-trained single-hand fine motor evaluation network to obtain the estimation results corresponding to the test data.

2. The method according to claim 1, characterized in that, In step (1), D is determined according to the Movement ABC-2 reference book. peg The process of annotating the dataset videos involves first annotating the start and end frames of repeated actions in the complete video, and then annotating the completion status of each action, that is, marking non-standard actions as 0 and standard actions as 1.

3. The method according to claim 1, characterized in that, The spatiotemporal region augmentation network F2 in step (2) includes: The feature extraction module consists of a pose estimation model and a video encoder model connected in series. The pose estimation model is used to obtain the position information of the hand region, and the video encoder model is used to obtain the features of the video segment. The spatiotemporal feature enhancement module, which consists of a location matching block and a self-attention block, is used to enhance the information of the hand region in the video clip feature and obtain the feature-enhanced video clip features. The feature aggregation module, consisting of 3D convolutional blocks, pooling layers, and linear layers, is used to aggregate the enhanced video segment features and obtain the evaluation result set R of the video segments. s ; The feature extraction module, spatiotemporal feature enhancement module, and feature aggregation module are cascaded to form a spatiotemporal region enhancement network F2. Its input is a video segment obtained by segmenting from the action localization set S, and its output is a set of repeated action evaluation results R. s .

4. The method according to any one of claims 1 to 2, characterized in that, Step (3) uses the training set D t The F1 localization network for repetitive actions was trained using gradient descent, as follows: (3a) Randomly select from training set D t Select x training sample videos and input them into the data preprocessing module for preprocessing to obtain the preprocessed data D. l ; (3b) D l The input is fed into a localization module based on LSTM and Transformer for feature fusion, and the output is an action probability density map DM; (3c) Select MSE as the loss function Loss of the repetitive action localization network F1, calculate the loss function value according to the action probability density map DM, and update the network parameters using the gradient descent algorithm; (3d) Repeat steps (3a) to (3c) until the loss function Loss converges, and obtain the trained repetitive action localization network F1.

5. The method according to any one of claims 1 to 3, characterized in that Step (3) uses the training set D t The spatiotemporal region enhancement network F2 was trained using gradient descent, as follows: (3e) Randomly select from training set D t Select y training sample videos and decompose them into video segments based on the start and end frames of repeated actions in the annotations; (3f) Input the video clip into the feature extraction module to obtain the location information of the hand region in the video clip and the features of the video clip; (3g) Input the hand region location information and video segment features into the spatiotemporal feature enhancement module to obtain the video segment features after feature enhancement; (3h) Input the enhanced video segment features into the feature aggregation module to obtain the video segment evaluation results; (3i) WeightedLoss is selected as the loss function Loss of the spatiotemporal region augmentation network F2. The loss function value is calculated based on the video segment evaluation results, and the network parameters are updated using the gradient descent algorithm. (3j) Repeat steps (3e) to (3i) until the loss function Loss converges, and obtain the trained spatiotemporal region augmentation network F2.

6. The method according to claim 1, characterized in that, Step (4) will test dataset D v The data is input into a pre-trained single-handed fine motor evaluation network to obtain the estimation results corresponding to the test data, as follows: (4a) Sequentially from test dataset D v Take c test sample videos in a batch, input the test sample videos into the trained single-hand fine movement evaluation network, and obtain the estimation results corresponding to c test samples; (4b) Repeat step (4a) until test set D is reached. v After iterating through all test data, obtain the estimates corresponding to all test data. result.

7. The method according to claim 1, characterized in that, The target detection model detects the complete nail insertion video and obtains object trajectory density data D. s The implementation is as follows: First, the existing object detection algorithm YOLOv5 is used to obtain the curve P of the number of objects—nails—in the video over time. Next, the curve P is filtered. That is, the position where the quantity of each object first appears is found through the curve P, and this position is used as the starting position of the next action. The position is subtracted by two frames to get the ending position of the action. Then, based on the start and end times of the action, the filtered curve P is obtained. This curve P is then subjected to differencing and smoothing to obtain the object trajectory density data D. s .

8. The method according to claim 3, characterized in that, The spatiotemporal feature enhancement module enhances the information of the hand region in the video clip features by first locating the position R of the hand region in the video clip features using the position matching block, and then enhancing the hand features at position R using the self-attention block.

9. A single-handed fine motor assessment system for developing coordination disorders to implement the method of claim 1, characterized in that, It is based on the development coordination barrier assessment tool book Movement ABC-2 and includes: information input unit, data collection unit, automatic assessment unit, result feedback unit and display unit; The information input unit is used to input the basic information of the test subject, including name, number, date of birth, height, weight, dominant hand and gender, and to persistently store the information in the database; The data acquisition unit is used to automatically call the Azure Kinect camera to capture the process of the test subject performing a one-handed task and save it to a specified folder. The automatic evaluation unit is used to automatically invoke the evaluation algorithm to perform a two-stage evaluation of the video data: In the first stage, repetitive actions are located in the video data. In the second stage, the video is segmented into video segments based on the repetitive location results. The video segments are then evaluated to obtain a set containing the evaluation results of each video segment. The evaluation result of the complete video is obtained based on whether the result set contains non-standard movements; If the result set does not contain any non-standard actions, the evaluation result of the complete video is qualified, recorded as 1, and the score result is obtained according to the standards of the Movement ABC-2 reference book based on the completion time of the task in the video. Otherwise, it is considered unqualified and recorded as 0; Save the evaluation results to the database; The result feedback unit is used to obtain the video evaluation results recorded in the database and feed them back to the test subject; The display unit is used to display the results of each functional unit.

Images