Apparatus for assessing fine motor skills of a target user
By combining multimodal data collection and machine learning algorithms, the problem of low accuracy in Chinese handwriting assessment in existing technologies has been solved, achieving highly accurate handwriting quality assessment and guidance, which is applicable to special education and primary and secondary school handwriting education.
Patent Information
- Application Number
- CN202511994394.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2024-12-30
- Filing Date
- 2025-12-26
- Publication Date
- 2026-06-30
Smart Images

Figure CN122310385A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an apparatus for assessing the fine motor skills of a target user, and particularly, but not exclusively, to an apparatus for assessing the handwriting of a target user. Background Technology
[0002] According to statistics from the Education Bureau, there were 333,551 registered students in Hong Kong primary schools in 2022 / 2023.
[0003] Writing easily occupies 60% of school time and is an important task for children. During school hours, children spend approximately 30-60% of their time on fine motor activities, especially writing tasks. Other fine motor activities, such as manipulating blocks, building puzzles or models, or drawing, are important skills that children develop in early childhood. As part of child development and continuing education care, it is beneficial to assess their performance as early as possible and provide any necessary correction, encouragement, or intervention as soon as possible. Summary of the Invention
[0004] According to a first aspect of the present invention, an apparatus for assessing the fine motor skills of a target user is provided, comprising: - A pressure sensing module arranged to capture pressure data associated with the pressure exerted by small muscles by the target user during the performance of the fine motor skill; - A motion tracking module, which is arranged to capture motion data associated with the movement of the small muscles during the performance of the fine motor skill by the target user; - An eye-tracking module arranged to capture gaze data associated with the gaze direction of the target user during the performance of the fine motor skill; and - A performance evaluation module is configured to determine one or more metrics in real time that are associated with the accuracy of the target user's fine motor skills, based on captured motion data, captured gaze data, and captured stress data.
[0005] According to the first aspect, it also includes a user interaction panel arranged to collect pressure data associated with pressure exerted by the small muscles of the target user through physical contact between the target user's small muscles and the user interaction panel.
[0006] According to the first aspect, the fine motor skill includes a writing task, and the performance evaluation module is arranged to determine the stroke trajectory and stroke pressure of the writing task based on pressure data associated with the pressure exerted by the target user's fine muscles through physical contact between the target user's fine muscles and the user interaction panel.
[0007] According to the first aspect, the fine motor skill includes the task of writing a single character with multiple strokes, and the pressure sensing module is arranged to capture the pressure data associated with the pressure exerted by the small muscles of the target user in each stroke of the writing.
[0008] According to the first aspect, the performance evaluation module is arranged to determine the start and end segments of the plurality of strokes.
[0009] According to the first aspect, the performance evaluation module is arranged to align the captured motion data and captured gaze data with the captured pressure data associated with the same segments of the plurality of strokes.
[0010] According to the first aspect, the performance evaluation module is arranged to receive user input that indicates a timestamp associated with the fine motor skill performed.
[0011] According to the first aspect, the motion tracking module includes multiple camera units, each camera unit capturing the movement of the target user from different orientations.
[0012] According to the first aspect, the performance evaluation module is arranged to compare body posture data derived from captured motion data and captured gaze data with handwritten data derived from captured pressure data.
[0013] According to the first aspect, the performance evaluation module is configured to determine the joint stability (JS) of the target user during the execution of the fine motor skill based on the captured motion data.
[0014] According to the first aspect, the pressure sensing module is arranged to capture pressure changes (PV) of the small muscles of the target user during the execution of the fine motor skill.
[0015] According to the first aspect, the performance evaluation module is configured to determine a gaze stability index (GSI) based on the captured gaze data, which is associated with the gaze direction of the target user during the performance of the fine motor skill.
[0016] According to the first aspect, the performance evaluation module is configured to determine the pause duration (PD) during the execution of the fine motor skill based on the captured stress data.
[0017] According to the first aspect, the eye-tracking module includes a head-mounted eye tracker.
[0018] According to the first aspect, the performance evaluation module further includes a machine learning network model, which is trained using multiple training data related to the performance of multiple test users in performing fine motor skills.
[0019] According to the first aspect, the plurality of training data includes the test user's body position, focus, and stress.
[0020] According to the first aspect, the machine learning network model is configured to convert multiple training data associated with the test user's saccades, gazes, and fixation durations into feature maps.
[0021] According to the first aspect, the machine learning network is configured to extract focus recognition from the feature map.
[0022] According to the first aspect, the machine learning network further includes a convolutional neural network (CNN) configured to extract spatial features from the feature map.
[0023] According to the first aspect, the machine learning network further includes a recurrent neural network (RNN) and an artificial neural network (ANN), configured to extract temporal features from the feature map. Attached Figure Description
[0024] Embodiments of the present invention will now be described by way of example, with reference to the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall architecture of an evaluation device according to an exemplary embodiment of the present invention.
[0025] Figure 2 This is a schematic diagram illustrating the first stage of a training method for a machine learning network model according to an exemplary embodiment of the present invention.
[0026] Figure 3 This is a schematic diagram illustrating the second stage of a training method for a machine learning network model according to an exemplary embodiment of the present invention.
[0027] Figure 4 This is a schematic diagram illustrating the third stage of a training method for a machine learning network model according to an exemplary embodiment of the present invention.
[0028] Figure 5 Depicting by Figure 1 The evaluation device assesses well-written strokes.
[0029] Figure 6 Depicting by Figure 1 The evaluation device assesses poorly written strokes.
[0030] Figure 7 Depicting by Figure 1 The eye tracker of the device captures gaze heatmaps of target users with good handwriting.
[0031] Figure 8 Depicting by Figure 1The eye tracker of the device captures gaze heatmaps of target users with poor handwriting.
[0032] Figure 9 Depicting by Figure 1 The device captures the visual-motor integration of target users with good handwriting.
[0033] Figure 10 Depicting by Figure 1 The device captures the visual-motor integration of target users with poor handwriting.
[0034] Figure 11 Depicting by Figure 1 The device captures well-written 3D stroke visualizations.
[0035] Figure 12 Depicting by Figure 1 The device captures 3D stroke visualizations of poor handwriting.
[0036] Figure 13A A diagram showing the statements analyzed by a multimodal writing analysis platform according to an exemplary embodiment of the present invention.
[0037] Figure 13B A diagram showing the statements analyzed by a multimodal writing analysis platform according to an exemplary embodiment of the present invention.
[0038] Figure 14 It is a startup screen that displays the software interface operation according to an example embodiment of the present invention.
[0039] Figure 15 It is a startup screen that displays the software interface operation according to another example embodiment of the present invention. Detailed Implementation
[0040] Without wanting to be bound by theory, the inventors found that mainstream children's handwriting assessment solutions on the market focused on alphabetic systems, while Chinese handwriting presents unique challenges due to its complex strokes and spatial structure.
[0041] Currently, mainstream handwriting assessment methods using artificial intelligence technology focus solely on computer vision, which assesses children's handwriting results after the writing process has begun. In other words, major AI-based handwriting assessment systems primarily collect offline data—students writing on paper—and then use computer vision methods to uniformly train the data and analyze and evaluate the writing results.
[0042] Secondly, many existing technologies are based on online writing platforms and use digital tablets to simulate writing scenarios to obtain students' real-time writing data. However, this technology is not based on real writing scenarios, so its application scope is limited. Furthermore, such inventions or research do not analyze and evaluate the writer's eye focus and body posture data, which will inevitably lead to low accuracy.
[0043] In practical applications, such as in scenarios where students' handwriting is evaluated for correctness and beauty, professional calligraphy teachers are often required to first assess the characters, label and classify the data as beautiful / unbeautiful, correct / incorrect, and then train an AI model that can achieve the desired results by combining machine learning algorithms with computer vision-related technologies. This involves a lot of work, and the accuracy is often unsatisfactory.
[0044] In one aspect of the invention, an artificial intelligence-based data collection and analysis system for eye-tracking trajectory, body posture, writing trajectory, and pressure is provided, thereby providing an intelligent writing assessment system with high accuracy, high responsiveness, high integration, and small size, which can solve one or more of the above-mentioned problems.
[0045] More specifically, this invention also provides a multimodal assessment scheme that highly integrates hardware devices such as eye trackers, cameras, and writing tablets. By using machine learning algorithms and eye-tracking instruments, cameras, writing tablets, and other hardware devices for communication, this invention can be applied to special education, primary and secondary school handwriting education, and other fields.
[0046] Reference Figure 1 This illustration shows an embodiment of the present invention. This embodiment is arranged to provide an apparatus 100 for evaluating the fine motor skills of a target user 10. The apparatus 100 for evaluating the fine motor skills of the target user 10 includes a pressure sensing module 110 arranged to capture pressure data associated with pressure exerted by small muscles by the target user during the performance of the fine motor skill. The apparatus 100 for evaluating the fine motor skills of the target user 10 also includes a motion tracking module 120 arranged to capture motion data associated with movement of the small muscles by the target user during the performance of the fine motor skill. The apparatus 100 for evaluating the fine motor skills of the target user 10 also includes an eye tracking module 130 arranged to capture gaze data associated with the gaze direction of the target user during the performance of the fine motor skill. Finally, the apparatus 100 for evaluating the fine motor skills of the target user 10 also includes a performance evaluation module 140 arranged to determine, in real time, one or more indicators associated with the accuracy of the target user's fine motor skills based on the captured motion data, the captured gaze data, and the captured pressure data.
[0047] For the purposes of this patent document, the phrase "fine motor skill" refers to the coordination of small muscles with eye, hand, and finger movements, and includes any type of small movement occurring in the wrist, hand, fingers, foot, and toes. The phrase "target user" includes children, patients, or the elderly. The phrase "indicators" includes various parameters such as the correctness and elegance of handwriting, stroke similarity, gaze, visual-motor integration, stroke pressure variation, etc.
[0048] Refer again Figure 1 To understand more details of the overall architecture of the writing evaluation system 100, i.e., the intelligent writing system according to an exemplary embodiment of the present invention, which is used to evaluate the writing of a target user 10 on a medium such as work paper 14 printed on a sheet of paper using a writing instrument 12 (e.g., a pen or pencil).
[0049] Essentially, the writing assessment system 100 includes a pressure sensing module 110 for sensing the pressure applied by the target user 10 to the work paper 14 via the pen 12 during writing. A motion tracking module 120 and an eye-tracking module 130 are also provided for sensing the target user 10's body posture relative to the work paper 14. These modules are operable to capture pressure data, motion data, and gaze data related to writing performance. These signals are then processed by the performance assessment module 140.
[0050] In one example embodiment, the pressure sensing module 110 can be embedded within a user interaction panel in the form of a smart writing tablet. The user interaction panel can collect pressure data associated with the pressure applied by the small muscles of the target user 10 through physical contact between the user interaction panel and the small muscles of the target user 10.
[0051] In one example embodiment, the motion tracking module 120 may further include one or more image capture modules 122, 124 for capturing posture data of the target user 10 during writing. For example, the first image capture module 122 may be a high-speed RGB camera for capturing a side 3D view of the target user 10, and the second image capture module 124 may be another high-speed RGB camera for capturing a front 3D view of the target user 10. The posture data may include the angles of the target user 10's limbs (e.g., neck and elbow) relative to a reference axis.
[0052] In one example embodiment, the eye-tracking module 130 may be provided in the form of a head-mounted eye tracker for capturing gaze data associated with the gaze direction of the target user 10 during writing. For example, the eye-tracking module 130 may emit one or more beams of light to detect the object or coordinates that the target user 10 is looking at. More preferably, the eye-tracking module 130 may emit multiple beams of light to detect the object or coordinates that the target user 10's left and right eyes are looking at, respectively. Thus, the eye-tracking module 130 can accurately monitor and analyze eye movements and the gaze direction of the eyes relative to the pen 12 or the work paper 14.
[0053] In one example embodiment, a performance evaluation module 140 may be provided to evaluate handwriting based on several metrics and references. The performance evaluation module 140 may provide rating scales based on the similarity between the written strokes and template strokes. For example, various components of the handwriting evaluation system 100 may be coordinated to track different factors, thereby generating one or more outputs based on processed data.
[0054] For example, the performance evaluation module 140 can determine the limb movements of the target user 10, such as the left and right elbows, and the left and right necks, based on a combination of captured motion data and captured gaze data. The performance evaluation module 140 can also determine the stroke trajectory and stroke pressure of the writing task based on captured pressure data.
[0055] In one example embodiment, the performance evaluation module 140 may include a computing module 141, such as a laptop computer, which includes suitable components necessary for receiving, storing, and executing appropriate computer instructions. Components may include a processing unit 142, which includes a central processing unit (CPU), a math coprocessor (math processor), a graphics processing unit (GPU) or a tensor processing unit (TPU) for tensor or multidimensional array computation or manipulation operations, read-only memory (ROM) 144, random access memory (RAM) 146, and input / output devices such as a disk drive 148, and a user interface 150 such as a keyboard or touchscreen. The processing unit 142 may be a single processor to provide combined functionality of multiple processors. In this example embodiment, the computing module 141 is configured to receive data associated with a writing exercise performed by the target user 10 and the environment measured by external sensing units 110, 120, 130.
[0056] The computing module 141 may include other input devices, such as an Ethernet port, a USB port, etc. A display 160, such as a liquid crystal display, a backlight display, or any other suitable display, and a communication link (i.e., a communication interface) 170. The display 160 may graphically represent template characters written by an expert so that the target user 10 can follow the strokes during writing exercises.
[0057] The computing module 141 may include instructions that can be contained in ROM 144, RAM 146, or disk drive 148 and executed by the processing unit 142. One or more communication interfaces (i.e., one or more communication links) may be provided, which can be connected to one or more computing devices, such as servers, personal computers, terminals, wireless or handheld computing devices, Internet of Things (IoT) devices, smart devices, and edge computing devices. At least one of the multiple communication links may be connected to an external computing network via a telephone line or communication link type. The communication interfaces are configured to allow data communication via any suitable communication network using any suitable protocol, such as, for example, Wi-Fi, Bluetooth, 4G, 5G, or any other suitable protocol.
[0058] The performance evaluation module 140 may further include a machine learning network model that is pre-trained using multiple training data related to the performance of multiple test users in performing fine motor skills. The machine learning network model may be stored locally in read-only memory (ROM) 144 or remotely on cloud server 180. Therefore, data can be processed locally in computing module 141 for AI, or alternatively transmitted via communication network 170 to cloud server 180 for processing along with other data and analytics. Advantageously, the evaluation of each round of writing by target user 10 can also be used to retrain the machine learning network model of performance evaluation module 140.
[0059] In one example embodiment, the technical solution adopted by the present invention includes a multimodal data analysis system 100, including a head-mounted eye tracker 130, two high-speed RGB cameras 122 and 124, a smart handwriting tablet 110, and a laptop computer 141 running a software system.
[0060] Centered on a laptop computer 141 equipped with an intelligent writing system 100, it can coordinate a head-mounted eye-tracking device 130, a high-speed RGB camera 120, an intelligent handwriting tablet 110, and other hardware devices, and interact with the data of each hardware device through a real-time communication protocol. After collecting data, machine learning algorithms are used to scientifically analyze and evaluate the writing results.
[0061] In a preferred embodiment, the computing module 141 may further store the core algorithm as one or more executable instructions in a read-only memory (ROM) 144, which can be executed by the processing unit 142. The core algorithm may include a data alignment algorithm and a stroke splitting algorithm.
[0062] For example, data alignment algorithms can be implemented using platforms developed with operating systems such as Python. Specifically, system 100 splits into multiple threads to monitor RGB camera 120 and scene camera 124 in real time and streams the data to thread processing program 140. Through frame segmentation and frame synchronization algorithms, system 100 compares the body posture data recorded by cameras 122 and 124 with the handwriting data, and ultimately compares the data from different devices, which will provide valuable raw data for subsequent overall handwriting quality assessment. Regarding the stroke segmentation algorithm, this invention provides a unique solution for extracting children's handwriting strokes with 100% accuracy and ultimately saving the handwriting data as individual characters as the basis for data analysis. For example, system 100 can receive signal input from target user 10, indicating the start and end of a single-character writing task.
[0063] In one specific embodiment, the intelligent writing system 100 may further include a button (not shown) that is in signal communication with the computing module 141, for example via USB or other wireless signal communication. The child 10 can press the button after completing a single-word writing task to align the strokes with data captured by the eye tracker 130 and camera 120, thereby calculating data at a finer granularity. When the child 10 presses the button after completing a single-word writing task, the computing module 141 can save the data of the single writing task, then distribute the writing pressure values to calculate the stroke data as the start and end segments, then align the data with the data captured by the eye-tracking device 130 and camera 120, ultimately achieving stroke segmentation, extraction, and calculation.
[0064] This invention also innovates a novel artificial intelligence writing quality assessment platform that combines multimodal features, such as gaze stability index (GSI), pressure variation (PV), joint stability (JS), and pause duration (PD), and can provide highly accurate writing quality assessment results based on real children's writing process, and provide effective guidance and suggestions for improving children's writing quality.
[0065] For example, the gaze stability index (GSI) is associated with the gaze direction of the target user 10 during writing and can be determined based on gaze data captured by the eye tracker 130. Pressure variation (PV) is associated with the pressure exerted by the small muscles of the target user 10 during writing and can be determined based on pressure data captured by the pressure sensing module 110. Joint stability (JS) is associated with the limb movements of the target user 10 during writing and can be determined by the motion tracking module 120 based on captured motion data. Pause duration (PD) is associated with the cumulative pause between consecutive strokes in a single-word writing task and can be determined based on pressure data captured by the pressure sensing module 110.
[0066] Reference Figures 2 to 4 This illustrates a method for training a machine learning network model according to an exemplary embodiment of the present invention. Illustratively, the implementation of the training method may be based on three main phases, such as an initial phase of data collection (step 200), an intermediate phase of data analysis (step 300), and a final phase of training the model (step 400).
[0067] Phase 1: Data Collection In this example embodiment, a multimodal data collection system is provided, which is trained based on eye movement, body posture, handwriting, pressure, and other data. The system's training requires the collection of a large amount of children's handwriting data. Figure 2 One of the pre-training settings shown provides a setup similar to the writing evaluation system 100 described above, and includes a user interaction panel 210, a pair of high-speed RGB cameras 222, 224, and a head-mounted eye tracker 230 for capturing data associated with writing tasks performed by the test user 20 using the writing tool 22.
[0068] Phase 2: Data Analysis Based on the analysis of massive amounts of data, the inventors identified factors significantly related to children's handwriting quality. For example, the body position, concentration, and pen pressure of test users 20 were significantly correlated with children's handwriting quality, and information was collected accordingly.
[0069] Phase 3: Training the Model Using the processed data, the inventors trained an artificial intelligence model using deep learning techniques capable of effectively assessing children's handwriting quality. The machine learning network model is configured to convert multiple training data associated with the saccades, gazes, and fixation durations of the test user 20 into feature maps 410, which map data vectors to a feature space. The machine learning network model may include a convolutional neural network (CNN) for providing spatial feature learning 420 from the feature maps 410. The machine learning network model may also include recurrent neural networks (RNNs) and artificial neural networks (ANNs) 430, configured to extract temporal features from the feature maps 410. Therefore, the machine learning network model can achieve attention recognition associated with the test user 20. Compared to existing technologies, the spatiotemporal fusion model of this invention can combine dynamic and morphological features and improve the accuracy of handwriting quality assessment.
[0070] Now refer to Figures 5 to 12 Further explanation is needed regarding the parameters or metrics used by the performance evaluation module 140 to assess handwriting quality. For example, handwriting assessment can be quantified using various metrics such as stroke similarity scores, gaze heatmaps, visual-motor integration, and stroke pressure variations.
[0071] In one exemplary embodiment, multiple test subjects 10 (e.g., a group of children) are required to complete the same writing task of some Chinese characters, and the writing will be evaluated by a performance evaluation module 140.
[0072] Figure 5 Good writing 500 of the first tested child is shown, while Figure 6 Poor writing 600 of the second tested child is shown. A comprehensive evaluation was carried out referring to several dimensions, as shown in the following table:
[0073] Table 1 Comprehensive evaluation of children's writing ability based on eye movement data.
[0074] A. Stroke similarity score: After processing the data through the above-mentioned stroke splitting algorithm, the slope and curvature of each single stroke are calculated respectively. Based on the above data, the stroke similarity between the template character written by an expert and the character written by a child is calculated, and this value is compressed to the range of 0 to 100.
[0075] Illustratively, as Figure 5 shown, the good writing 500 scores 90 in the evaluation by a human expert, while as Figure 6 shown, the poor writing 600 only scores 50 in the evaluation by the same human expert. The good writing 500 obtains a stroke similarity score of 60, while the poor writing 600 obtains a stroke similarity score of only 38. Based on the comparison between the stroke similarity score and the human expert evaluation, it shows that there is a positive correlation between the writing quality, that is, the human score and the AI score based on the stroke similarity evaluation.
[0076] The fixation heatmap and visual-motor integration are also determined during the writing task. For illustration, the fixation heatmap and visual-motor integration of a single writing task of the Chinese character "乖" are recorded by Figure 7 and Figure 9 respectively, while the fixation heatmap and visual-motor integration of a single writing task of the Chinese character "花" are recorded by Figure 8 and Figure 10 respectively.
[0077] B. Gazing at the heat map: The gaze direction of test target 10 was captured by eye tracker 130 and plotted as heatmaps 700 and 800. Heatmap 700 for a child 10 with good handwriting shows that the child's gaze is focused on the worksheet area 710, while heatmap 800 for a child 10 with poor handwriting shows that the child's gaze is scattered across the screen area 810, the left hand area 820, and the keyboard area 830. Based on the gaze convergence generated by the eye tracker, the gaze of a child with good handwriting is more focused, while the gaze of a child with poor handwriting is more scattered.
[0078] C. Visual-motor integration: Writing ability is defined by the hand-eye coordination coefficient. Children who write well usually have a more stable gaze path and only focus on the necessary information (work paper, pen, screen).
[0079] The posture data (e.g., hand movements) of test target 10 are captured by image capture modules 122 and 124, while the gaze direction is captured by eye tracker 130. The gaze path in the integration 900 of a child 10 with good handwriting shows that the child only stays on the necessary information within the worksheet area 910, pen area 920, and screen area 930, while the gaze path in the integration 1000 of a child 10 with poor handwriting shows that the child stays on some unnecessary information around the keyboard area 1010 in addition to the screen area 1020.
[0080] D. Changes in stroke pressure: Stroke and writing data are captured by pressure sensing module 110 and processed to be projected onto a 3D visualization of X-coordinates, Y-coordinates, and stroke numbers. Specifically, the pressure magnitude of each stroke is captured by pressure sensing module 110, and then the pressure variation between consecutive strokes is calculated.
[0081] The 3D stroke visualization of a child with good handwriting (10) shows that the pressure of each stroke in a single character is consistent, while the 3D stroke visualization of a child with poor handwriting (10) shows that the pressure of each stroke in a single character is inconsistent, with low pressure at the beginning (1210) and high pressure at the end (1220) of each stroke. By reproducing the pressure distribution of strokes through 3D visualization, it is clear that children with good handwriting exhibit less variation in pen tip pressure while writing. Figure 11 As shown, the overall color is more uniform. For example, as Figure 5 The 3D stroke visualization of good writing 500 shows a stroke pressure change of 389.660, while... Figure 6 The 3D stroke visualization of poor handwriting 600 shows a stroke pressure change of 547.628, which is much higher than that of good handwriting 500.
[0082] Reference will now be made to Figures 13A to 13B describe the details of a multimodal writing analysis platform according to an exemplary embodiment of the present invention. According to an exemplary embodiment of the present invention, a target user 10 is tested under a writing evaluation system 100, and an interface 1300 of the multimodal writing analysis platform is as Figures 13A to 13B shown, which depicts the test results.
[0083] Specifically, the tested student 10 is required to perform a single writing task of the Chinese character "look". The student information is recorded in the information section 1310. Stroke analysis is performed based on the pressure data captured by the pressure sensing module 110, and the analyzed information is displayed on the analysis section 1320. For example, the analyzed font, stroke count, average writing speed, standard writing speed, writing path length, average pressure change speed, and standard pressure change speed are determined and displayed on the analysis section 1320.
[0084] The gaze data of the left and right eyes captured by the eye tracker 130 is processed and graphically represented as an eye movement trajectory video 1330. The X and Y coordinates of the gaze direction are also plotted in the plotting section 1340 for the nib pressure recorded by the pressure sensing module 110.
[0085] Preferably, the gaze data captured by the eye tracker 130 and the pose data captured by the image capture modules 122, 124 can be processed by the processor 142 to determine the pose of the tested student 10. For example, the motion data and gaze direction data captured in the image can be processed to determine the limb movements of the tested student 10, such as the left and right elbow angles and the left and right neck angles, respectively. The elbow and neck movements of the limbs are recorded over time, and the frequencies of different angle ranges are also recorded. The average angle and standard angle of the limbs are also calculated.
[0086] For example, the frequency histogram of the left elbow angle and the left elbow angle over time are displayed in the pose left elbow analysis section 1350, and the frequency histogram of the right elbow angle and the right elbow angle over time are displayed in the pose right elbow analysis section 1360. Similarly, the frequency histogram of the left neck angle and the left neck angle over time are displayed in the pose left neck analysis section 1370, and the frequency histogram of the right neck angle and the right neck angle over time are displayed in the pose right neck analysis section 1380.
[0087] Finally, reference will now be made to Figures 14 to 15 describe the operation of a software interface according to an exemplary embodiment of the present invention.
[0088] Figure 14The interface 1400 of the multimodal intelligent writing evaluation platform is depicted, which can graphically present the corresponding data on the display panel. The “Realsense1” sector 1410 depicts motion data captured by the first high-speed RGB camera 122, while the “Realsense2” sector 1420 depicts motion data captured by the second high-speed RGB camera 124. The “Handwritingpad” sector 1430 depicts pressure data captured by the pressure sensing module 110, while the “Eyetracker” sector 1440 depicts gaze data captured by the eye tracker 130.
[0089] Figure 15 Another interface 1500 of the multimodal intelligent handwriting assessment platform is depicted, which is a master data management database. Multiple target users 10 can be tested by the handwriting assessment system 100 and subsequently stored for further analysis.
[0090] Advantageously, this invention is based on an AI model with multimodal data that is self-designed and self-trained, enabling it to reproduce children's writing process to the greatest extent possible in real-life scenarios. Modalities include eye movement, skeleton, stroke trajectory, and stroke pressure. This invention also provides a children's handwriting quality assessment platform trained using multimodal data fusion and artificial intelligence technologies such as deep learning. This platform integrates eye movement, pressure changes, and skeletal dynamics to provide higher accuracy and reliability than other handwriting quality assessment platforms.
[0091] Advantageously, the technical solution of this invention produces a variety of beneficial effects. By implementing a multimodal fusion handwriting quality assessment system based on machine learning algorithms, this expands the data collection scope of traditional handwriting scenarios and adds hardware devices such as eye trackers and cameras. Handwriting assessment based on real-world scenarios significantly improves data accuracy and dynamic response capabilities. Simultaneously, it meets the requirements of high integration, synchronous alignment of data from multiple hardware devices, unified management, statistics, and analysis. The architecture of this invention is also applicable to special education, and its application prospects for improving handwriting quality and assessment among primary and secondary school students of all ages are broad.
[0092] This invention is given by way of example only, and those skilled in the art can make various other modifications and / or alterations to the described embodiments without departing from the scope of the invention as defined in the appended claims. Those skilled in the art will understand that various changes and / or modifications can be made to the invention shown in the specific embodiments without departing from the spirit or scope of the invention as broadly described. Therefore, these embodiments are to be considered illustrative rather than restrictive in all respects.
[0093] Any reference to prior art in this document should not be construed as an admission that the information is common general knowledge, unless otherwise stated.
Claims
1. A device for assessing the fine motor skills of a target user, characterized in that, include: - A pressure sensing module arranged to capture pressure data associated with the pressure exerted by small muscles by the target user during the performance of the fine motor skill; - A motion tracking module, which is arranged to capture motion data associated with the movement of the small muscles during the performance of the fine motor skill by the target user; - An eye-tracking module arranged to capture gaze data associated with the gaze direction of the target user during the performance of the fine motor skill; as well as - A performance evaluation module, configured to determine one or more metrics in real time that are associated with the accuracy of the target user's fine motor skills based on captured motion data, captured gaze data, and captured stress data.
2. The apparatus according to claim 1, characterized in that, It also includes a user interaction panel arranged to collect pressure data associated with pressure exerted by the target user's small muscles through physical contact between the target user's small muscles and the user interaction panel.
3. The apparatus according to claim 2, characterized in that, in, The fine motor skills include writing tasks, and the performance evaluation module is arranged to determine the stroke trajectory and stroke pressure of the writing task based on pressure data associated with the pressure exerted by the target user's fine muscles through physical contact between the target user's fine muscles and the user interaction panel.
4. The apparatus according to claim 3, characterized in that, in, The fine motor skills include the task of writing a single character with multiple strokes, and the pressure sensing module is arranged to capture pressure data associated with the pressure exerted by the target user's small muscles in each stroke of the writing.
5. The apparatus according to claim 4, characterized in that, in, The performance evaluation module is configured to determine the start and end segments of the plurality of strokes.
6. The apparatus according to claim 5, characterized in that, in, The performance evaluation module is arranged to align the captured motion data and captured gaze data with the captured pressure data associated with the same segments of the plurality of strokes.
7. The apparatus according to claim 6, characterized in that, in, The performance evaluation module is configured to receive user input that indicates a timestamp associated with the fine motor skill performed.
8. The apparatus according to claim 1, characterized in that, in, The motion tracking module includes multiple camera units, each capturing the movement of the target user from different orientations.
9. The apparatus according to claim 1, characterized in that, in, The performance evaluation module is configured to compare body posture data derived from captured motion data and captured gaze data with handwriting data derived from captured pressure data.
10. The apparatus according to claim 1, characterized in that, in, The performance evaluation module is configured to determine the joint stability of the target user during the execution of the fine motor skill based on the captured motion data.
