Calligraphy teaching digital system and method

Abstract

The invention relates to a calligraphy teaching digital system and method, and belongs to the field of computer education software, and the system comprises a multi-modal data input module which comprises a binocular camera, a pressure induction pen and a gyroscope sensor, the binocular camera collects a writing video, the pressure induction pen and the gyroscope sensor collect pen wielding track, force and angle data, and the multi-modal data input module is used for inputting the writing video; the data is preprocessed; the intelligent analysis module is used for constructing a calligraphy evaluation model based on an attention mechanism and carrying out feature extraction and fusion on the multi-modal data input module to obtain calligraphy practice evaluation; and the resource management module comprises a video library and a copybook library. Various teaching decomposition videos and copybooks are stored; and the personalized learning module is used for analyzing the calligraphy practice evaluation result, associating the corresponding teaching decomposition video and copybook according to the analysis result, and carrying out personalized learning customization. Through the full-closed-loop teaching process of data acquisition, intelligent analysis and real-time feedback adjustment, the writing level of a writer can be quickly improved.

Description

Technical Field

[0001] The present invention relates to the field of computer education software, and in particular to a digital calligraphy teaching system and method. Background Art

[0002] Calligraphy is a core component of Chinese traditional culture and carries rich historical and cultural information.

[0003] However, traditional calligraphy teaching relies on face-to-face instruction by teachers, but there are many problems: for example, the number of calligraphy teachers is limited, the cost of traditional face-to-face teaching is high, etc. However, with the rapid development of modern science and technology, especially breakthroughs in computer vision, sensor technology, deep learning and other fields, it has provided unprecedented possibilities for the digitization and intelligence of calligraphy teaching; the digital calligraphy teaching system came into being in such a technological context. It aims to innovate the traditional calligraphy teaching model by integrating these advanced technologies, realize the digitization, intelligence and personalization of the teaching process, thereby improving the efficiency and quality of calligraphy teaching and promoting the inheritance and development of calligraphy culture. However, the current calligraphy teaching system mainly uses online teaching, which leads to the following problems:

[0004] 1. The subjectivity and inconsistency of calligraphy evaluation. Evaluation is highly dependent on the teacher's personal experience and aesthetic standards, resulting in highly subjective evaluation results and a lack of unified standards.

[0005] 2. Lack of personalized learning and correction exercises. Different learners have different needs and error patterns in calligraphy learning, and traditional teaching models cannot provide targeted guidance.

[0006] 3. Observing students’ calligraphy practice through video alone cannot fully discover students’ mistakes, and cannot accurately correct students’ calligraphy practice.

[0007] 4. Teaching resources are scattered and difficult to share, and there is a lack of comprehensive special training to correct students' errors. Summary of the Invention

[0008] In order to solve the above-mentioned problems in the prior art, the present invention provides a digital calligraphy teaching system and method.

[0009] A digital calligraphy teaching system, including

[0010] Multimodal data input module: includes a binocular camera, a pressure-sensitive pen, and a gyroscope sensor. The binocular camera collects writing videos, while the pressure-sensitive pen and gyroscope sensor collect pen trajectory, force, and angle data, and pre-processes the data.

[0011] Intelligent analysis module: Builds a calligraphy evaluation model based on the attention mechanism, extracts and fuses features from the multimodal data input module, and obtains calligraphy practice evaluation;

[0012] Resource management module: including video library and copybook library. It stores various teaching videos and copybooks;

[0013] Personalized learning module: Analyze the evaluation results of calligraphy practice, associate the corresponding teaching decomposition videos and copybooks based on the analysis results, and customize personalized learning.

[0014] Furthermore, the calligraphy evaluation model includes

[0015] Feature extraction module: includes visual feature extraction branch and temporal feature extraction branch;

[0016] The visual feature extraction branch uses a 3D-CNN neural network to extract the spatial-temporal features of the pen tip motion and wrist posture in the writing video;

[0017] The temporal feature extraction branch processes the sensor sequence through the LSTM encoder to extract the dynamic features of the pen trajectory, force and angle during the pen movement process;

[0018] Attention fusion module: including cross-modal attention fusion, spatial attention fusion and temporal attention fusion;

[0019] The cross-modal attention fusion uses scaled dot product attention to calculate the similarity matrix between visual features and temporal features, generates a cross-modal context vector, fuses visual and sensor information, and thus learns the association between modalities;

[0020] The spatial attention fusion generates a spatial attention weight map through a convolutional layer. The weight value reflects the importance of each pixel and highlights the key writing areas in the video.

[0021] The temporal attention fusion uses temporal attention weights to weight feature representations to highlight key time steps or dynamic patterns, capturing the dynamic changes in force and angle during the pen movement process.

[0022] Evaluation module: The features after attention fusion are input into the fully connected layer to perform a comprehensive evaluation of the calligraphy quality.

[0023] Furthermore, when training the calligraphy evaluation model, the loss function optimizes the associated regression task and classification task simultaneously through multi-task learning: L total =λL MSE +(1-λ)L CE ,

[0024] Among them, L total is the joint loss function; λ is a hyperparameter used to control the weights of the two tasks; L MSE is the mean square error of the regression task; L CEis the cross entropy loss for classification tasks.

[0025] Furthermore, the comprehensive evaluation of calligraphy quality includes:

[0026] Force mutation detection: Use the temporal attention mechanism to identify mutation points in the pressure sequence and calculate the mutation frequency as an indicator of force stability.

[0027] Angle deviation detection: Compare the difference between the actual angle and the standard writing angle, generate an angle deviation heat map, and highlight the areas with large deviations;

[0028] Pen stroke fluency analysis: Calculate the variance or standard deviation of the velocity sequence of the pen stroke trajectory to generate a fluency index and identify pauses or jitters in the pen stroke;

[0029] Content recognition and positioning: Perform OCR processing on video frames to identify written text content, identify individual strokes through image segmentation technology, and associate the results of force mutation detection, angle deviation detection, and pen smoothness analysis with the text or stroke segmentation results.

[0030] Furthermore, the preprocessing process of the multimodal data input module includes:

[0031] Sensor data synchronization: Based on the timeline, the spatial motion in the video frame is aligned with the pen trajectory, force, and angle data collected by the pressure-sensitive pen and gyroscope sensor. OCR is used to recognize the written text content, and image segmentation technology is used to separate the characters or strokes and associate them with the timeline.

[0032] Noise filtering: eliminate noise through Kalman filtering;

[0033] Data standardization: Pressure value normalization: P is the original pressure sensor acquisition value; P min is the minimum range of the pressure sensor; P max is the maximum range of the pressure sensor; P′ is the normalized pressure value, ranging from [0,1]; Angle data conversion: angle to radian formula: θ deg is the angle value collected by the gyroscope, θ rad is the converted radian value.

[0034] Furthermore, the personalized learning module specifically includes

[0035] Evaluation and Analysis Module: Based on the calligraphy practice evaluation results, the types and locations of errors made by the writer are identified and prioritized. Errors are prioritized based on their severity and impact on the overall aesthetics of the calligraphy.

[0036] Video library association module: Based on the error type, it retrieves video clips that provide relevant teaching techniques and demonstrations from the teaching video library, and retrieves teaching videos of specific characters or strokes based on the error location;

[0037] Copybook library association module: retrieve related copybooks based on error location;

[0038] Practice customization module: Based on the results of the evaluation analysis module, the video library association module and the copybook library association module, teaching videos and copybooks are generated in descending order of the severity of writing errors for writers to practice and correct.

[0039] Furthermore, the system also includes a writing posture detection module, which determines whether the sitting posture and pen-holding posture of the writer in the video need to be corrected by analyzing the sitting posture and pen-holding posture.

[0040] Furthermore, the system also includes a teacher management module for connecting to the teacher side for calligraphy teachers to provide real-time online teaching.

[0041] A calligraphy teaching method comprises the following steps:

[0042] S1. Collect the writer's writing video, pen trajectory, force and angle data;

[0043] S2. A calligraphy evaluation model based on an attention mechanism extracts and fuses features from the data to generate a calligraphy practice evaluation. The evaluation metrics for calligraphy practice include the force stability, angle accuracy, and brush movement fluency of the written characters or strokes.

[0044] S3. Determine whether you have passed the written exercise based on the evaluation;

[0045] S4. If the result is not passed, the calligraphy practice evaluation results are analyzed and the corresponding teaching videos and copybooks of the characters or strokes are associated. After watching the teaching videos, the writer performs feedback practice according to the associated copybook and returns to S1. If the result is passed, the calligraphy practice proceeds to the next stage.

[0046] Beneficial effects of the present invention:

[0047] 1. Improve the objectivity and accuracy of calligraphy evaluation: Through the calligraphy evaluation model based on the attention mechanism, the system can evaluate the quality of calligraphy practice more objectively and accurately, reducing the interference of subjective factors and improving the fairness and accuracy of the evaluation.

[0048] 2. Meet personalized learning needs: The system can intelligently recommend corresponding teaching videos and copybooks based on the writer's practice evaluation results, providing writers with personalized learning resources, meeting the personalized learning needs of different writers, and improving learning efficiency and learning effects.

[0049] 3. Promote the digitization and intelligence of calligraphy teaching: The system integrates modern technologies such as computer vision, sensor technology, deep learning, etc., which promotes the digitization and intelligence of calligraphy teaching, brings new possibilities and opportunities to calligraphy education, and improves the modernization level of calligraphy teaching.

[0050] 4. Improve the fun and interactivity of calligraphy teaching: By combining multimedia resources such as videos and copybooks, the system can improve the fun and interactivity of calligraphy teaching, making the learning process more vivid and interesting, stimulating learners' interest and motivation, and improving learners' participation and satisfaction.

[0051] 5. Improve teaching efficiency and quality: Through automated and intelligent evaluation methods, the system can reduce the workload of teachers and improve teaching efficiency. At the same time, the system can also provide personalized learning resources for calligraphers, helping them to improve their writing skills more quickly, thereby improving the overall teaching quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a system structure diagram of the present invention.

[0053] Figure 2 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0054] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Example 1: Reference Figure 1 The present invention provides a digital calligraphy teaching system, comprising:

[0056] Multimodal data input module: includes a binocular camera, a pressure-sensitive pen, and a gyroscope sensor. The binocular camera collects writing videos, while the pressure-sensitive pen and gyroscope sensor collect pen trajectory, force, and angle data, and pre-processes the data.

[0057] Intelligent analysis module: Builds a calligraphy evaluation model based on the attention mechanism, extracts and fuses features from the multimodal data input module, and obtains calligraphy practice evaluation;

[0058] Resource management module: including video library and copybook library. It stores various teaching videos and copybooks;

[0059] Personalized learning module: Analyzes calligraphy practice evaluation results, associates corresponding teaching videos and copybooks based on the analysis results, and provides personalized learning customization, including:

[0060] Evaluation and Analysis Module: Based on the calligraphy practice evaluation results, the error types and locations are identified and prioritized. Errors are prioritized based on their severity and impact on the overall aesthetics of the calligraphy. Error types include: insufficient force, angle deviation, unsmooth brushwork, missing or extra strokes, etc. Error location: The specific character or stroke where the error occurred is determined, as well as the specific location within the character or stroke (e.g., starting stroke, turning stroke, ending stroke).

[0061] Video library association module: Based on the error type, it retrieves video clips that provide relevant teaching techniques and demonstrations from the teaching video library, and retrieves teaching videos of specific characters or strokes based on the error location;

[0062] Copybook library association module: retrieve related copybooks based on error location;

[0063] Practice customization module: Based on the results of the evaluation analysis module, the video library association module and the copybook library association module, teaching videos and copybooks are generated in descending order of the severity of writing errors for writers to practice and correct.

[0064] In this embodiment, the pressure-sensitive pen has a built-in pressure sensing module (0.1-1.2N accuracy) and a three-axis gyroscope to record the pen tip pressure changes, tilt angle (15°-75°) and rotation acceleration (0.3-2.5m / s) in real time. 2 ); The gyroscope sensor uses a nine-axis inertial sensor, attached to the wrist, to capture the acceleration, angular velocity, and posture angle data of the pen trajectory, generating a 9-dimensional time-series pen movement sequence; the binocular camera system: dynamically captures writing postures and combines the OpenPose algorithm to identify incorrect postures such as hooked wrists and shrugging shoulders.

[0065] In this embodiment, the calligraphy evaluation model includes

[0066] Feature extraction module: includes visual feature extraction branch and temporal feature extraction branch;

[0067] The visual feature extraction branch uses a 3D-CNN neural network to extract the spatial-temporal features of the pen tip motion and wrist posture in the writing video;

[0068] The temporal feature extraction branch processes the sensor sequence through the LSTM encoder to extract the dynamic features of the pen trajectory, force and angle during the pen movement process;

[0069] Attention fusion module: including cross-modal attention fusion, spatial attention fusion and temporal attention fusion;

[0070] Cross-modal attention fusion: Use scaled dot product attention to calculate the similarity matrix between visual features and temporal features, generate a cross-modal context vector, fuse visual and sensor information, and thus learn inter-modal associations (such as the association between "wrist rotation angle" and "stroke inclination").

[0071] Spatial attention fusion: Focuses on key writing actions in the video (such as starting and turning strokes), suppresses background interference, inputs a video feature map, generates a spatial attention weight map through a convolutional layer, and outputs a weighted feature map to highlight the key writing areas in the video.

[0072] Temporal Attention Fusion: This method captures the dynamic changes in force and angle during the pen movement (e.g., the rhythm of starting, moving, and ending the stroke). It uses the sensor time series (pressure, angle) as input and weights the feature representation using temporal attention weights to highlight key time steps or dynamic patterns.

[0073] Evaluation module: The features after attention fusion are input into the fully connected layer to perform a comprehensive evaluation of the calligraphy quality.

[0074] Among them, when training the calligraphy evaluation model, the loss function optimizes the associated regression task and classification task simultaneously through multi-task learning: L total =λL MSE +(1-λ)L CE ,

[0075] Among them, L total is the joint loss function; λ is a hyperparameter used to control the weights of the two tasks; L MSE is the mean square error of the regression task; L CE is the cross entropy loss for classification tasks.

[0076] Among them, the comprehensive assessment of calligraphy quality includes:

[0077] Force mutation detection: Use the temporal attention mechanism to identify mutation points in the pressure sequence and calculate the mutation frequency as an indicator of force stability.

[0078] Angle deviation detection: Compare the difference between the actual angle and the standard writing angle, generate an angle deviation heat map, and highlight the areas with large deviations;

[0079] Pen stroke fluency analysis: Calculate the variance or standard deviation of the velocity sequence of the pen stroke trajectory to generate a fluency index and identify pauses or jitters in the pen stroke;

[0080] Content recognition and positioning: Perform OCR processing on video frames to identify written text content, identify individual strokes through image segmentation technology, and associate the results of force mutation detection, angle deviation detection, and pen smoothness analysis with the text or stroke segmentation results.

[0081] In this embodiment, the preprocessing process of the multimodal data input module includes:

[0082] Sensor data synchronization: Based on the timeline, the spatial motion in the video frame is aligned with the pen trajectory, force, and angle data collected by the pressure-sensitive pen and gyroscope sensor. OCR is used to recognize the written text content, and image segmentation technology is used to separate the characters or strokes and associate them with the timeline.

[0083] Noise filtering: eliminate noise through Kalman filtering;

[0084] Data standardization: Pressure value normalization: P is the original pressure sensor acquisition value; P min is the minimum range of the pressure sensor; P max is the maximum range of the pressure sensor; P′ is the normalized pressure value, ranging from [0,1]; Angle data conversion: angle to radian formula: θ deg is the angle value collected by the gyroscope, θ rad is the converted radian value.

[0085] In another preferred embodiment, the system further includes a writing posture detection module: used to detect the writer's sitting posture and pen-holding posture, and determine whether the sitting posture and pen-holding posture are correct.

[0086] In another preferred embodiment, the system also includes a teacher management module for connecting to the teacher terminal for calligraphy teachers to provide real-time online teaching.

[0087] In another preferred embodiment, Figure 2 As shown, a calligraphy teaching method includes the following steps:

[0088] S1. Collect the writer's writing video, pen trajectory, force and angle data;

[0089] S2. A calligraphy evaluation model based on an attention mechanism extracts and fuses features from the data to generate a calligraphy practice evaluation. The evaluation metrics for calligraphy practice include the force stability, angle accuracy, and brush movement fluency of the written characters or strokes.

[0090] S3. Determine whether you have passed the written exercise based on the evaluation;

[0091] S4. If the result is not passed, the calligraphy practice evaluation results are analyzed and the corresponding teaching videos and copybooks of the characters or strokes are associated. After watching the teaching videos, the writer performs feedback practice according to the associated copybook and returns to S1. If the result is passed, the calligraphy practice proceeds to the next stage.

[0092] Through the fully closed-loop teaching process of "data collection-intelligent analysis-real-time feedback adjustment" in this program, precise teaching of "millimeter-level movement correction" can be achieved.

[0093] In the description of the embodiments of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "center", "top", "bottom", "top", "bottom", "inside", "outside", "inner side", "outer side" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. Among them, "inside" refers to an internal or enclosed area or space. "Periphery" refers to the area surrounding a specific component or specific area.

[0094] In the description of the embodiments of the present invention, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly specifying the number of the technical features indicated. Therefore, a feature specified as "first," "second," "third," or "fourth" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0095] In the description of the embodiments of the present invention, specific features, structures, materials or characteristics may be combined in an appropriate manner in any one or more embodiments or examples.

[0096] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to these embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the appended claims and their equivalents.

Claims

1. A digital calligraphy teaching system, characterized by: include Multimodal data input module: includes a binocular camera, a pressure-sensitive pen, and a gyroscope sensor. The binocular camera collects writing videos, while the pressure-sensitive pen and gyroscope sensor collect pen trajectory, force, and angle data, and pre-processes the data. Intelligent analysis module: Builds a calligraphy evaluation model based on the attention mechanism, extracts and fuses features from the multimodal data input module, and obtains calligraphy practice evaluation; Resource management module: including video library and copybook library, which stores various teaching videos and copybooks; Personalized learning module: Analyze the evaluation results of calligraphy practice, associate the corresponding teaching decomposition videos and copybooks based on the analysis results, and customize personalized learning.

2. A digital calligraphy teaching system according to claim 1, characterized in that: The calligraphy evaluation model includes Feature extraction module: includes visual feature extraction branch and temporal feature extraction branch; The visual feature extraction branch uses a 3D-CNN neural network to extract the spatial-temporal features of the pen tip motion and wrist posture in the writing video; The temporal feature extraction branch processes the sensor sequence through the LSTM encoder to extract the dynamic features of the pen trajectory, force and angle during the pen movement process; Attention fusion module: including cross-modal attention fusion, spatial attention fusion and temporal attention fusion; The cross-modal attention fusion uses scaled dot product attention to calculate the similarity matrix between visual features and temporal features, generates a cross-modal context vector, fuses visual and sensor information, and thus learns the association between modalities; The spatial attention fusion generates a spatial attention weight map through a convolutional layer. The weight value reflects the importance of each pixel and highlights the key writing areas in the video. The temporal attention fusion uses temporal attention weights to weight feature representations to highlight key time steps or dynamic patterns, capturing the dynamic changes in force and angle during the pen movement process. Evaluation module: The features after attention fusion are input into the fully connected layer to perform a comprehensive evaluation of the calligraphy quality.

3. A digital calligraphy teaching system according to claim 2, characterized in that: When training the calligraphy evaluation model, the loss function optimizes the associated regression and classification tasks simultaneously through multi-task learning: L total =λL MSE +(1-λ)L CE , Among them, L total is the joint loss function; λ is a hyperparameter used to control the weights of the two tasks; L MSE is the mean square error of the regression task; L CE is the cross entropy loss for classification tasks.

4. A digital calligraphy teaching system according to claim 2, characterized in that: The comprehensive assessment of calligraphy quality includes: Force mutation detection: Use the temporal attention mechanism to identify mutation points in the pressure sequence and calculate the mutation frequency as an indicator of force stability. Angle deviation detection: Compare the difference between the actual angle and the standard writing angle, generate an angle deviation heat map, and highlight the areas with large deviations; Pen stroke fluency analysis: Calculate the variance or standard deviation of the velocity sequence of the pen stroke trajectory to generate a fluency index and identify pauses or jitters in the pen stroke; Content recognition and positioning: Perform OCR processing on video frames to identify written text content, identify individual strokes through image segmentation technology, and associate the results of force mutation detection, angle deviation detection, and pen smoothness analysis with the text or stroke segmentation results.

5. A digital calligraphy teaching system according to claim 1, characterized in that: The preprocessing process of the multimodal data input module includes: Sensor data synchronization: Based on the timeline, the spatial motion in the video frame is aligned with the pen trajectory, force, and angle data collected by the pressure-sensitive pen and gyroscope sensor. OCR is used to recognize the written text content, and image segmentation technology is used to separate the characters or strokes and associate them with the timeline. Noise filtering: eliminate noise through Kalman filtering; Data standardization: Pressure value normalization: P is the original pressure sensor acquisition value; P min is the minimum range of the pressure sensor; P max is the maximum range of the pressure sensor; P′ is the normalized pressure value, ranging from [0,1]; Angle data conversion: angle to radian formula: θ deg is the angle value collected by the gyroscope, θ rad is the converted radian value.

6. A digital calligraphy teaching system according to claim 1, characterized in that: The personalized learning module specifically includes Evaluation and Analysis Module: Based on the calligraphy practice evaluation results, the types and locations of errors made by the writer are identified and prioritized. Errors are prioritized based on their severity and impact on the overall aesthetics of the calligraphy. Video library association module: Based on the error type, it retrieves video clips that provide relevant teaching techniques and demonstrations from the teaching video library, and retrieves teaching videos of specific characters or strokes based on the error location; Copybook library association module: retrieve related copybooks based on error location; Practice customization module: Based on the results of the evaluation analysis module, the video library association module and the copybook library association module, teaching videos and copybooks are generated in descending order of the severity of writing errors for writers to practice and correct.

7. A digital calligraphy teaching system according to claim 1, characterized in that: The system also includes a writing posture detection module, which determines whether the sitting posture and pen-holding posture of the writer in the video need to be corrected by analyzing the sitting posture and pen-holding posture.

8. A digital calligraphy teaching system according to claim 1, characterized in that: The system also includes a teacher management module, which is used to connect to the teacher side for calligraphy teachers to provide real-time online teaching.

9. A calligraphy teaching method, using the system according to any one of claims 1 to 8, characterized in that: Including steps: S1. Collect the writer's writing video, pen trajectory, force and angle data; S2. A calligraphy evaluation model based on an attention mechanism extracts and fuses features from the data to generate a calligraphy practice evaluation. The evaluation metrics for calligraphy practice include the force stability, angle accuracy, and brush movement fluency of the written characters or strokes. S3. Determine whether you have passed the written exercise based on the evaluation; If the result is not passed, the calligraphy practice evaluation results are analyzed and the corresponding teaching videos and calligraphy copybooks are associated. After watching the teaching videos, the calligrapher practices according to the associated calligraphy copybooks and returns to S1. If you pass, you will proceed to the next stage of calligraphy practice.

Images