Intelligent calligraphy copying system based on multi-modal fusion and AI assistance

Abstract

The invention discloses an intelligent calligraphy copying system based on multi-modal fusion and AI assistance, and the system comprises a modal sensing module which is used for capturing the stroke track, the pressure feature and the pen wielding audio rhythm of a learner during writing in real time, and converting the stroke track, the pressure feature and the pen wielding audio rhythm into multi-modal information; the analysis module is used for fusing the multi-modal information by using an AI deep learning algorithm, comparing the writing of the learner with the mass calligraphy famous works data based on the multi-modal fusion information, and judging whether the correctness of the strokes of the learner and the writing rhythm are reasonable or not in real time; the feedback module is used for providing instant feedback for the learner based on an analysis result of the analysis module; and the environment simulation module is used for creating a vivid calligraphy imitation scene through a virtual reality (VR) or augmented reality (AR) technology. According to the invention, a mode of combining multi-modal fusion and the AI technology is adopted, so that the efficiency and quality of calligraphy copying are greatly improved, learners are helped to master calligraphy skills more quickly, and the inheritance and development of calligraphy culture are promoted.

Description

Technical Field

[0001] The present invention belongs to the field of cultural education and intelligent technology, and in particular relates to an intelligent calligraphy copying system based on multimodal fusion and AI assistance. Background Art

[0002] Traditional calligraphy copying relies primarily on copybooks and human guidance. Learners manually copy the characters from the copybook, imitating the stroke structure and order of the calligrapher's strokes. While this traditional method can help learners understand character structure to a certain extent, it has many limitations. First, it makes it difficult for learners to accurately grasp subtle changes in stroke force, speed, and angle, which are crucial to calligraphy expression. Traditional copybooks cannot fully demonstrate these dynamic details. Relying solely on static character shapes and calligrapher demonstrations, it is difficult to fully master calligraphy techniques.

[0003] Although human guidance can make up for some shortcomings, due to the time and space constraints between teachers and students, learners often cannot get timely feedback and efficient guidance, resulting in poor learning effects.

[0004] With technological advancements, electronic calligraphy imitation tools have gradually emerged. These tools simulate the effect of writing on paper through devices such as digital tablets and touch screens. However, most existing electronic calligraphy imitation tools suffer from limited functionality and poor simulation quality. While some electronic tools can achieve basic character shape rendering, they cannot fully simulate the real writing environment, such as the pressure sensing, speed changes, and pen tip turning dynamic factors of the strokes. As a result, they are unable to meet the needs of calligraphy learners for a high-quality, intelligent calligraphy experience.

[0005] Currently, some technologies based on artificial intelligence (AI) and deep learning have begun to be applied to the field of calligraphy teaching, but there are still many shortcomings. Traditional AI-assisted calligraphy tools mainly focus on glyph recognition and stroke order prompts, and lack control over details in the calligraphy process, such as the strength, speed, and tilt angle of the brushstrokes. More importantly, existing technologies are not yet able to provide personalized copying guidance based on the characteristics of each learner, and cannot achieve intelligent feedback and customized learning plans. Therefore, there is an urgent need for a new technical solution that can effectively combine AI technology, multimodal fusion, and intelligent interaction to provide a comprehensive and personalized calligraphy copying experience.

[0006] To sum up, the existing calligraphy copying tools cannot meet the learners' needs for personalized, intelligent, and realistic simulation of the calligraphy copying process. There is an urgent need for an intelligent calligraphy copying system based on multimodal fusion and AI assistance to better adapt to the needs of different learners and improve the efficiency and quality of calligraphy learning. Summary of the Invention

[0007] The present invention aims to solve the problems in the prior art and proposes the following technical solutions:

[0008] The intelligent calligraphy copying system based on multimodal fusion and AI assistance includes:

[0009] Multimodal perception module: This module captures the learner's writing stroke trajectory, pressure characteristics, and pen movement audio rhythm in real time and converts them into multimodal information;

[0010] Analysis module: Utilizes AI deep learning algorithms to fuse multimodal information. Based on this multimodal fusion information, the learner's handwriting is compared with a large amount of data on the works of famous calligraphers to determine in real time the correctness of the learner's strokes and whether the writing rhythm is reasonable.

[0011] Feedback module: provides learners with immediate feedback based on the analysis results of the analysis module;

[0012] Environmental simulation module: Create a realistic calligraphy copying scene through virtual reality (VR) or augmented reality (AR) technology.

[0013] As a preferred embodiment of the above technical solution, the devices configured in the multimodal perception module include a high-resolution camera, a pressure sensor and a microphone;

[0014] The high-resolution camera captures the learner's hand movements and stroke trajectories during writing in real time; the pressure sensor accurately senses the changes in force during the writing process and obtains pressure characteristics; the microphone collects the sound of the pen during writing and grasps the audio rhythm of the pen movement.

[0015] As a preferred embodiment of the above technical solution, the AI ​​deep learning algorithm includes multi-module information fusion, multi-modal feature fusion evaluation and dynamic judgment.

[0016] As a preferred embodiment of the above technical solution, the multi-module information fusion includes:

[0017] Visual trajectory sequence modeling: Let the standard calligraphy stroke trajectory be V std ={(X t ,Y t )|t=1,2,...,T}, the user trajectory is V use ={(X' t ,Y' t )}, calculate the trajectory deviation after timing alignment through dynamic time warping (DTW):

[0018]

[0019] Among them, φ(t) represents the optimal alignment path solved by the DTW algorithm;

[0020] Pressure feature quantification: extracting the mean value μ of the pressure waveform p and standard deviation σ p , calculate the intensity feature similarity:

[0021]

[0022] Among them, μ std is the standard calligraphy pressure waveform, μ usr is the user pressure waveform, σ std is the standard deviation of standard calligraphy, σ usr is the user standard deviation;

[0023] Audio rhythm analysis: Extract frequency domain features through short-time Fourier transform and define rhythm synchronization:

[0024]

[0025] Among them, F std (t,f) is the frequency domain feature extracted by the standard calligraphy short-time Fourier transform, F usr (t,f) is the frequency domain feature extracted by user short-time Fourier transform.

[0026] As a preferred embodiment of the above technical solution, the multimodal feature fusion evaluation includes:

[0027] Composite evaluation function:

[0028] E=α·e -βΔV +γ·S p +δ·R

[0029] Among them, α+γ+δ=1 is the modal weight coefficient, which is dynamically adjusted through the attention mechanism:

[0030] α=softmax(W a [ΔV,S p ,R] T )

[0031] Among them, the weight matrix W a It is a massive amount of data on works by famous calligraphers.

[0032] As a preferred embodiment of the above technical solution, the dynamic determination includes:

[0033] Dynamic threshold generation:

[0034]

[0035] Among them, θ0 is the basic threshold, k is the sensitivity coefficient, E avg is the historical evaluation mean;

[0036] Decision logic:

[0037]

[0038] Among them, Δ is the buffer interval to avoid frequent state switching.

[0039] As a preferred embodiment of the above technical solution, the feedback module includes vibration reminders and screen prompts.

[0040] The beneficial effects of the present invention are:

[0041] The system of the present invention uses multimodal fusion technology, combines multiple sensory inputs such as video, audio, and images, and cooperates with the deep learning and real-time feedback functions of AI technology, so that it can accurately analyze the learner's writing process and automatically detect and judge key factors such as the strength, speed, and angle of the strokes; learners can not only view the character shape through a graphical interface, but also understand the details such as the rise, development, transition, and change of the strokes through real-time dynamic demonstration, as if interacting with a calligraphy master in person. This fusion method makes the learning process more vivid and intuitive, and enhances the learner's perception and understanding; it enables learners to quickly correct errors in writing and quickly master calligraphy skills, reducing the time wasted in repeated revisions and blind imitation in traditional methods, thereby greatly improving learning efficiency and quality; the present invention provides innovative solutions for the digitization and intelligence of calligraphy education by integrating modern AI technology with traditional calligraphy art, greatly expanding the audience group of calligraphy education, and providing new impetus for the inheritance and development of calligraphy culture. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments.

[0043] Example

[0044] The intelligent calligraphy copying system based on multimodal fusion and AI assistance includes:

[0045] Multimodal perception module: captures the learner's stroke trajectory, pressure characteristics and pen audio rhythm when writing in real time, and converts them into multimodal information; the system configuration equipment in the multimodal perception module includes a high-resolution camera, a pressure sensor and a microphone; the high-resolution camera captures the learner's hand movements and stroke trajectory when writing in real time; the pressure sensor accurately senses the changes in force during the writing process and obtains the pressure characteristics; the microphone collects the pen movement sound during writing and grasps the pen movement audio rhythm.

[0046] Analysis module: Utilizes AI deep learning algorithms to fuse multimodal information. Based on this multimodal fusion information, the learner's handwriting is compared with a large amount of data on the works of famous calligraphers to determine in real time the correctness of the learner's strokes and whether the writing rhythm is reasonable.

[0047] The AI ​​deep learning algorithm includes multi-module information fusion, multi-modal feature fusion evaluation and dynamic judgment.

[0048] The multi-module information fusion includes:

[0049] Visual trajectory sequence modeling: Assume that the standard calligraphy stroke trajectory is

[0050] V std ={(X t ,Y t )|t=1,2,...,T}, the user trajectory is V use ={(X' t ,Y' t )}, calculate the trajectory deviation after timing alignment through dynamic time warping (DTW):

[0051]

[0052] Among them, φ(t) represents the optimal alignment path solved by the DTW algorithm;

[0053] Pressure feature quantification: extracting the mean value μ of the pressure waveform p and standard deviation σ p , calculate the intensity feature similarity:

[0054]

[0055] Among them, μ std is the standard calligraphy pressure waveform, μ usr is the user pressure waveform, σ std is the standard deviation of standard calligraphy, σ usr is the user standard deviation;

[0056] Audio rhythm analysis: Extract frequency domain features through short-time Fourier transform and define rhythm synchronization:

[0057]

[0058] Among them, F std (t,f) is the frequency domain feature extracted by the standard calligraphy short-time Fourier transform, F usr (t,f) is the frequency domain feature extracted by user short-time Fourier transform.

[0059] The multimodal feature fusion evaluation includes:

[0060] Composite evaluation function:

[0061] E=α·e -βΔV +γ·S p +δ·R

[0062] Among them, α+γ+δ=1 is the modal weight coefficient, which is dynamically adjusted through the attention mechanism:

[0063] α=softmax(W a [ΔV,S p ,R] T )

[0064] Among them, the weight matrix W a It is a massive amount of data on works by famous calligraphers.

[0065] The dynamic determination includes:

[0066] Dynamic threshold generation:

[0067]

[0068] Among them, θ0 is the basic threshold, k is the sensitivity coefficient, E avg is the historical evaluation mean;

[0069] Decision logic:

[0070]

[0071] Among them, Δ is the buffer interval to avoid frequent state switching.

[0072] Feedback module: Provides learners with instant feedback based on the analysis results of the analysis module; the feedback module includes vibration reminders and screen prompts.

[0073] When E≥θ+Δ, in excellent condition, there is no feedback; when θ≤E<θ+Δ, in qualified condition, there is a slight vibration (which does not affect normal writing) to remind the writer; when E<θ, in need of improvement condition, a corresponding screen prompt will appear. The writer can not only view the shape of his or her handwritten characters through the screen interface, but also understand the details of the strokes, changes in pen tips, etc. through real-time dynamic demonstration of the control screen, so as to make timely corrections.

[0074] Environmental Simulation Module: Using virtual reality (VR) or augmented reality (AR) technology, a realistic calligraphy copying environment is created. Learners feel as if they are in an ancient study, facing authentic calligraphy copybooks, brushes, inkstones, paper, and inkstones. This enhances the immersive and engaging nature of calligraphy copying, improving learning outcomes.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same.

Claims

1. An intelligent calligraphy copying system based on multimodal fusion and AI assistance, characterized by: include: Multimodal perception module: This module captures the learner's writing stroke trajectory, pressure characteristics, and pen movement audio rhythm in real time and converts them into multimodal information; Analysis module: Utilizes AI deep learning algorithms to fuse multimodal information. Based on this multimodal fusion information, the learner's handwriting is compared with a large amount of data on the works of famous calligraphers to determine in real time the correctness of the learner's strokes and whether the writing rhythm is reasonable. Feedback module: provides learners with immediate feedback based on the analysis results of the analysis module; Environmental simulation module: Create a realistic calligraphy copying scene through virtual reality (VR) or augmented reality (AR) technology.

2. The intelligent calligraphy copying system based on multimodal fusion and AI assistance according to claim 1 is characterized in that: The equipment configured in the multimodal perception module system includes a high-resolution camera, a pressure sensor and a microphone; The high-resolution camera captures the learner's hand movements and stroke trajectories in real time while writing; the pressure sensor accurately senses the force changes during the writing process and obtains the pressure characteristics; The microphone collects the sound of the pen when writing and grasps the audio rhythm of the pen.

3. The intelligent calligraphy copying system based on multimodal fusion and AI assistance according to claim 1 is characterized in that: The AI ​​deep learning algorithm includes multi-module information fusion, multi-modal feature fusion evaluation and dynamic judgment.

4. The intelligent calligraphy copying system based on multimodal fusion and AI assistance according to claim 3 is characterized in that: The multi-module information fusion includes: Visual trajectory sequence modeling: Let the standard calligraphy stroke trajectory be V std ={(X t ,Y t )|t=1,2,...,T}, the user trajectory is V use ={(X' t ,Y' t )}, calculate the trajectory deviation after timing alignment through dynamic time warping (DTW): Among them, φ(t) represents the optimal alignment path solved by the DTW algorithm; Pressure feature quantification: extracting the mean value μ of the pressure waveform p and standard deviation σ p , calculate the intensity feature similarity: Among them, μ std is the standard calligraphy pressure waveform, μ usr is the user pressure waveform, σ std is the standard deviation of standard calligraphy, σ usr is the user standard deviation; Audio rhythm analysis: Extract frequency domain features through short-time Fourier transform and define rhythm synchronization: Among them, F std (t,f) is the frequency domain feature extracted by the standard calligraphy short-time Fourier transform, F usr (t,f) is the frequency domain feature extracted by user short-time Fourier transform.

5. The intelligent calligraphy copying system based on multimodal fusion and AI assistance according to claim 4 is characterized in that: The multimodal feature fusion evaluation includes: Composite evaluation function: E=α·e -βΔV +γ·S p +δ·R Among them, α+γ+δ=1 is the modal weight coefficient, which is dynamically adjusted through the attention mechanism: α=softmax(W a [ΔV,S p ,R] T ) Among them, the weight matrix W a It is a massive amount of data on works by famous calligraphers.

6. The intelligent calligraphy copying system based on multimodal fusion and AI assistance according to claim 5 is characterized in that: The dynamic determination includes: Dynamic threshold generation: Among them, θ0 is the basic threshold, k is the sensitivity coefficient, E avg is the historical evaluation mean; Decision logic: Among them, Δ is the buffer interval to avoid frequent state switching.

7. The intelligent calligraphy copying system based on multimodal fusion and AI assistance according to claim 1 is characterized in that: The feedback module includes vibration reminders and screen prompts.

Images