Handwriting real-time restoration method integrating deep learning and filtering
The integration of deep learning and filtering algorithms in a real-time handwriting restoration method addresses delays in digital writing by using DGAT and GRU for accurate trajectory prediction and smoothing, enhancing user experience and accuracy.
Patent Information
- Application Number
- CN202411272895.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-12
- Publication Date
- 2024-12-03
AI Technical Summary
Existing digital writing technologies suffer from delays in touch sampling point transmission, leading to reduced continuity and experience in handwriting input due to hardware limitations, especially in mobile smart devices and touch devices.
A real-time handwriting restoration method integrating deep learning and filtering, utilizing a directed graph neural network (DGAT) for data encoding and a double-layer GRU for prediction, combined with least squares filtering for post-processing to ensure smooth transitions and accurate trajectory prediction.
Enhances user experience by improving handwriting prediction accuracy and real-time performance across diverse writing speeds and styles, providing a natural and personalized electronic writing experience.
Smart Images

Figure CN119068496A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to multiple fields such as trajectory prediction algorithms, filtering algorithms, data processing and analysis based on deep learning, and aims to provide a real-time handwriting restoration method integrating deep learning and filtering. Background Art
[0002] Nowadays, digital writing technology has become an indispensable part of life, work and learning scenarios, especially in mobile smart devices, tablets, smart conferences, teaching whiteboards and other touch devices. However, due to the differences and limitations of hardware performance, there is a large delay in the transmission and drawing of touch sampling points, that is, there is a certain distance between the real-time displayed trajectory point and the finger touch position. This limitation reduces the continuity and experience of handwriting input. Therefore, the development of an innovative technical solution that aims to predict the writing path in real time and accurately, while being compatible with the unique writing habits of each user, has become the focus of current research. This technological breakthrough will not only further promote the natural experience of electronic writing, but will also have a far-reaching impact on many fields such as smart education, remote collaboration, and personalized digital art creation. Summary of the invention
[0003] The purpose of the present invention is to provide a real-time handwriting restoration method integrating deep learning and filtering in order to solve the above-mentioned problems.
[0004] The technical solution adopted by the present invention is as follows: A real-time handwriting restoration method integrating deep learning and filtering comprises the following steps:
[0005] S1: Acquire the real writing point reporting data generated by real-time writing on the touch screen, and perform data normalization on the acquired raw data;
[0006] S2: The normalized data sequence {L i (x i ,y i ,t i ),i=1,2,3,...,4} is sent to the deep learning trajectory prediction model composed of the information interaction encoding module and the prediction decoding module to perform primary trajectory prediction;
[0007] S3: To solve the overfitting problem of the deep learning model during the writing process, the least squares filtering algorithm is used to post-process the trajectory points predicted by the deep learning model to ensure a smooth transition between the real strokes and the predicted strokes.
[0008] In a preferred embodiment, the writing point reporting data of S1, the point reporting data used in the present invention includes screen point coordinates (x, y) and corresponding point timestamp t.
[0009] In a preferred embodiment, the S2 information interaction encoding module is characterized in that the present invention takes into account the differences in effects between different modal data, uses a directed graph to represent the relationship between trajectory historical data at the same time, and uses a graph neural network DGAT to model and extract relational features between data.
[0010] In a preferred embodiment, the DGAT model calculation method, DGAT weights and aggregates the features of neighboring nodes based on the self-attention mechanism, and is used to update the node itself. The input of DGAT is historical trajectory data, and the output is an encoded feature sequence. The attention coefficient between node pairs in the graph neural network is:
[0011]
[0012] In the formula, α ij is the attention coefficient of node i to node j; N i is the set of neighbor nodes of node i in the graph neural network; e is the scoring function used to calculate the initial score of each edge; the softmax function is used to normalize the initial attention coefficients of all neighbor nodes of node i.
[0013] In a preferred embodiment, the S2 prediction decoding module uses the output of the information interaction encoding module of claim 3 as the model input, and the module uses a double-layer GRU as the backbone network to extract deep features and output the predicted trajectory coordinate point {(x′ i ,y′ i ),i=1,2,3,...,6}.
[0014] In a preferred embodiment, the S2 deep learning model training data set is constructed by collecting data from multiple people writing Chinese character strokes, Arabic numerals, English characters and regular shapes, and the data set is divided into a training set and a test set in a ratio of 8:2.
[0015] In a preferred embodiment, the process of post-processing the trajectory points predicted by the deep learning model in S3 is as follows:
[0016] S31: In the normal writing process, based on the point coordinates {(x i ,y i ),i=1,2,3,...,4} and the corresponding timestamp {t i , i=1,2,3,...,4}, and construct t i and x i ,t i and iThe n-order fitting polynomial of , and experimental research found that when n = 4, the handwriting prediction accuracy is the best, the two polynomials can be expressed as follows:
[0017] x(t)=f(a,t)=a0+a1t+a2t 2 +a3t 3 +a4t 4
[0018] y(t)=f(a,t)=a0+a1t+a2t 2 +a3t 3 +a4t 4
[0019] S32: In order to obtain the optimal solution of the coefficients, the present invention selects the parameter a so that the residual of the fitting model and the actual observation value at each point is and The weighted sum of squares is minimized, and the least squares criterion is used to find the extreme value of the multivariate function, and then the coefficient matrix [a0, a1, a2, a3, a4] corresponding to x and y is obtained, and the corresponding fitting equation is obtained;
[0020] S33: Combine deep learning to predict trajectory points {(x′ i ,y′ i ), i=1,2,3,...,6}, the horizontal coordinate sequence {x′ i , i=1,2,3,...,6} and the 4th-order fitting polynomial x(t) are used to calculate the time series {t′ corresponding to the prediction point i , i=1,2,3,...,6}, input the obtained time series into y(t) in turn to calculate the corresponding ordinate sequence {y″ i ,i=1,2,3,...,6};
[0021] S34: {y″ i ,i=1,2,3,...,6} and {y′ i , i=1,2,3,...,6} are subtracted in sequence, that is, {y″ i -y′ i ,i=1,2,3,...,6}, and keep the difference less than or equal to the threshold d w =3, and together with the corresponding x', form the coordinate points of the final handwriting prediction output for display on the screen.
[0022] In a preferred embodiment, a computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any one of the methods for real-time handwriting restoration integrating deep learning and filtering is implemented.
[0023] In a preferred embodiment, an electronic whiteboard writing device includes a memory and a processor, wherein the memory is pre-installed with a specially designed computer program. The program, driven by the processor, implements any one of the methods for real-time handwriting restoration integrating deep learning and filtering.
[0024] In summary, due to the adoption of the above technical solution, the beneficial effects of the present invention are:
[0025] In the present invention, an innovative lightweight handwriting trajectory prediction method integrating deep learning and filtering algorithms is proposed, aiming to improve the user experience of electronic writing in terms of natural fluency and personalized expression. The core of the present invention is to construct a comprehensive method system, covering the whole process from the acquisition of original trajectory data, normalization, prediction of deep learning model, to post-processing optimization, effectively solving the problem of handwriting prediction accuracy and real-time performance under diverse writing speeds, styles and complex backgrounds. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 Schematic diagram of model prediction;
[0027] Figure 2 Model structure diagram;
[0028] Figure 3 Handwriting prediction logic block diagram. DETAILED DESCRIPTION
[0029] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0030] Reference Figure 1-3 ,
[0031] A real-time handwriting restoration method integrating deep learning and filtering comprises the following steps:
[0032] S1: Acquire the real writing point reporting data generated by real-time writing on the touch screen, and perform data normalization on the acquired raw data;
[0033] S2: The normalized data sequence {L i (x i ,y i ,t i ),i=1,2,3,...,4} is sent to the deep learning trajectory prediction model composed of the information interaction encoding module and the prediction decoding module to perform primary trajectory prediction;
[0034] S3: To solve the overfitting problem of the deep learning model during the writing process, the least squares filtering algorithm is used to post-process the trajectory points predicted by the deep learning model to ensure a smooth transition between the real strokes and the predicted strokes.
[0035] In a preferred embodiment, the writing point reporting data of S1, the point reporting data used in the present invention includes screen point coordinates (x, y) and corresponding point timestamp t.
[0036] The S2 information interaction encoding module is characterized in that the present invention takes into account the differences in effects between different modal data, uses a directed graph to represent the relationship between trajectory historical data at the same time, and uses a graph neural network DGAT to model and extract relational features between data.
[0037] The DGAT model calculation method, DGAT weights and aggregates the features of neighboring nodes based on the self-attention mechanism and uses it to update the node itself. The input of DGAT is historical trajectory data, and the output is an encoded feature sequence. The attention coefficient between node pairs in the graph neural network is:
[0038]
[0039] In the formula, α ij is the attention coefficient of node i to node j; N i is the set of neighbor nodes of node i in the graph neural network; e is the scoring function used to calculate the initial score of each edge; the softmax function is used to normalize the initial attention coefficients of all neighbor nodes of node i.
[0040] The S2 prediction decoding module uses the output of the information interaction encoding module as the model input, and uses a double-layer GRU as the backbone network to extract deep features and output the predicted trajectory coordinate point {(x′ i ,y′ i ),i=1,2,3,...,6}.
[0041] The S2 deep learning model training data set is constructed by collecting data from multiple people writing Chinese character strokes, Arabic numerals, English characters and regular shapes, and the data set is divided into a training set and a test set in a ratio of 8:2.
[0042] The process of post-processing the trajectory points predicted by the deep learning model described in S3 is as follows:
[0043] S31: In the normal writing process, based on the point coordinates {(x i ,y i ),i=1,2,3,...,4} and the corresponding timestamp {t i, i=1,2,3,...,4}, and construct t i and x i ,t i and i The n-order fitting polynomial of , and experimental research found that when n = 4, the handwriting prediction accuracy is the best, the two polynomials can be expressed as follows:
[0044] x(t)=f(a,t)=a0+a1t+a2t 2 +a3t 3 +a4t 4
[0045] y(t)=f(a,t)=a0+a1t+a2t 2 +a3t 3 +a4t 4
[0046] S32: In order to obtain the optimal solution of the coefficients, the present invention selects the parameter a so that the residual of the fitting model and the actual observation value at each point is and The weighted sum of squares is minimized, and the least squares criterion is used to find the extreme value of the multivariate function, and then the coefficient matrix [a0, a1, a2, a3, a4] corresponding to x and y is obtained, and the corresponding fitting equation is obtained;
[0047] S33: Combine deep learning to predict trajectory points {(x′ i ,y′ i ), i=1,2,3,...,6}, the horizontal coordinate sequence {x′ i , i=1,2,3,...,6} and the 4th-order fitting polynomial x(t) are used to calculate the time series {t′ corresponding to the prediction point i , i=1,2,3,...,6}, input the obtained time series into y(t) in turn to calculate the corresponding ordinate sequence {y″ i ,i=1,2,3,...,6};
[0048] S34: {y″ i ,i=1,2,3,...,6} and {y′ i , i=1,2,3,...,6} are subtracted in sequence, that is, {y″ i -y′ i ,i=1,2,3,...,6}, and keep the difference less than or equal to the threshold d w =3, and together with the corresponding x', form the coordinate points of the final handwriting prediction output for display on the screen.
[0049] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements any one of the methods for real-time handwriting restoration integrating deep learning and filtering.
[0050] An electronic whiteboard writing device comprises a memory and a processor, wherein the memory is pre-installed with a specially designed computer program. The program, driven by the processor, implements any one of the methods for real-time handwriting restoration integrating deep learning and filtering.
[0051] In the present invention, an innovative lightweight handwriting trajectory prediction method integrating deep learning and filtering algorithms is proposed, aiming to improve the user experience of electronic writing in terms of natural fluency and personalized expression. The core of the present invention is to construct a comprehensive method system, covering the whole process from the acquisition of original trajectory data, normalization, prediction of deep learning model, to post-processing optimization, effectively solving the problem of handwriting prediction accuracy and real-time performance under diverse writing speeds, styles and complex backgrounds.
[0052] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0053] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A real-time handwriting restoration method integrating deep learning and filtering, characterized in that: The steps include: S1: Acquire the real writing point reporting data generated by real-time writing on the touch screen, and perform data normalization on the acquired raw data; S2: The normalized data sequence {L i (x i ,y i ,t i ),i=1,2,3,...,4} is sent to the deep learning trajectory prediction model composed of the information interaction encoding module and the prediction decoding module to perform primary trajectory prediction; S3: To solve the overfitting problem of the deep learning model during the writing process, the least squares filtering algorithm is used to post-process the trajectory points predicted by the deep learning model to ensure a smooth transition between the real strokes and the predicted strokes.
2. The real-time handwriting restoration method integrating deep learning and filtering according to claim 1, characterized in that: The writing point reporting data of S1, the point reporting data used in the present invention includes screen point coordinates (x, y) and corresponding point timestamp t.
3. The real-time handwriting restoration method integrating deep learning and filtering according to claim 1, characterized in that: The S2 information interaction encoding module is characterized in that the present invention takes into account the differences in effects between different modal data, uses a directed graph to represent the relationship between trajectory historical data at the same time, and uses a graph neural network DGAT to model and extract relational features between data.
4. The real-time handwriting restoration method integrating deep learning and filtering according to claim 3 is characterized in that: The DGAT model calculation method, DGAT weights and aggregates the features of neighboring nodes based on the self-attention mechanism and uses it to update the node itself. The input of DGAT is historical trajectory data, and the output is an encoded feature sequence. The attention coefficient between node pairs in the graph neural network is: In the formula, αij is the attention coefficient of node i to node j; Ni is the set of neighbor nodes of node i in the graph neural network; e is the scoring function used to calculate the initial score of each edge; The softmax function is used to normalize the initial attention coefficients of all neighboring nodes of node i.
5. The real-time handwriting restoration method integrating deep learning and filtering according to claim 1, characterized in that: The S2 prediction decoding module uses the output of the information interaction encoding module as the model input, and uses a double-layer GRU as the backbone network to extract deep features and output the predicted trajectory coordinate point {(x i ',y i '),i=1,2,3,...,6}.
6. The real-time handwriting restoration method integrating deep learning and filtering according to claim 1, characterized in that: The S2 deep learning model training data set is constructed by collecting data from multiple people writing Chinese character strokes, Arabic numerals, English characters and regular shapes, and the data set is divided into a training set and a test set in a ratio of 8:
2.
7. The real-time handwriting restoration method integrating deep learning and filtering according to claim 1, characterized in that: The process of post-processing the trajectory points predicted by the deep learning model described in S3 is as follows: S31: In the normal writing process, based on the point coordinates {(x i ,y i ),i=1,2,3,...,4} and the corresponding timestamp {t i , i=1,2,3,...,4}, and construct t i and x i ,t i and i The n-order fitting polynomial of , and experimental research found that when n = 4, the handwriting prediction accuracy is the best, the two polynomials can be expressed as follows: x(t)=f(a,t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 y(t)=f(a,t)=a0+a1t+a2t 2 +a3t 3 +a4t 4 S32: In order to obtain the optimal solution of the coefficients, the present invention selects the parameter a so that the residual of the fitting model and the actual observation value at each point is and The weighted sum of squares is minimized, and the least squares criterion is used to find the extreme value of the multivariate function, and then the coefficient matrix [a0, a1, a2, a3, a4] corresponding to x and y is obtained, and the corresponding fitting equation is obtained; S33: Combine deep learning to predict trajectory points {(x i ',y i '),i=1,2,3,...,6}, the horizontal coordinate sequence {x i ', i = 1, 2, 3, ..., 6} and the 4th-order fitting polynomial x(t) to calculate the time series {t i ', i = 1, 2, 3, ..., 6}, input the obtained time series into y(t) in turn to calculate the corresponding ordinate sequence {y i ”, i = 1, 2, 3, ..., 6}; S34: {y i ”,i=1,2,3,...,6} and {y i ',i=1,2,3,...,6} are subtracted in sequence, that is, {y i ”-y i ', i = 1, 2, 3, ..., 6}, and retain the y' whose difference is less than or equal to the threshold value dw = 3, and form the coordinate point of the final handwriting prediction output with the corresponding x' for display on the screen.
8. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a processor, a real-time handwriting restoration method integrating deep learning and filtering as described in any one of claims 1-7 is implemented.
9. An electronic whiteboard writing device, characterized in that: It comprises a memory and a processor, wherein the memory is pre-installed with a specially designed computer program, which, driven by the processor, implements a real-time handwriting restoration method integrating deep learning and filtering as described in any one of claims 1 to 7.
Citation Information
Cited By
Method for positioning handwriting position based on computer touchpad
CN120045089A
