Handwriting real-time prediction method based on hybrid time sequence convolutional network

Abstract

The invention provides a handwriting real-time prediction method based on a hybrid time sequence convolutional network. The method comprises the following steps: acquiring real track data of a user in a touch screen writing process; preprocessing the real trajectory data to form a multi-dimensional time sequence feature sequence with a preset length; inputting the multi-dimensional time sequence feature sequence into a trained trajectory point prediction model constructed based on a hybrid time sequence convolutional network, and outputting predicted trajectory coordinates; on the basis of the real trajectory data and the predicted trajectory coordinates, trajectory smoothing processing is carried out, and predicted trajectory data used for drawing is generated; and drawing the predicted trajectory data on the touch screen in real time. According to the method, hardware delay is compensated from the algorithm level, writing chirality and real-time performance are improved, dependence on high-end hardware is reduced through end-to-end optimization, and low-cost and high-simulation-degree paper and pen writing experience is provided for touch equipment.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and in particular to a real-time handwriting prediction method based on hybrid temporal convolutional networks. Background Technology

[0002] With the popularization of digital office and education, touch devices have become an important tool for daily writing, communication and creation. However, existing real-time handwriting rendering technology still faces many challenges: on the one hand, due to hardware performance limitations, there is an inherent delay in the transmission and rendering of touch sampling points, resulting in a deviation between the displayed trajectory and the actual position of the finger, affecting the smoothness of writing; on the other hand, existing prediction models mostly rely on architectures such as GRU and Transformer, which have problems such as high computational complexity or insufficient generalization ability when capturing long-term temporal dependencies, especially in complex scenarios such as sudden changes in writing speed and stroke transitions, which are prone to prediction bias or overfitting.

[0003] Temporal Convolutional Networks (TCNs) in deep learning, with their causal dilated convolution properties, can capture long-range temporal dependencies while maintaining low computational complexity. However, their single structure struggles to fully exploit local interactions between features when processing multi-dimensional handwriting features. Therefore, there is an urgent need for a hybrid model architecture that integrates efficient temporal feature extraction and spatial feature interaction, coupled with robust training and post-processing strategies, to achieve a balance between real-time performance, accuracy, and robustness in handwriting prediction while controlling hardware costs. Summary of the Invention

[0004] In view of the above-mentioned deficiencies of the prior art, the present invention provides a real-time handwriting prediction method based on hybrid temporal convolutional networks to solve the technical problems of existing touch handwriting delay, inaccurate prediction, and unnatural experience.

[0005] To achieve the above and other related objectives, this invention provides a real-time handwriting prediction method based on a hybrid temporal convolutional network, comprising: acquiring real trajectory data of a user writing on a touchscreen; preprocessing the real trajectory data to form a multidimensional temporal feature sequence of a preset length; inputting the multidimensional temporal feature sequence into a trained trajectory point prediction model constructed based on a hybrid temporal convolutional network, and outputting predicted trajectory coordinates; performing trajectory smoothing processing based on the real trajectory data and the predicted trajectory coordinates to generate predicted trajectory data for drawing; and drawing the predicted trajectory data on the touchscreen in real time.

[0006] In one embodiment of the present invention, the real trajectory data includes at least the screen coordinates and timestamps of the trajectory points; the real trajectory data is preprocessed to form a multidimensional temporal feature sequence of a preset length, including: calculating the average velocity and angular velocity of the current trajectory point based on the coordinates and timestamps of the current trajectory point and its predecessor trajectory points; combining the coordinates, average velocity, and angular velocity of each trajectory point, and normalizing the combined features; arranging a preset number of normalized trajectory point features in chronological order to form the multidimensional temporal feature sequence.

[0007] In one embodiment of the present invention, the real trajectory data further includes writing pressure value and touch tilt angle; combining the coordinates, average velocity and angular velocity of each trajectory point includes: combining the coordinates, writing pressure value, touch tilt angle, average velocity and angular velocity of each trajectory point.

[0008] In one embodiment of the present invention, the trajectory point prediction model includes: a feature encoding module for performing preliminary feature encoding on the multidimensional temporal feature sequence; a multi-branch feature fusion module connected to the feature encoding module for performing temporal dependency feature extraction and spatial dimension interaction feature extraction in parallel on the encoded features, and fusing the two types of extracted features; and a regression output module connected to the multi-branch feature fusion module for outputting the predicted trajectory coordinates based on the fused features.

[0009] In one embodiment of the present invention, the feature encoding module includes: a convolutional layer, which uses multiple convolutional kernels to perform convolution operations on the multidimensional temporal feature sequence; a first-order normalization layer, which normalizes the output of the convolutional layer; and a first activation function layer, which performs a nonlinear transformation on the features processed by the first-order normalization layer.

[0010] In one embodiment of the present invention, the multi-branch feature fusion module includes: a temporal feature extraction branch, employing a temporal convolutional network based on causal dilated convolution, for extracting time-dependent features from the encoded features; a spatial feature extraction branch, employing a convolutional network with a kernel size of 1×1, for extracting interaction features between feature dimensions from the encoded features; and a feature fusion unit, for fusing the outputs of the temporal feature extraction branch and the spatial feature extraction branch.

[0011] In one embodiment of the present invention, the temporal feature extraction branch includes: a first temporal sub-branch, which employs a causal dilated convolution with a first dilation rate to extract temporal dependency features at a first scale; and a second temporal sub-branch, which employs a causal dilated convolution with a second dilation rate to extract temporal dependency features at a second scale; wherein the first dilation rate is different from the second dilation rate.

[0012] In one embodiment of the present invention, both the first temporal sub-branch and the second temporal sub-branch include: a feature processing sub-unit, which includes at least two sequentially connected processing units, each of the processing units including a zero-padding layer, a causal dilated convolutional layer, a second batch normalization layer, a second activation function layer, and a first dropout layer, which are sequentially connected; the zero-padding layer is used to pad zeros at the beginning of the time dimension; the causal dilated convolutional layer has 128 kernels, a size of 3, and a corresponding dilation rate; the first dropout layer is used to perform random deactivation processing with a dropout rate of 0.2; and a first residual output sub-unit is used to add the final output of the feature processing sub-unit to the input features of the temporal sub-branch, and input the addition result to the third activation function layer for processing to generate the output features of the temporal sub-branch.

[0013] In one embodiment of the present invention, the spatial feature extraction branch includes: a feature transformation subunit, which includes, in sequence: a first one-dimensional convolutional layer with 128 kernels and a size of 1; a third batch normalization layer; a fourth activation function layer using the GELU activation function; a second dropout layer with a dropout rate of 0.2; a second one-dimensional convolutional layer with 128 kernels and a size of 1; a fourth batch normalization layer; a fifth activation function layer using the GELU activation function; and a second residual output subunit, used to add the final output of the feature transformation subunit to the input features of the spatial feature extraction branch, and input the addition result to a sixth activation function layer for processing to generate the output features of the spatial feature extraction branch.

[0014] In one embodiment of the present invention, the regression output module includes, in sequence: a global average pooling layer, used to compress the input features in the time dimension and output a feature vector of fixed dimension; and a fully connected layer, used to map the feature vector to the predicted trajectory coordinates.

[0015] In one embodiment of the present invention, the trajectory point prediction model is trained and optimized using the following robust training strategy: a training dataset is constructed, which contains handwritten trajectory samples of Chinese characters, Arabic numerals, English letters and symbols from multiple users; a hybrid loss function is used to train the model, which is a weighted combination of the mean square error loss between predicted coordinates and true coordinates, and the direction consistency loss between the true trajectory direction vector and the predicted trajectory direction vector; an adaptive learning rate optimizer is used to optimize the hybrid loss function; and during the training process, an early stopping mechanism and a learning rate decay strategy are used to control the training process through callbacks.

[0016] In one embodiment of the present invention, trajectory smoothing processing is performed based on the real trajectory data and the predicted trajectory coordinates to generate predicted trajectory data for drawing. This includes: calculating the Euclidean distance between the predicted trajectory coordinates and the latest real trajectory point; if the distance is greater than a preset distance threshold, it is determined to be an anomaly point, and a replacement point is generated using a linear interpolation method; using a parametric curve to smoothly fit continuous trajectory points including the latest real trajectory point and the predicted trajectory coordinates to generate a smoothly transitioned trajectory segment; and resampling the smoothly transitioned trajectory segment according to the device screen refresh rate to generate the predicted trajectory data that matches the refresh rate and is used for real-time incremental drawing.

[0017] The beneficial effects of this invention are as follows: This invention proposes a real-time handwriting prediction method based on a hybrid temporal convolutional network. This method acquires the user's actual writing trajectory data and preprocesses it to form a temporal feature sequence that reflects the writing dynamics. This sequence is then input into a specially designed prediction model based on a hybrid temporal convolutional network, which can efficiently and accurately predict future handwriting coordinates. Subsequently, through innovative trajectory smoothing processing, the predicted trajectory is seamlessly and naturally connected to the actual trajectory, and finally drawn on the screen in real time. This method effectively compensates for the inherent latency of hardware sampling and rendering at the algorithm level, allowing the handwriting on the screen to closely follow the user's pen tip, significantly improving the real-time performance and responsiveness of writing. Simultaneously, through end-to-end optimization, the algorithm's dependence on high-end hardware is reduced. While ensuring prediction accuracy and trajectory naturalness, it provides a lower-cost, more realistic paper-and-pen digital writing solution for various touch devices. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. The accompanying drawings are incorporated in and constitute a part of this specification, illustrating embodiments consistent with this application, and are used together with the description to explain the principles of this application. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0019] Figure 1 A flowchart of a real-time handwriting prediction method provided in an embodiment of the present invention;

[0020] Figure 2 A flowchart illustrating the acquisition of multidimensional time-series feature sequences according to an embodiment of the present invention;

[0021] Figure 3 This is an architecture diagram of a trajectory point prediction model provided in an embodiment of the present invention;

[0022] Figure 4 The following is an architecture diagram of a timing sub-branch provided in an embodiment of the present invention;

[0023] Figure 5 This is an architecture diagram of a spatial feature extraction branch provided in an embodiment of the present invention;

[0024] Figure 6 This is a flowchart of trajectory post-processing provided in an embodiment of the present invention. Detailed Implementation

[0025] The following specific embodiments illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. It should be noted that, unless otherwise specified, the following embodiments and features can be combined with each other. In addition to the specific methods, equipment, and materials used in the embodiments, based on the knowledge of the prior art and the description of the present invention by those skilled in the art, any prior art methods, equipment, and materials similar to or equivalent to the methods, equipment, and materials in the embodiments of the present invention can be used to implement the present invention.

[0026] It should be understood that the terminology used in the embodiments of this invention is for describing specific implementations and not for limiting the scope of protection of this invention. Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art.

[0027] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In some embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0028] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of methods and computer program products that may be implemented according to various embodiments disclosed in the present invention. In this regard, each block in the flowchart or block diagram may represent a module, a segment of a program, or a part of code that contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions noted in the blocks may occur in a different order than that noted in the accompanying drawings. For example, two consecutive blocks shown may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a dedicated hardware-based system that performs the specified functions or operations, or may be implemented by a combination of dedicated hardware and computer instructions.

[0029] Please refer to Figure 1 , Figure 1 A real-time handwriting prediction method based on a hybrid temporal convolutional network provided by an embodiment of the present invention includes steps S101 to S105.

[0030] Step S101: Obtain the real trajectory data during the user's writing process on the touch screen. The real trajectory data refers to the original information sequence that is collected in real time by the device's touch sensor when the user writes on the touch screen. These data usually exist in the form of a series of "report points", and each report point represents the state of the pen tip or finger at a certain moment. For example, during the process of writing a Chinese character "one", the system will collect hundreds or thousands of continuous trajectory point data from the start of writing to the end of writing. Each trajectory point data usually contains at least its two-dimensional coordinates on the screen and the timestamp when the coordinate is collected. More advanced devices can also provide more dimensional data, such as the pressure value applied during writing, and the tilt angle of the stylus relative to the screen plane. These multi-dimensional data together constitute the basis for characterizing the user's writing behavior.

[0031] Step S102: Preprocess the real trajectory data to form a multi-dimensional temporal feature sequence of a preset length. There are several problems with raw trajectory data directly obtained from the device: First, the data is streaming and of variable length, making it unsuitable for direct input into deep learning models that require fixed-dimensional input; second, the numerical range of the raw data (such as coordinates) may be large and unstable, hindering rapid model convergence; finally, the raw coordinates and timestamps only reflect static positions and cannot fully express the dynamic characteristics of writing (such as speed and transitions). Therefore, preprocessing is necessary. Its objectives include: organizing the streaming data into fixed-length temporal segments that the model can process; deriving physical features (such as velocity) from the basic data that better characterize the motion state; normalizing all features to eliminate the influence of dimensions and accelerate training; and finally forming a well-organized, information-rich multi-dimensional temporal feature sequence as input to the model.

[0032] Please see Figure 2 In a specific embodiment of the present invention, the actual trajectory data includes at least the screen coordinates and timestamps of the trajectory points. Step S102 includes steps S201 to S203.

[0033] Step S201: Calculate the average velocity and angular velocity of the current trajectory point based on the coordinates and timestamps of its predecessor points. Introducing average velocity and angular velocity allows the model to not only "see" where the pen tip is, but also "understand" how it moves. Average velocity reflects the writing pace, changing during stroke transitions or connections. Angular velocity reflects the rate of change in writing direction, crucial for capturing details such as stroke transitions and curvature. For example, the angular velocity increases significantly when writing a hook. These two derived features provide the model with key dynamic information, enabling it to more accurately predict the future trajectory of handwriting under the influence of inertia, user intent, and other factors, thus maintaining prediction accuracy even when the user's writing speed changes abruptly or when performing complex strokes.

[0034] Specifically, the average velocity and angular velocity can be calculated using the following formulas:

[0035] ,

[0036] ,

[0037] ,

[0038] In the formula, The time interval between adjacent trajectory points is represented by t, which is the timestamp, x and y are the coordinates, v is the average velocity, w is the angular velocity, and the subscript n indicates the nth trajectory point.

[0039] Step S202: Combine the coordinates, average velocity, and angular velocity of each trajectory point, and normalize the combined features. During normalization, the min-max normalization method can be used to map all features to the [0,1] interval. Combining features such as coordinates, average velocity, and angular velocity essentially constructs a multivariate vector describing the "state" of each trajectory point. The subsequent normalization is crucial; it eliminates numerical differences caused by variations in the units and ranges of different features, allowing the model to pay equal attention to all features during training and preventing certain features from dominating the training process due to excessively large values. This greatly improves the stability and convergence speed of model training and helps improve the accuracy of the final prediction.

[0040] Step S203: Arrange a predetermined number of normalized trajectory point features in chronological order to form a multi-dimensional temporal feature sequence. In this step, the multi-dimensional temporal feature sequence uses a fixed sequence length and is dynamically updated, removing the oldest recorded feature data and adding newly collected feature data, looping until the current stroke is completed. Arranging a fixed number of trajectory point features in chronological order forms a standard temporal data sample. A "sliding window" mechanism is used to dynamically update this sequence (removing the oldest point and adding the newest point), ensuring that the model always predicts based on the most recent historical trajectory, thus satisfying the model's input requirements and achieving real-time, online processing of the continuous writing process.

[0041] In one specific embodiment of the present invention, the real trajectory data also includes writing pressure value and touch tilt angle. Correspondingly, the coordinates, average velocity, and angular velocity of each trajectory point are combined, including: combining the coordinates, writing pressure value, touch tilt angle, average velocity, and angular velocity of each trajectory point. When the device supports it, introducing writing pressure value and touch tilt angle can greatly enrich the feature information. Writing pressure value can simulate the weight and stroke of the handwriting, while touch tilt angle reflects the user's pen grip posture. By combining these features with motion features, the model can more comprehensively "perceive" the user's writing style and intention. For example, increased pressure may indicate the start or emphasis of a stroke, and changes in tilt angle may be related to a specific stroke direction. This makes the predicted trajectory generated by the model not only accurate in location, but also closer to real writing in terms of the "flavor" and personalized expression of the handwriting.

[0042] Step S103: Input the multi-dimensional temporal feature sequence into the trained trajectory point prediction model built on a hybrid temporal convolutional network, and output the predicted trajectory coordinates. The trained trajectory point prediction model is the core agent of the entire method. It receives the preprocessed temporal feature sequence, learns the nonlinear mapping relationship from historical trajectory patterns to future positions through its complex internal network structure, and outputs a prediction of the coordinates of the pen tip position at a future moment. This step transforms the data into predictive knowledge.

[0043] Please see Figure 3 In a specific embodiment of the present invention, the trajectory point prediction model includes a feature encoding module, a multi-branch feature fusion module, and a regression output module. The feature encoding module performs preliminary feature encoding on the multi-dimensional temporal feature sequence. The multi-branch feature fusion module, connected to the feature encoding module, performs parallel temporal dependency feature extraction and spatial dimension interaction feature extraction on the encoded features, and fuses the two types of features. The regression output module, connected to the multi-branch feature fusion module, outputs the predicted trajectory coordinates based on the fused features. The feature encoding module performs basic feature extraction on the input; the multi-branch feature fusion module is the core innovation, simultaneously mining the evolution of the trajectory in the time dimension and the intrinsic correlation of each feature in the spatial dimension; the regression output module decodes the fused high-level abstract features into specific coordinate values. This modular design makes the model structure clear and its functions well-defined.

[0044] In a specific embodiment of the present invention, the feature encoding module includes a convolutional layer, a first batch normalization layer, and a first activation function layer. The convolutional layer uses multiple convolutional kernels to perform convolution operations on the multidimensional temporal feature sequence; the first batch normalization layer normalizes the output of the convolutional layer; and the first activation function layer performs a nonlinear transformation on the features processed by the first batch normalization layer. The convolutional layer in the feature encoding module typically uses multiple (e.g., 64) small convolutional kernels of size 3 to capture local dependencies between adjacent points in the input sequence. The subsequent batch normalization layer (BN) stabilizes the distribution of intermediate features and accelerates training. The first activation function layer typically employs ReLU (Rectified Linear Unit), which introduces nonlinear transformation capabilities, enabling the model to learn more complex patterns.

[0045] In a specific embodiment of the present invention, the multi-branch feature fusion module includes a temporal feature extraction branch, a spatial feature extraction branch, and a feature fusion unit. The temporal feature extraction branch employs a temporal convolutional network based on causal dilated convolution to extract time-dependent features from the encoded features. The spatial feature extraction branch employs a convolutional network with a kernel size of 1×1 to extract interaction features between feature dimensions from the encoded features. The feature fusion unit fuses the outputs of the temporal and spatial feature extraction branches. The innovation of the multi-branch feature fusion module lies in "spatiotemporal separation and re-fusion." The temporal feature extraction branch specifically processes the information flow along the timeline, capturing the sequential dependencies of handwriting; the spatial feature extraction branch focuses on analyzing the interactions and influences between different features such as coordinates, speed, and pressure at the same point in time. Finally, the feature fusion unit integrates the two types of information, enabling the model to grasp both the temporal dynamics of writing and the collaborative relationships between features, thereby making more comprehensive predictions.

[0046] In a specific embodiment of the present invention, the temporal feature extraction branch includes two temporal sub-branches: a first temporal sub-branch and a second temporal sub-branch. The first temporal sub-branch employs a causal dilated convolution with a first dilation rate to extract temporal dependency features at a first scale. The second temporal sub-branch employs a causal dilated convolution with a second dilation rate to extract temporal dependency features at a second scale. The first and second dilation rates are different; for example, the dilation rate of the first temporal sub-branch is 3, and the dilation rate of the second temporal sub-branch is 5. Using two temporal sub-branches with different dilation rates is to construct a multi-scale temporal receptive field. The first temporal sub-branch focuses on capturing short- to medium-range, relatively fine-grained temporal dependencies (such as the trend of the current stroke); the second temporal sub-branch can capture longer-range, more macroscopic temporal patterns (such as the influence of the previous few strokes on the current stroke, or the user's overall writing rhythm). This design enables the model to process information across different time spans simultaneously, effectively handling both fast short strokes and slow long strokes, enhancing its adaptability and robustness in different writing scenarios.

[0047] Please see Figure 4In a specific embodiment of the present invention, both the first and second temporal sub-branches include a feature processing sub-unit and a first residual output sub-unit. The feature processing sub-unit includes at least two sequentially connected processing units, optimally two in total. Each processing unit includes a zero-padding layer, a causal dilated convolutional layer, a second batch normalization layer, a second activation function layer, and a first dropout layer, all sequentially connected. The zero-padding layer is used to pad zeros at the beginning of the temporal dimension. The causal dilated convolutional layer has 128 kernels, a size of 3, and a corresponding dilation rate. The second activation function layer uses the ReLU activation function. The first dropout layer performs random deactivation processing with a dropout rate of 0.2. The first residual output sub-unit adds the final output of the feature processing sub-unit to the input features of the temporal sub-branch and inputs the sum to the third activation function layer for processing to generate the output features of the temporal sub-branch. The third activation function layer uses the ReLU activation function.

[0048] In this embodiment, each temporal sub-branch employs two stacked layers of "zero-padding-causal dilated convolution-BN-activation-Dropout" processing units. Zero-padding ensures the causality of the convolution, making the prediction independent of future information. Stacking two layers increases network depth and non-linear expressive power. The dropout layer after each layer randomly "shuts down" some neurons during training, serving as an effective regularization technique to prevent the model from overfitting to the training data and improve generalization ability. Finally, residual connections bypass these two complex transformations and directly add the original input to the output, greatly alleviating the gradient vanishing problem in deep networks, ensuring that deep networks can be effectively trained, and stabilizing the training process.

[0049] Please see Figure 5In a specific embodiment of the present invention, the spatial feature extraction branch includes a feature transformation subunit and a second residual output subunit. The feature transformation subunit comprises, in sequence: a first one-dimensional convolutional layer with 128 kernels and a size of 1; a third batch normalization layer; a fourth activation function layer using the GELU activation function; a second dropout layer with a dropout rate of 0.2; a second one-dimensional convolutional layer with 128 kernels and a size of 1; a fourth batch normalization layer; and a fifth activation function layer using the GELU activation function. The second residual output subunit adds the final output of the feature transformation subunit to the input features of the spatial feature extraction branch, and inputs the sum to a sixth activation function layer for processing to generate the output features of the spatial feature extraction branch. The sixth activation function layer uses the GELU activation function. The spatial feature extraction branch uses 1x1 convolutional kernels, and its special feature is that it does not perform convolution across time steps, but rather performs linear combination and nonlinear transformation on all feature dimensions (channels) at the same time point. This is equivalent to teaching the model how different features (such as x-coordinate, pressure, and velocity) should interact and be weighted at the same time. The GELU activation function has smoother non-linearity than ReLU, potentially leading to better performance. This branch enables the model to gain a deeper understanding of the complex states of handwriting.

[0050] See also Figure 3 In a specific embodiment of the present invention, the regression output module includes a global average pooling layer and a fully connected layer connected in sequence. The global average pooling layer compresses the input features in the time dimension, outputting a feature vector of fixed dimension. The fully connected layer maps the feature vector to predicted trajectory coordinates. The global average pooling layer compresses the feature map, which still retains the time dimension after the complex processing, into a fixed-length feature vector by calculating the average value of each feature channel across all time steps. This vector encapsulates the global information of the entire input sequence. Subsequently, a simple fully connected layer (output layer) maps this global feature vector to the final two-dimensional coordinate prediction value.

[0051] In a specific embodiment of the present invention, the aforementioned fully connected layer may include: a first fully connected layer, a batch normalization layer, an activation function layer, and a second fully connected layer. The first fully connected layer performs a linear transformation on the feature vector; the batch normalization layer normalizes the output of the first fully connected layer; the activation function layer uses the ReLU activation function to perform a nonlinear transformation on the normalized features; and the second fully connected layer maps the nonlinearly transformed features to the predicted trajectory coordinates. As an optimized variant of the regression output module, after global average pooling, a fully connected layer is first used for dimensionality transformation or feature enhancement, followed by BN and ReLU for stabilization and nonlinear activation, and finally, a fully connected layer outputs the coordinates. This structure adds a small "bottleneck" layer, which helps to further refine features before final regression, sometimes resulting in a slight improvement in prediction performance and training stability.

[0052] In a specific embodiment of the present invention, the above trajectory point prediction model needs to be trained before it can predict trajectory points. During the training process, the trajectory point prediction model is trained and optimized by the following robust training strategies: (1) Construct a training dataset, which contains handwritten trajectory samples of Chinese characters, Arabic numerals, English letters and symbols from multiple users; (2) Train the model using a hybrid loss function, which is a weighted combination of the mean square error loss between the predicted coordinates and the true coordinates, and the direction consistency loss between the true trajectory direction vector and the predicted trajectory direction vector; (3) Optimize the hybrid loss function using an adaptive learning rate optimizer; (4) During the training process, use an early stopping mechanism and a learning rate decay strategy to control the training process.

[0053] A robust training strategy is crucial for the successful application of a model. First, constructing a diverse dataset containing various characters and graphics from multiple users is fundamental to ensuring the model's generalization ability and avoiding overfitting to specific handwriting styles. Second, an innovative hybrid loss function is used, requiring not only accurate predicted point locations (MSE loss) but also that the predicted trajectory direction aligns with the true trend (direction consistency loss), which geometrically constrains the reasonableness of the predictions. An adaptive learning rate optimizer (such as the Adam optimizer, with an initial learning rate set to 1×e) is employed. -4 , β1=0.9, β2=0.999, ε=1×e -7 This can automatically adjust the learning pace and accelerate convergence. Early stopping mechanism (patience=20, min_delta=1×e) -4The learning rate decay strategy (factor=0.5, patience=10) and training rate decay strategy are engineering techniques to prevent overfitting and intelligently terminate training or adjust the learning rate when model performance no longer improves. Together, they ensure the training of a high-performance and highly stable prediction model. The model can also be set to save periodically (once every 20 epochs, retaining the optimal model with validation loss) and training diagnostic tools to monitor overfitting in real time.

[0054] Specifically, the formula for calculating the hybrid loss function is as follows:

[0055] ,

[0056]

[0057] In the formula, 0.6≤α≤0.9, MSE is the mean square error between the predicted coordinates and the true coordinates, and cosθ is the cosine similarity between the true trajectory direction vector and the predicted trajectory direction vector.

[0058] Step S104: Based on the real trajectory data and the predicted trajectory coordinates, perform trajectory smoothing processing to generate predicted trajectory data for drawing.

[0059] Please see Figure 6 In a specific embodiment of the present invention, step S104 includes steps S601 to S603.

[0060] Step S601: Calculate the Euclidean distance between the predicted trajectory coordinates and the latest true trajectory point. If this distance is greater than a preset distance threshold, it is identified as an anomaly, and a replacement point is generated using a linear interpolation method. Anomaly filtering is the first safety checkpoint in post-processing. Model predictions are not perfect and may occasionally produce "jump points" that deviate too far. By calculating the Euclidean distance between the predicted point and the last true point and comparing it with an empirical threshold, these anomalies can be quickly identified. Once identified, a simple linear interpolation method is used to generate a reasonable replacement point based on the recent true trajectory trend, thereby effectively avoiding obvious trajectory breaks or jumps on the screen caused by these erroneous predictions and ensuring visual continuity.

[0061] Step S602: Use a parametric curve to smoothly fit the continuous trajectory points, including the latest real trajectory points and predicted trajectory coordinates, to generate a smoothly transitioning trajectory segment. The trajectory smoothing step aims to eliminate any abrupt connections between the endpoint of the real trajectory and the starting point of the predicted trajectory. Using a parametric curve, such as a third-order Bézier curve, and taking the latest real and predicted points as control points, a smooth transition path is fitted. This curve ensures the continuity of the tangent direction at the connection points, making the extension of the handwriting appear as a result of the user's natural writing action, rather than a rigid splicing by the algorithm, greatly improving the naturalness and aesthetics of the handwriting on the screen.

[0062] Step S603: Based on the device screen refresh rate, resample the smoothly transitioned trajectory segments to generate predicted trajectory data that matches the refresh rate for real-time incremental drawing. The rapid drawing data generation step adapts the smoothed mathematical curve to the specific hardware drawing process. Based on the device screen refresh rate (e.g., 60Hz or 120Hz), a series of dense, equally timed drawing points are resampled on the smooth curve. This is done to synchronize the drawing rhythm with the screen refresh rate. Combined with incremental updates (drawing only the newly added portion) and double buffering techniques, drawing latency and screen flicker can be minimized, ensuring that the predicted trajectory is presented to the user in real-time with the highest efficiency and smoothness, completing the final step of the "prediction-drawing" closed loop.

[0063] Step S105: Plot the predicted trajectory data on the touch screen in real time.

[0064] It should be noted that the steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they contain the same logical relationship, they are all within the scope of protection of this application. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.

[0065] In summary, this invention provides a complete real-time handwriting prediction solution, encompassing data acquisition, feature engineering, model prediction, and post-processing rendering. Its core lies in the innovative design of a hybrid temporal convolutional network model, cleverly integrating multi-scale temporal feature extraction and spatial feature interaction capabilities, supplemented by targeted robust training strategies and a meticulous trajectory post-processing workflow. This method effectively overcomes problems in existing technologies such as handwriting display latency, inaccurate prediction in complex scenes, and unnatural trajectories. Ultimately, without relying on high-end hardware, it significantly improves the real-time performance, accuracy, and smoothness of writing on touch devices, bringing users a natural and responsive digital writing experience that is infinitely close to that of real pen and paper, demonstrating significant application value and market potential.

[0066] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A real-time handwriting prediction method based on hybrid temporal convolutional networks, characterized in that, include: Acquire the actual trajectory data of the user during the touchscreen writing process; The real trajectory data is preprocessed to form a multidimensional temporal feature sequence of a preset length; The multidimensional temporal feature sequence is input into a trained trajectory point prediction model based on a hybrid temporal convolutional network, and the predicted trajectory coordinates are output. Based on the real trajectory data and the predicted trajectory coordinates, trajectory smoothing is performed to generate predicted trajectory data for drawing. The predicted trajectory data is plotted on the touchscreen in real time.

2. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 1, characterized in that, include: The actual trajectory data includes at least the screen coordinates and timestamps of the trajectory points; The real trajectory data is preprocessed to form a multidimensional time-series feature sequence of a preset length, including: Calculate the average velocity and angular velocity of the current trajectory point based on the coordinates and timestamps of the current trajectory point and its predecessor trajectory points; The coordinates, average velocity, and angular velocity of each trajectory point are combined, and the combined features are normalized. The multidimensional temporal feature sequence is formed by arranging a continuously preset number of normalized trajectory point features in chronological order.

3. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 2, characterized in that, The actual trajectory data also includes writing pressure value and touch tilt angle; The coordinates, average velocity, and angular velocity of each trajectory point are combined, including: The coordinates, writing pressure value, touch tilt angle, average speed, and angular velocity of each trajectory point are combined.

4. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 1, characterized in that, The trajectory point prediction model includes: The feature encoding module is used to perform preliminary feature encoding on the multidimensional time-series feature sequence; A multi-branch feature fusion module, connected to the feature encoding module, is used to extract temporal-dependent features and spatial-dimensional interaction features in parallel from the encoded features, and to fuse the two types of features extracted; and The regression output module, connected to the multi-branch feature fusion module, is used to output the predicted trajectory coordinates based on the fused features.

5. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 4, characterized in that, The multi-branch feature fusion module includes: The temporal feature extraction branch employs a temporal convolutional network based on causal dilated convolution to extract time-dependent features from the encoded features; The spatial feature extraction branch employs a 1×1 convolutional network to extract interaction features between feature dimensions from the encoded features; and The feature fusion unit is used to fuse the outputs of the temporal feature extraction branch and the spatial feature extraction branch.

6. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 5, characterized in that, The temporal feature extraction branch includes: The first temporal sub-branch uses a causal dilated convolution with a first dilation rate to extract temporal dependency features at the first scale. The second temporal sub-branch employs causal dilated convolution with a second dilation rate to extract temporal dependency features at the second scale. The first expansion rate is different from the second expansion rate.

7. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 6, characterized in that, Both the first timing sub-branch and the second timing sub-branch include: The feature processing subunit includes at least two sequentially connected processing units. Each processing unit includes a zero-padding layer, a causal dilated convolutional layer, a second batch normalization layer, a second activation function layer, and a first dropout layer, all sequentially connected. The zero-padding layer is used to pad zeros at the beginning of the time dimension. The causal dilated convolutional layer has 128 kernels, a size of 3, and a corresponding dilation rate. The first dropout layer is used to perform random deactivation processing with a dropout rate of 0.

2. The first residual output subunit is used to add the final output of the feature processing subunit to the input features of the temporal sub-branch, and input the addition result to the third activation function layer for processing to generate the output features of the temporal sub-branch.

8. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 5, characterized in that, The spatial feature extraction branch includes: Feature transformation subunit, which includes sequentially connected: The first one-dimensional convolutional layer has 128 kernels and a size of 1. The third batch of normalized layers; The fourth activation function layer uses the GELU activation function; The second discard layer has a discard rate of 0.

2. The second one-dimensional convolutional layer has 128 kernels and a size of 1. The fourth batch of normalized layers; The fifth activation function layer uses the GELU activation function; The second residual output subunit is used to add the final output of the feature transformation subunit to the input features of the spatial feature extraction branch, and input the addition result to the sixth activation function layer for processing to generate the output features of the spatial feature extraction branch.

9. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 1, characterized in that, The trajectory point prediction model is trained and optimized using the following robust training strategy: Construct a training dataset, which contains handwritten trajectory samples of Chinese characters, Arabic numerals, English letters and symbols from multiple users; The model is trained using a hybrid loss function, which is a weighted combination of the mean square error loss between the predicted coordinates and the true coordinates, and the directional consistency loss between the true trajectory direction vector and the predicted trajectory direction vector. The hybrid loss function is optimized using an adaptive learning rate optimizer. During training, an early stopping mechanism and a learning rate decay strategy are used to control the training process through callbacks.

10. The real-time handwriting prediction method based on hybrid temporal convolutional networks according to claim 1, characterized in that, Based on the actual trajectory data and the predicted trajectory coordinates, trajectory smoothing processing is performed to generate predicted trajectory data for drawing, including: Calculate the Euclidean distance between the predicted trajectory coordinates and the latest true trajectory point. If the distance is greater than a preset distance threshold, it is determined to be an anomaly point, and a replacement point is generated using a linear interpolation method. Parametric curves are used to smoothly fit continuous trajectory points, including the latest true trajectory points and the predicted trajectory coordinates, to generate trajectory segments with smooth transitions. Based on the device screen refresh rate, the smoothly transitioned trajectory segment is resampled to generate the predicted trajectory data that matches the refresh rate and is used for real-time incremental drawing.

Images