A stroke trajectory prediction calligraphy evaluation method based on a transformer

Abstract

The present application belongs to the technical field of image intelligent scoring, and particularly relates to a handwriting evaluation method based on stroke trajectory prediction of a Transformer, comprising: S1, obtaining an input stroke image to be evaluated; S2, performing stroke segmentation on the stroke image to be evaluated by using an Attention U-Net network, and outputting a mask area of each stroke; S3, inputting the stroke segmentation result into a sequence-to-sequence network model, and outputting a predicted stroke dynamic trajectory sequence; S4, comparing the dynamic trajectory sequence with a standard reference trajectory by using a DTW algorithm, respectively calculating three deviations, and outputting a comprehensive score after weighted fusion by using a weighted fusion formula; and S5, outputting. The present application realizes high-precision recovery from a static image to a dynamic writing process by introducing a Transformer architecture, improves stroke segmentation accuracy in a complex scene by combining an attention mechanism, constructs a geometric consistency constraint to ensure physical reasonableness of a trajectory, and realizes multi-dimensional process dynamic intelligent evaluation of calligraphy.

Description

Technical Field

[0001] This invention belongs to the field of intelligent image scoring, specifically involving a calligraphy evaluation method based on Transformer-based stroke trajectory prediction. Background Technology

[0002] Current intelligent calligraphy evaluation systems primarily rely on static image segmentation techniques (such as U-Net and YOLO), using traditional image processing and convolutional neural networks to segment and compare strokes of completed characters. This method completely loses the dynamic information during the writing process, including stroke order, pen speed, pressure rhythm, and the continuity between strokes. However, the core of calligraphy teaching lies precisely in guiding the penmanship process (such as stroke order and pressure control). Existing technologies only provide result-based feedback by judging the correctness of the final character shape, failing to meet the requirements of process-oriented and formative evaluation in teaching. Furthermore, current systems typically use only RGB images as input, representing a single-modal analysis. Quality evaluation dimensions in calligraphy art, such as "penetrating the paper" and "five shades of ink," are closely related to ink density, wetness, and brushstroke shape. RGB images alone cannot effectively capture these subtle yet crucial details, limiting the depth and cultural connotation of the evaluation system. Furthermore, when faced with complex scenarios, such as background interference from cursive script, semi-cursive script, or paper, the stroke segmentation accuracy of traditional segmentation models (such as standard U-Net) drops significantly. Moreover, the structure of the segmentation model is relatively redundant, resulting in long single-character reasoning time (approximately 2 seconds per character). This makes it impossible to provide immediate feedback in scenarios such as classroom interaction and real-time practice, thus affecting user experience and teaching efficiency.

[0003] In summary, current intelligent calligraphy evaluation technology mainly relies on static image analysis, which has limitations such as missing dynamic writing information, one-sided evaluation dimensions, and insufficient accuracy in complex scenarios, making it difficult to achieve teaching feedback with process guidance value. Summary of the Invention

[0004] To address the aforementioned problems in existing technologies, this invention provides a Transformer-based stroke trajectory prediction method for calligraphy evaluation. This method solves the problems of current intelligent calligraphy evaluation technologies, which mainly rely on static image analysis, resulting in a lack of dynamic writing information, one-sided evaluation dimensions, and insufficient accuracy in complex scenarios.

[0005] A calligraphy evaluation method based on Transformer stroke trajectory prediction, characterized in that the method includes: S1: Obtain the RGB or grayscale image of the input stroke image to be evaluated; S2: The Attention U-Net network, which integrates channel attention and spatial attention mechanisms, performs pixel-level stroke segmentation on the stroke image to be evaluated and outputs the mask region of each stroke. S3: Input the stroke segmentation results into a sequence-to-sequence network model composed of DenseNet and Transformer Decoder, and output the predicted dynamic trajectory sequence of strokes, which includes the starting point, the stroke path and the ending point. S4: The dynamic trajectory sequence is compared with the standard reference trajectory using the DTW algorithm. The geometric deviation, stroke order deviation and fluency deviation are calculated respectively. The geometric deviation, stroke order deviation and fluency deviation are weighted and fused using a pre-set weighted fusion formula, and the comprehensive score is output. S5: Output and visualize the evaluation results based on the calculated comprehensive score.

[0006] Preferably, in S2, the Attention U-Net network incorporates channel attention and spatial attention mechanisms in the skip connections and is pre-trained using a pre-set hybrid loss function; The hybrid loss function includes Dice loss, binary cross-entropy loss, and boundary loss. The calculation formula for the hybrid loss function is as follows: ; In the formula, For a mixed loss function, For Dice's loss, Binary cross-entropy (BCE) loss, For boundary loss, , and These are the weighting coefficients.

[0007] Preferably, S3 includes the following sub-steps: S31: Input the input image into the DenseNet backbone network for feature extraction to obtain the feature map; S32: Flatten the feature map into a one-dimensional visual token sequence, and add a two-dimensional spatial code to each token sequence to preserve spatial information; S33: Discretize the continuous pen tip trajectory and calculate each trajectory point of the pen tip trajectory; S34: Using the cross-attention mechanism of the Transformer Decoder, predict the trajectory point token at the current moment based on the visual feature sequence and the generated trajectory history; S35: Construct geometric consistency regularization terms to constrain trajectory prediction based on spatial location and direction of motion.

[0008] Preferably, in S33, each trajectory point after discretization modeling... It is represented as: ; ; In the formula, The stroke state at time step t. This represents the relative x-direction displacement of the pen tip between adjacent time steps. This represents the relative y-direction displacement of the pen tip between adjacent time steps. This represents the pen state category: 0 for pen lifted, 1 for pen lowered, and 2 for pen finished. Let R be a two-dimensional real number space, and let R be the set of real numbers.

[0009] Preferably, in S34, predicting the trajectory point token at the current moment includes the following process: The trajectory points of the previous time step are converted into embedded vector representations, and combined with the time step position encoding of the current time step and the hidden state of the decoder in the previous time step, the starting state of the current decoding is formed. Calculate the cross-attention weights at the current time step; The context vector is input into a Transformer decoder containing self-attention and feedforward networks to update the current hidden state. This current hidden state is then projected onto the trajectory joint vocabulary dimension through a linear layer, and the probability distribution of each possible token is obtained through Softmax. The formula for calculating the probability distribution is as follows: ; In the formula, The predicted probability distribution value output by the model at time step t. To output the projection matrix, This represents the decoder's hidden state at the current time step t.

[0010] Preferably, in S35, the formula for calculating the geometric consistency regularization term is: ; In the formula, For geometric consistency regularization, For spatial location constraints, This is a constraint term for penalty for directional deviation. For directional consistency weighting coefficients, For the trajectory tangent, For the direction of the strokes, For foreground segmentation mask, For element-wise multiplication, A rasterization function that renders the predicted trajectory to a pixel plane. The angle between the tangent of the trajectory and the direction of the stroke field.

[0011] Preferably, S4 includes the following steps: The geometric deviation is obtained by calculating the positional error between the user trajectory points and the standard trajectory points; The stroke order deviation is obtained by calculating the difference between the user's pen state sequence and the standard sequence using edit distance; The smoothness deviation is obtained by calculating the second-order difference smoothness of the dynamic trajectory sequence.

[0012] Preferably, the formula for calculating the smoothness deviation is: ; In the formula, , Let T be the coordinates of the user's dynamic trajectory and the standard reference trajectory at the t-th alignment point, respectively, and T be the number of trajectory point pairs aligned by DTW.

[0013] Preferably, the evaluation method also includes employing an end-to-end joint training strategy, which involves constructing a total loss function to perform end-to-end joint optimization of the stroke segmentation network, trajectory prediction network, and geometric prior constraints. The formula for calculating the total loss function is as follows: ; In the formula, , , These are the weight coefficients for each loss term, used to balance the training objectives across different tasks. For stroke segmentation loss, For trajectory reconstruction loss, This is a geometric consistency regularization term.

[0014] The beneficial effects of this invention are as follows: Compared to existing technologies that perform static segmentation and comparison based on completed execution, this invention utilizes a sequence-to-sequence network model composed of DenseNet and Transformer Decoder to achieve end-to-end prediction of complete dynamic stroke trajectories from a single static calligraphy image. It accurately recovers dynamic information such as the starting point, stroke path, and ending point during the writing process, including key writing elements such as stroke order, stroke direction, and pressure rhythm. During stroke segmentation, channel attention and spatial attention mechanisms are introduced on top of the traditional U-Net, and nested convolutional layers are added to skip connections, effectively solving segmentation challenges in complex scenarios such as messy handwriting, cursive writing, and paper background interference. Simultaneously, a hybrid loss function including Dice loss, binary cross-entropy loss, and boundary loss is used for pre-training, making the segmentation network more sensitive to stroke boundaries, significantly improving the clarity of segmentation boundaries, and effectively suppressing the problem of stroke adhesion. Furthermore, a Sobel orientation field is introduced as a spatial structure prior, and a geometric consistency constraint loss is constructed to ensure that the predicted trajectory strictly fits the segmented stroke region in spatial position and conforms to the local edge orientation of the image in motion direction. Attached Figure Description

[0015] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings.

[0016] Figure 1 This is a schematic diagram illustrating the steps of the evaluation method of the present invention; Figure 2 This is a schematic diagram of a sub-step of step S3 in the evaluation method of the present invention. Detailed Implementation

[0017] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided.

[0018] Please see Figure 1 This embodiment provides a calligraphy evaluation method based on Transformer stroke trajectory prediction, the method including the following steps: S1: Obtain the RGB or grayscale image of the input stroke to be evaluated, including real-time acquisition of the image data of the completed characters during the user's writing process through terminal devices (such as mobile phones, tablets, or digital writing boards). The input modality supports RGB color images or grayscale images to preserve the stroke shape, ink depth, and ink thickness information, providing high-quality visual input for subsequent stroke segmentation and trajectory prediction.

[0019] S2: The Attention U-Net network, which integrates channel attention and spatial attention mechanisms, performs pixel-level stroke segmentation on the stroke image to be evaluated and outputs the mask region of each stroke. S2 includes the following implementation process: A: According to S1, the input image is or grayscale The input RGB color or grayscale image is processed by the ResNet18 backbone of the Attention U-Net network for feature extraction. Attention U-Net uses ResNet18 as the feature extraction backbone in the encoder stage, replacing the encoder composed of ordinary convolutional layers in the original U-Net. It completes the extraction of deep features through residual connections, generating feature maps at different levels. This provides high-dimensional semantic information for the subsequent decoding stage. The feature extraction formula is as follows: In the formula, Let l be the feature map of the i-th layer, and l be the total number of layers (or stages) of the output feature map. Feature extraction is performed on the input image I at the i-th layer (stage) of ResNet18.

[0020] B: To fully utilize the multi-scale information of the input image and improve the sensitivity of the Attention U-Net network to fine strokes in complex backgrounds, nested convolutional layers are added to the skip connections of the Attention U-Net network during the decoding stage to fuse features at different scales. The feature fusion formula is as follows: In the formula, For the output features of the j-th node on the i-th decoding layer, For the output feature of the (j-1)th node in the coding layer of layer i, This is a combined operation operator of convolution, batch normalization (BN), and ReLU activation. Since the skip connections in the original U-Net directly concatenate the feature maps of the encoder (deep semantic features) and the decoder (shallow detail features), the semantic differences between the two are significant, resulting in poor fusion. The feature fusion formula uses nested convolutional layers... (Convolution + BN + ReLU) performs a non-linear transformation on the concatenated features, making the cross-layer features more semantically aligned and reducing information redundancy and conflict after fusion.

[0021] C: To address illegible handwriting and background noise, channel attention and spatial attention mechanisms are introduced in skip connections: Channel attention (SE block form): ; ; In the formula, GAP(·) represents global average pooling, δ is the ReLU function, σ is the Sigmoid function, and ⊙ represents element-wise multiplication. For channel attention weights, The characteristics are those of the channel after weighting. This is the dimensionality-reduced weight matrix in channel attention. Let F be the weight matrix in the channel attention function, and F be the original feature map from the encoder. Spatial attention: ; ; In the formula, Features after spatial weighting Spatial attention weights, For 7x7 convolution kernel operations, For max pooling operations, Operations, Here, F is the sigmoid function, and F is the original feature map from the encoder. The decoder output is a fused feature map containing deep semantic features after passing through the Attention U-Net encoding-decoding path (including attention weighting and multi-scale feature fusion in skip connections). The formula for calculating the fused feature map is: = + ; The decoder upsamples progressively to obtain the segmentation result: ; in ∈ This is a stroke segmentation prediction map (segmentation mask). Each pixel value in the map (pixel value ranges from 0 to 1) represents the probability that the position belongs to the stroke category. We use point convolution to perform channel-weighted summation.

[0022] To further improve the physical rationality and consistency of subsequent trajectory prediction with writing rules, a Sobel orientation field based on image gradients is introduced as a spatial structure prior. The specific implementation process includes: The original input image I is convolved using the Sobel operators in both the horizontal and vertical directions to obtain the gradient components. and ; Calculate the gradient direction (angle) field for each pixel. With gradient magnitude field , and Together they constitute the Sobel directional field V, which is used to characterize the local edge tangent direction of the image at that pixel location, i.e. the geometric trend of stroke extension. Will As spatial weights, the Sobel orientation field V is filtered, retaining only the gradient direction information corresponding to the stroke regions to form the stroke region orientation field. It is used to filter out background noise and ensure that prior information is focused on the actual writing area.

[0023] By introducing the Sobel trajectories as differentiable and structured prior knowledge, this method establishes an explicit geometric consistency bridge between the "segmentation" and "trajectory prediction" stages. This effectively solves the problem that relying solely on pixel-level segmentation cannot guarantee the rationality of trajectory motion, and improves the physical credibility and evaluation accuracy of the overall system in the stroke restoration stage.

[0024] Furthermore, to improve segmentation accuracy under illegible handwriting and complex backgrounds, a pre-set hybrid loss function is used for pre-training. This hybrid loss function includes Dice loss, binary cross-entropy loss, and boundary loss. The formula for calculating the hybrid loss function is as follows: ; In the formula, For a mixed loss function, , and These are the weighting coefficients, The Dice loss is based on the Dice similarity coefficient and is suitable for segmentation tasks where the number of foreground (stroke) and background pixels is imbalanced. Binary Cross-Entropy (BCE) loss is used to measure the probability error of correctly classifying each pixel as a stroke or not when processing pixel classification (determining which type of stroke a character belongs to, such as horizontal, left-falling, vertical, and right-falling strokes). For boundary loss, addressing the common stroke merging problem in cursive or cursive handwriting, boundary loss enhances the consistency between the predicted and ground truth masks in the boundary region, improving the clarity of segmentation boundaries and preventing excessive merging between strokes. The calculation formulas for each loss are as follows: ; = ; ; In the formula, This represents gradient calculation, emphasizing the consistency between the prediction and the true boundary, where N is the total number of pixels. This is the actual value of the i-th pixel, taking a value of 0 or 1, representing either a non-stroke or a stroke. Let be the predicted probability value of the i-th pixel. To predict the gradient value of the mask at the i-th pixel, Let be the gradient value of the real mask at the i-th pixel; Furthermore, during the trajectory prediction process, the Attention U-Net network also includes constraints on the alignment of the trajectory with the segmented foreground, as well as directional consistency constraints on the positive field of the geometric skeleton introduced by the trajectory tangent and orientation. S3: Input the stroke segmentation results into a sequence-to-sequence network model composed of DenseNet and Transformer Decoder, and output the predicted dynamic trajectory sequence of the strokes. The dynamic trajectory sequence includes the starting point, the stroke path, and the ending point. Please refer to [link / reference]. Figure 2 S3 includes the following sub-steps: S31: The contents of S1 or grayscale The feature map F is obtained by inputting it into the backbone network DenseNet for feature extraction. D is the number of channels in the feature map, representing the length of the feature vector extracted from each spatial location. Its value is determined by the network structure of DenseNet. S32: Perform feature sequence encoding: The feature map F is transformed into a one-dimensional token sequence suitable for Transformer processing. This is achieved by flattening the feature map F into a one-dimensional visual feature sequence X, and adding a two-dimensional spatial location code to each feature vector to preserve its spatial location information in the original image. The calculation formula for the flattened visual token sequence is as follows: ; In the formula, The visual token sequence is flattened and encoded, where N is the number of tokens after flattening the feature map. The nth visual token in the sequence is obtained by linearly mapping the corresponding local image features and adding positional encoding. For the feature map F in spatial location ( The eigenvector at position () It is a linear mapping matrix. For vectorization operations, the two-dimensional spatial index is mapped to a one-dimensional sequence index. It is a two-dimensional spatial location encoding.

[0025] S33: Discretized Modeling of the Trajectory: The model discretizes the continuous pen tip trajectory. Each trajectory point... It is represented as: ; In the formula, The stroke state at time step t. This represents the relative x-direction displacement of the pen tip between adjacent time steps. This represents the relative y-direction displacement of the pen tip between adjacent time steps. R represents the pen state category: 0 for pen raised, 1 for pen lowered, and 2 for pen finished. R is the set of real numbers, which includes all rational and irrational numbers, i.e. all numbers that can be represented as decimals (positive, negative, and zero). It is a two-dimensional real space, which is a set of ordered pairs (a,b) of two real numbers, each ordered pair corresponding to a point or vector in a Cartesian coordinate system; relative displacement Quantized into a discrete positional vocabulary , pen state Quantized into a state vocabulary The position vocabulary and the state vocabulary are combined to form the joint vocabulary V for the final prediction by the model. The task of the sequence-to-sequence network model is to predict the sequence of tokens in the joint vocabulary V; S34: The Transformer Decoder's cross-attention mechanism predicts the trajectory point Token at the current time t based on the global visual features X of the image and the generated trajectory history. This includes the following processes: S341: Token of the trajectory point predicted in the previous moment, which includes displacement and pen state information. Convert to embedded vector representation e( ), and combined with time step position encoding Hidden state at decoder time step t-1 (previous time step) This forms the initial state for the current decoding process; S342: Calculate cross-attention (Encoder-Decoder Attention): The formula for calculating the attention weight distribution at time step t is: ; ; In the formula, This is a context vector, used to dynamically extract the most relevant visual information summary from the image based on the current generation requirements. , , Let be the linear projection matrices of the query, key, and value, respectively, and d be the feature dimension of the attention space. S343: Input the context vector into the Transformer decoder, which includes self-attention and feedforward networks, and update the hidden state at the current time step. The hidden state at the current moment encodes all historical trajectory information up to the current moment, as well as the relevant image context. The update formula is: ; The output layer uses discrete classification to quantize the vocabulary, including... By projecting the linear layer onto the dimension of the trajectory joint vocabulary V, and then applying Softmax, the probability distribution of each possible token is obtained. The calculation formula is as follows: ; In the formula, The predicted probability distribution value output by the model at time step t. To output the projection matrix, The decoder hidden state at the current time step t; During training, the token with the highest probability is selected as the prediction value, and during inference, it is used to generate an autoregressive value.

[0026] The predicted probability distribution values ​​output by the model are compared with the true trajectory tokens. Cross-entropy loss is used to construct the trajectory reconstruction loss. The total trajectory reconstruction loss is obtained by averaging the losses over all time steps T, and is used to guide the model in learning how to generate correct trajectory sequences from images. The calculation formula is as follows: ; In the formula, The trajectory reconstruction loss is calculated using the average error. For cross-entropy loss, For the trajectory token at time step t, The predicted probability distribution value output by the model at time step t is T, where T is the length of the trajectory sequence. The trajectory reconstruction loss is calculated by the cross-entropy between the predicted trajectory points and the real trajectory points. It directly guides the module to learn the mapping relationship from static visual features to dynamic temporal actions and is the key training signal for recovering dynamic information such as stroke order and path.

[0027] S35: Construct the geometric consistency regularization term corresponding to the geometric skeleton consistency regularization, used to constrain trajectory prediction from spatial location and motion direction, and simultaneously constrain the segmentation network (Attention U-Net network) and the trajectory prediction network (sequence-to-sequence network model), letting... A rasterization function that renders the predicted trajectory to a pixel plane. The geometric consistency regularization term, used to constrain trajectory fitting and segment the foreground, is calculated using the following formula: ; In the formula, For geometric consistency regularization, As a spatial constraint term, the degree to which the trajectory falls into the background region is quantified using the L1 norm. The smaller this value, the closer the trajectory is to the foreground segmentation of the strokes. This is a directional deviation penalty constraint term, used to quantify the degree of deviation between the trajectory direction and the stroke direction. This is the directional consistency weighting coefficient, used to adjust the proportion of spatial position constraints and motion direction constraints in the total regularization cost, achieving a dynamic balance between the two types of constraints. It is the trajectory tangent of the predicted trajectory. This is the stroke orientation field extracted based on stroke segmentation results. This is the foreground segmentation mask, specifically the stroke segmentation foreground mask output by Attention-U-Net. It is a binary matrix whose size matches the input image, with foreground pixels (stroke regions) having a value of 1 and background pixels having a value of 0. For element-wise multiplication, To predict the trajectory tangent, A rasterization function that renders the predicted trajectory to a pixel plane. The angle between the tangent of the trajectory and the direction of the stroke field.

[0028] The geometric consistency regularization term is the regularization loss value that needs to be minimized during the optimization process. Its value directly reflects the degree of consistency between the predicted trajectory and the geometric features of the stroke. By imposing dual constraints on the predicted trajectory in terms of spatial position and motion direction, the predicted trajectory is forced to fit the foreground region of the stroke segmentation and maintain consistency with the stroke direction. This is used to solve the problem that traditional trajectory prediction is prone to deviating from the stroke region, direction and writing form, and effectively improves the geometric rationality and accuracy of the trajectory prediction results.

[0029] S4: The dynamic trajectory sequence is compared with the standard reference trajectory using the DTW algorithm. Geometric deviation, stroke order deviation, and fluency deviation are calculated separately. A pre-set weighted fusion formula is used to weight and fuse the geometric deviation, stroke order deviation, and fluency deviation, and then a comprehensive score is output. The implementation process includes the following: S41: Aligning the sequence points corresponding to the dynamic trajectory and the standard reference trajectory. When evaluating a user's writing trajectory, it is first necessary to align the user's handwriting trajectory with the standard reference trajectory. Since the user's writing speed and stroke duration may differ from the standard, the Dynamic Time Warping (DTW) algorithm is used to automatically align the corresponding points of the user's trajectory with the standard trajectory. This ensures that even if the user writes faster or slower, it can maintain a reasonable match with the standard trajectory in the time dimension, thus enabling a fairer subsequent evaluation. The DTW algorithm is used to align the predicted user's dynamic trajectory with the standard reference trajectory: Alignment predicts the dynamic trajectory of users Compared with standard reference trajectory Each trajectory point contains two-dimensional coordinates. By finding an optimal alignment path that minimizes the cumulative distance between corresponding sequence points of two trajectories, the calculation formula is as follows: = ; In the formula, To align the paths, a series of paired indices (i,j) satisfying boundary conditions, monotonicity, and continuity constraints are used to align the predicted user dynamic trajectory with the standard reference trajectory in time. A pair in the alignment path represents the user's dynamic trajectory. and standard reference trajectory Alignment Let be the Euclidean distance (L2 norm) between the two trajectories.

[0030] After alignment, the T corresponding points are obtained as follows: ; In the formula, , , respectively, are the trajectory coordinates in the user dynamic trajectory and the standard reference trajectory at the t-th alignment point, where T is the number of trajectory point pairs aligned by DTW; S42: Calculate the geometric deviation, stroke order deviation, and fluency deviation respectively. The geometric deviation is obtained by calculating the positional error between the user's handwriting and the standard handwriting, which is used to quantify the difference in geometric shape between the user's handwriting and the standard handwriting. The formula for calculating the positional error is as follows: ; Geometric deviation measures the overall spatial deviation of a user's trajectory from a standard trajectory, including differences in the starting and ending points, stroke length, and direction. A smaller value indicates that the geometric shape is closer to the standard.

[0031] Stroke order deviation is calculated by measuring the difference between the user's pen state sequence and the standard sequence using edit distance. It is used to measure whether the user's writing stroke order conforms to the standard. The formula for calculating stroke order deviation is: ; In the formula, This refers to the edit distance (Levenshtein distance), which is the distance between the sequence and the edit distance obtained through insertion, deletion, and replacement operations. Convert to The minimum number of operations required is used to measure the difference between the user's writing order and the standard stroke order, and can detect stroke reordering, omissions, or redundancies. The larger the edit distance, the more serious the stroke order error.

[0032] The fluency bias is obtained by calculating the second-order difference smoothness of the predicted dynamic trajectory sequence. The fluency bias is used to quantify whether the writing trajectory is continuous and whether there are obvious pauses or jitters. The formula for calculating the second-order difference smoothness is as follows: ; In the formula, , , These are the predicted dynamic trajectory sequences, each representing the first... The two-dimensional coordinates of points t and t-1 represent the spatial position of the pen tip during the writing process. Let T be the squared magnitude of the vector. This formula normalizes the trajectory by calculating the sum of squares of the second-order differences of the points in the trajectory and dividing by T, reflecting the smoothness and coherence of the written trajectory. The larger the value, the smoother the trajectory and the more fluid the stroke.

[0033] The overall score is calculated using a weighted formula, which is as follows: ; In the formula, , , This is the fusion coefficient.

[0034] S5: Output and visualize the evaluation results based on the calculated comprehensive score, specifically including the following processes: Based on the similarity calculation results and the set threshold, the system identifies excellent and incorrect stroke annotations in the dynamic trajectory sequence and outputs the annotation results and the user's pen stroke process.

[0035] Furthermore, to achieve accurate prediction of dynamic stroke trajectories from static handwriting images, an end-to-end joint training strategy is adopted. This involves constructing a total loss function to jointly optimize the stroke segmentation network, trajectory prediction network, and geometric prior constraints end-to-end. The formula for calculating the total loss function is as follows: ; In the formula, , , These are the weight coefficients for each loss term, used to balance the training objectives across different tasks. The stroke segmentation loss is a hybrid loss function used to train the stroke segmentation network (Attention UNet). It is a weighted combination of Dice loss, binary cross-entropy loss, and boundary loss, designed to jointly address class imbalance, pixel classification, and boundary adhesion in calligraphy image segmentation. The trajectory reconstruction loss is used to optimize the prediction accuracy of the sequence-to-sequence network model for dynamic trajectory sequences. The geometric consistency regularization term is used to constrain the predicted trajectory to fit the segmented stroke region in spatial location and to conform to the image orientation field in motion direction.

[0036] During the training phase, gradient descent is used to adjust the total loss. Backpropagation is performed, and all model parameters of the stroke segmentation network (Attention U-Net) and trajectory prediction network (DenseNet+Transformer Decoder) are updated simultaneously. Through the joint optimization mechanism, the two models can learn the stroke morphology features and dynamic writing rules at the same time, so as to achieve high-precision recovery from static images to dynamic trajectories.

[0037] Step 1: During the training phase, input a batch of calligraphy image data with real stroke masks and standard trajectory sequences; Step 2: Perform forward propagation: The image is passed through a segmentation network and a trajectory prediction network to obtain the predicted stroke mask and dynamic trajectory sequence; Step 3: Calculate the three loss items mentioned above separately, and sum them according to the set weights to obtain the total loss; Step 4: Perform backpropagation using the total loss, while updating all trainable parameters in the stroke segmentation network and trajectory prediction network; Repeat steps 1-4 for iterative optimization until the model converges. This allows the two models to promote each other during training. The segmentation network provides accurate stroke region priors for trajectory prediction, while the trajectory prediction network guides the segmentation network to focus on structural features such as stroke direction through geometric consistency constraints. Ultimately, this achieves high-precision recovery from static images to dynamic trajectories, enabling the models to learn to collaboratively complete accurate stroke segmentation and reasonable dynamic trajectory reasoning.

[0038] Compared to existing technologies, this application focuses on static segmentation and comparison of completed execution. By employing a sequence-to-sequence network model composed of DenseNet and Transformer Decoder, this invention achieves end-to-end prediction of complete dynamic stroke trajectories from a single static calligraphy image. It accurately recovers dynamic information such as the starting point, stroke path, and ending point during the writing process, including key writing elements such as stroke order, stroke direction, and pressure rhythm. During stroke segmentation, channel attention and spatial attention mechanisms are introduced on top of the traditional U-Net, and nested convolutional layers are added to skip connections, effectively solving segmentation challenges in complex scenarios such as messy handwriting, cursive writing, and paper background interference. Simultaneously, a hybrid loss function including Dice loss, binary cross-entropy loss, and boundary loss is used for pre-training, making the segmentation network more sensitive to stroke boundaries, significantly improving the clarity of segmentation boundaries, and effectively suppressing the problem of stroke adhesion. Furthermore, a Sobel orientation field is introduced as a spatial structure prior, and a geometric consistency constraint loss is constructed to ensure that the predicted trajectory strictly fits the segmented stroke region in spatial position and conforms to the local edge orientation of the image in motion direction.

[0039] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A calligraphy evaluation method based on Transformer stroke trajectory prediction, characterized in that: The methods include: S1: Obtain the RGB or grayscale image of the input stroke image to be evaluated; S2: The Attention U-Net network, which integrates channel attention and spatial attention mechanisms, performs pixel-level stroke segmentation on the stroke image to be evaluated and outputs the mask region of each stroke. S3: Input the stroke segmentation results into a sequence-to-sequence network model composed of DenseNet and Transformer Decoder, and output the predicted dynamic trajectory sequence of strokes, which includes the starting point, the stroke path and the ending point. S4: The dynamic trajectory sequence is compared with the standard reference trajectory using the DTW algorithm. The geometric deviation, stroke order deviation and fluency deviation are calculated respectively. The geometric deviation, stroke order deviation and fluency deviation are weighted and fused using a pre-set weighted fusion formula, and the comprehensive score is output. S5: Output and visualize the evaluation results based on the calculated comprehensive score.

2. The calligraphy evaluation method based on Transformer stroke trajectory prediction according to claim 1, characterized in that: In S2, the Attention U-Net network incorporates channel attention and spatial attention mechanisms in the skip connections and is pre-trained using a pre-set hybrid loss function; The hybrid loss function includes Dice loss, binary cross-entropy loss, and boundary loss. The calculation formula for the hybrid loss function is as follows: ； In the formula, For a mixed loss function, For Dice's loss, Binary cross-entropy (BCE) loss, For boundary loss, , and These are the weighting coefficients.

3. The calligraphy evaluation method based on Transformer stroke trajectory prediction according to claim 1, characterized in that: S3 includes the following sub-steps: S31: Input the input image into the DenseNet backbone network for feature extraction to obtain the feature map; S32: Flatten the feature map into a one-dimensional visual token sequence, and add a two-dimensional spatial code to each token sequence to preserve spatial information; S33: Discretize the continuous pen tip trajectory and calculate each trajectory point of the pen tip trajectory; S34: Using the cross-attention mechanism of Transformer Decoder, predict the trajectory point token at the current moment based on the visual feature sequence and the generated trajectory history; S35: Construct geometric consistency regularization terms to constrain trajectory prediction based on spatial location and direction of motion.

4. The calligraphy evaluation method based on Transformer stroke trajectory prediction according to claim 3, characterized in that: In S33, each trajectory point after discretization modeling It is represented as: ； ； In the formula, The stroke state at time step t. This represents the relative x-direction displacement of the pen tip between adjacent time steps. This represents the relative y-direction displacement of the pen tip between adjacent time steps. This represents the pen state category: 0 for pen lifted, 1 for pen lowered, and 2 for pen finished. Let R be a two-dimensional real number space, and let R be the set of real numbers.

5. The calligraphy evaluation method based on Transformer stroke trajectory prediction according to claim 3, characterized in that: In S34, predicting the trajectory point token at the current moment includes the following process: The trajectory points of the previous time step are converted into embedded vector representations, and combined with the time step position encoding of the current time step and the hidden state of the decoder in the previous time step, the starting state of the current decoding is formed. Calculate the cross-attention weights at the current time step; The context vector is input into a Transformer decoder containing self-attention and feedforward networks to update the current hidden state. This current hidden state is then projected onto the trajectory joint vocabulary dimension through a linear layer, and the probability distribution of each possible token is obtained through Softmax. The formula for calculating the probability distribution is as follows: ； In the formula, The predicted probability distribution value output by the model at time step t. To output the projection matrix, This represents the decoder's hidden state at the current time step t.

6. The calligraphy evaluation method based on Transformer stroke trajectory prediction according to claim 3, characterized in that: In S35, the formula for calculating the geometric consistency regularization term is: ； In the formula, For geometric consistency regularization, For spatial location constraints, This is a constraint term for penalty for directional deviation. For directional consistency weighting coefficients, For the trajectory tangent, For the direction of the strokes, For foreground segmentation mask, For element-wise multiplication, A rasterization function that renders the predicted trajectory to a pixel plane. The angle between the tangent of the trajectory and the direction of the stroke field.

7. The calligraphy evaluation method based on Transformer stroke trajectory prediction according to claim 1, characterized in that: S4 includes the following steps: The geometric deviation is obtained by calculating the positional error between the user trajectory points and the standard trajectory points; The stroke order deviation is obtained by calculating the difference between the user's pen state sequence and the standard sequence using edit distance; The smoothness deviation is obtained by calculating the second-order difference smoothness of the dynamic trajectory sequence.

8. The calligraphy evaluation method based on Transformer stroke trajectory prediction according to claim 7, characterized in that: The formula for calculating the smoothness deviation is: ； In the formula, , Let T be the coordinates of the user's dynamic trajectory and the standard reference trajectory at the t-th alignment point, respectively, and T be the number of trajectory point pairs aligned by DTW.

9. The calligraphy evaluation method based on Transformer stroke trajectory prediction according to claim 1, characterized in that: The evaluation method also includes employing an end-to-end joint training strategy, which performs end-to-end joint optimization of the stroke segmentation network, trajectory prediction network, and geometric prior constraints by constructing a total loss function. The formula for calculating the total loss function is as follows: ； In the formula, , , These are the weight coefficients for each loss term, used to balance the training objectives across different tasks. For stroke segmentation loss, For trajectory reconstruction loss, This is a geometric consistency regularization term.

Images