Handwriting image pen sequence restoration method and system

Abstract

The invention provides a handwriting image pen sequence restoration method and system, and the method comprises the steps: 1, carrying out the preprocessing of an original handwriting image through an image semantic segmentation network, and outputting a skeletonized binary image, a pen wielding eight-direction image, and a writing time gray-scale image; 2, constructing a directed grid graph based on the feature graph through a directed graph convolutional network, processing node features of the graph, and outputting key point information, relative writing time in strokes and judgment whether nodes belong to the same stroke or not; 3, learning the key nodes and the embedded features of the key nodes and the special points by using an encoder network, and outputting encoded node features to obtain a global relationship between the nodes; and 4, inputting the node features into a decoder as a memory, inputting the predicted sequence point as a target into a decoder network, decoding the features of the next prediction point, carrying out difference comparison calculation on the features of the next prediction point and the features of each point in the memory, and outputting the closest point as the next prediction point. The method can effectively solve the problem that a satisfactory effect is difficult to achieve when the number of strokes is large and the number of cross nodes is large in the pen sequence restoration method of the offline signature image.

Description

Technical Field

[0001] This application relates to the fields of image processing and pattern recognition technology, and more specifically to the technology of restoring the stroke order of handwriting images. Background Technology

[0002] Traditional handwriting image stroke order reconstruction techniques primarily rely on a series of complex and independent algorithmic steps, including image skeletalization, keypoint extraction, connectivity construction, and path search-based stroke detection and sorting. This process typically involves preprocessing the handwriting image, such as using mathematical morphology methods to thin it into a skeletalized image, then constructing a graph model by analyzing pixel connectivity, and finally using a graph search algorithm to determine the stroke sequence. Some schemes also incorporate machine learning techniques, such as training an image semantic segmentation network to predict the writing time of each pixel, using this as auxiliary information to determine the stroke order and direction.

[0003] However, this method has significant limitations. On the one hand, it heavily relies on traditional algorithms, such as skeleton thinning algorithms, graph construction algorithms, and path retrieval algorithms. These algorithms often fail to achieve ideal results when faced with complex and varied handwriting, especially when there are a large number of strokes and numerous intersecting nodes. This is because skeleton thinning may produce breaks or false connections, affecting the accuracy of subsequent graph construction and search algorithms; and path-based search algorithms are prone to confusion when dealing with densely populated areas of strokes, making it impossible to accurately reconstruct the stroke order.

[0004] On the other hand, despite the introduction of deep learning models to predict writing time, this approach still lacks a comprehensive consideration of global and local features in handwriting images, especially in capturing transitions between strokes, internal details of strokes, and handwriting style features. This makes it difficult to reconstruct a completely accurate stroke sequence even in simple handwriting, let alone in complex handwriting. Furthermore, this staged processing method increases computational costs, as each step needs to be executed separately, resulting in low overall efficiency. Moreover, due to the lack of an end-to-end training mechanism, the algorithm optimization at each stage is difficult to coordinate, affecting the quality of the final stroke sequence reconstruction.

[0005] In summary, while existing offline signature image stroke order restoration techniques can address certain handwriting restoration needs, they have significant shortcomings in handling complex handwriting, maintaining handwriting style features, and improving algorithm efficiency. There is an urgent need for a more effective, accurate, and comprehensive handwriting image stroke order restoration method. Summary of the Invention

[0006] To address the limitations of existing technologies, this application proposes an improved method for stroke order reconstruction of handwriting images. It employs an end-to-end deep learning approach combining multiple network models, enabling the entire network to progressively learn and complete each step of the method. This completely overcomes the limitations of traditional algorithms, generating satisfactory stroke order reconstruction results even for complex handwriting. Furthermore, the proposed method is highly efficient, easy to implement, and has low algorithmic complexity.

[0007] The technical solution of this application is as follows:

[0008] One aspect of this application provides a method for restoring the stroke order of a handwriting image, comprising the following steps:

[0009] Step 1: Use an image semantic segmentation network to preprocess the original handwriting image and output a multi-channel feature map, specifically including a skeletonized binary map, an eight-direction pen movement map, and a grayscale map of writing time.

[0010] Step 2: Construct a directed mesh graph based on these feature maps using a directed graph convolutional network, process the node features of the graph, and output key point information, relative writing time within strokes, and a judgment on whether nodes belong to the same stroke. When constructing the directed mesh graph, the foreground of the skeletal binary map is used as the graph node, the eight-direction stroke map is used as the graph connection edge, and the grayscale image of writing time is used as the graph node feature to construct the mesh graph;

[0011] Step 3: Use the encoder network to learn the embedding features of key nodes and special points. The network outputs the encoded node features. By comparing the encoded features pairwise, predict whether the nodes are connected in the sequence, thereby obtaining the global relationship between nodes.

[0012] Step 4: Using a decoder network, the encoded node features are input into the decoder as memory, and the predicted sequence points are input into the decoder network as targets. The features of the next predicted point are decoded and compared with the features of each point in memory. The point with the closest similarity is output as the next predicted point, until the complete handwriting sequence is restored.

[0013] Furthermore, in the preprocessing step of the handwriting image, the height and width images of the original image are concatenated with the original image as additional channels to capture the local positional features of the handwriting, thereby improving the network's sensitivity to pixel positions and recognition accuracy. The height and width images are obtained through normalization of the height and width coordinates of the corresponding pixels; the output skeletalized binary map is used to identify the handwriting skeleton, the pen direction map indicates the pen movement trend, and the writing time grayscale map estimates the handwriting timing.

[0014] Furthermore, the method for constructing a directed mesh graph is as follows: Foreground pixels in the skeletalized image of the handwriting are defined as graph nodes. Directed edges are established between adjacent nodes based on the pen stroke direction map. Each node is assigned features related to writing time information, forming a directed mesh graph. This graph is input into a directed graph convolutional network. The network transmits and aggregates node feature information according to the direction of the directed edges, learns stroke-level feature representations, and outputs a feature vector for each node. This vector is used to predict whether a node is a key point in the handwriting and the relative writing time of the node within its stroke. Based on the node features output by the network, the feature similarity between nodes is calculated to determine whether node pairs belong to the same stroke, thereby performing stroke segmentation on all nodes. Finally, based on the predicted relative writing time of the nodes, the writing order within and between each stroke is determined.

[0015] Furthermore, the method for calculating the feature similarity between nodes to determine whether they belong to the same stroke is as follows: calculate the difference in output feature vectors between all node pairs, input the difference into a linear classification layer to obtain a node relationship matrix, and predict whether the corresponding two nodes belong to the same stroke based on the matrix element values.

[0016] Furthermore, the stroke segmentation is implemented as follows: first, key nodes are selected based on key point prediction, and then the key nodes and non-key nodes are hierarchically clustered in combination with the node relationship matrix. Each clustering result represents an independent stroke.

[0017] Further, step 3 specifically includes: Node sequence construction method: Select all points predicted as key nodes and supplement with sampling of non-key nodes to construct a candidate node sequence; Temporal relationship encoding: Combine the features of each node in the candidate node sequence with the special point embedding features of the writing stage information, input them into the encoder network, model the global dependency relationship between all nodes through a self-attention mechanism, and output the encoded node features; Temporal connection prediction: Based on the encoded node features, predict whether there is a direct writing connection relationship between any two nodes in the candidate node sequence.

[0018] Furthermore, when supplementing sampling of non-critical nodes, the minimum spatial distance between the newly added node and any node in the candidate node set is limited to be greater than a preset distance threshold; the fusion method is to splice the features of each node with the embedded features of special points.

[0019] Furthermore, the specific operation for predicting the writing successor relationship is as follows: calculate the differences between each pair of encoded node features, input the differences into the classification layer, and obtain the node relationship matrix. The value of each element in the matrix reflects the probability that the next writing node of the corresponding row node is the corresponding column node.

[0020] Further, step 4 specifically includes: a temporal sequence generation process: using the candidate node feature set containing global temporal relationships output by the encoding network as a memory bank, and the previously decoded point sequence as a generated sequence; the decoder network predicts the features of the next writing node based on the memory bank and the currently generated sequence, selects the node with the best matching predicted features through matching degree calculation, and adds it to the generated sequence; repeating the iteration until a marker representing the end point of the handwriting is predicted, and if a stroke segmentation point is predicted during the process, the pen lifting event is recorded, and finally outputting a complete handwriting sequence composed of the predicted node sequence and the pen lifting events marked therein.

[0021] Furthermore, the input to the decoder network is specifically: extracting features corresponding to each node in the currently generated sequence from the memory, and superimposing positional encodings to form the input features of the decoder network.

[0022] Furthermore, the matching degree calculation specifically involves: calculating the difference vector between the predicted features and the features of each node in the memory, inputting it into the linear layer to obtain the matching score, and selecting the node with the highest matching score.

[0023] Furthermore, it also includes the model training process, which uses batch handwriting point sequence data to generate training supervision data for each layer of the network, and performs multi-task supervised training on skeleton map, time map, direction map, key points, relative time within strokes, stroke attribution, subsequent relationship of nodes, and sequence target; calculates the loss function with the output of each layer of the model network using the supervision data, and guides the iterative optimization of model parameters to minimize the total loss function.

[0024] Furthermore, the loss function includes binary cross-entropy loss, mean squared error loss, and cross-entropy loss.

[0025] Another aspect of this application provides a handwriting image stroke sequence restoration system, comprising:

[0026] The image semantic segmentation module is equipped with an image semantic segmentation network, which receives the original handwriting image and outputs a multi-channel feature map containing skeleton, orientation and time features.

[0027] The graph construction and convolution processing unit contains a multi-layer directed graph convolutional network, which is used to construct a directed mesh graph and process graph node features to output key points and stroke information.

[0028] The encoding and decoding processing unit integrates the encoder network and the decoder network. The encoder network learns the global relationships of key points, and the decoder network generates subsequent handwriting sequences based on the decoded sequence points. The learning of the encoder network depends on the key point information, and the generation process of the decoder network depends on the features output by the encoder network.

[0029] Furthermore, the image semantic segmentation module uses the U-Net basic backbone network to preprocess the original handwriting image and output a multi-channel feature map that integrates multiple features.

[0030] Furthermore, the graph construction and convolution processing unit contains a multi-layer GCN directed graph convolutional network, which is particularly suitable for handling long strokes or multiple connected strokes.

[0031] Furthermore, in addition to considering node features, the encoder network in the encoding and decoding processing unit also includes embedded features of the handwriting start point, stroke segmentation point, and handwriting end point to enhance the understanding of the global relationships of the handwriting.

[0032] The advantages of this application are:

[0033] This application employs a concatenated network of different types. First, an image semantic segmentation network predicts the skeletal map and eight-direction map, constructing a mesh map accordingly. Next, a directed graph convolutional network predicts key points and relative writing times within strokes, comparing whether pairs of nodes belong to the same stroke. This task allows the model to supervise the learning of stroke characteristics for each stroke. Then, a transformer encoder and decoder transform the graph into a sequence, ensuring that the decoded point sequence originates from key nodes in the graph, such as endpoints, inflection points, and intersections. This guarantees that the reconstructed handwriting sequence retains the original handwriting style features to the greatest extent possible. This allows the model to progressively transform handwriting from an image modality to a graph modality, and finally to a sequence modality. Each network layer has a different model structure, leveraging its unique model advantages to complete the task at that stage. This application designs different supervision tasks for each network layer to ensure that each layer extracts the desired features as much as possible.

[0034] This application innovatively proposes a method for transforming different data modalities between layers, which can ensure complete information interaction between different modalities and also ensure gradient differentiation between model layers.

[0035] This application enables end-to-end model training, is easy to supervise, does not rely on traditional unsupervised algorithms, and can quickly, accurately and efficiently restore the stroke order of handwriting to achieve satisfactory results.

[0036] In summary, the handwriting image stroke order restoration method and system provided in this application not only significantly improve the accuracy and speed of handwriting analysis, but also maintain excellent performance under complex conditions. It can be widely used in multiple fields such as electronic signature verification, handwritten note analysis, and calligraphy teaching, bringing new solutions and technological breakthroughs to fields such as electronic document security and educational technology. Attached Figure Description

[0037] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0038] Figure 1 This is a flowchart of a method for restoring the stroke order of handwriting images;

[0039] Figure 2 It is the original image;

[0040] Figure 3 It's a skeletal diagram;

[0041] Figure 4 It is an eight-direction diagram;

[0042] Figure 5 It is a time-based grayscale image;

[0043] Figure 6 It is a directed mesh graph;

[0044] Figure 7 It is a stroke diagram;

[0045] Figure 8 It is a directed graph;

[0046] Figure 9 It is a sequence diagram. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0048] It should be noted that the terms "comprising" and "having" and any variations thereof in the specification, claims and accompanying drawings of this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product or device.

[0049] This application provides a method for restoring the stroke order of handwriting images. It employs an end-to-end deep learning network model that converts handwriting images into handwriting sequences. This network model consists of four sub-networks connected in series: an image semantic segmentation network, a graph convolutional network, an encoder network, and a decoder network. (See [link to relevant documentation]). Figure 1The stroke order restoration method generally includes the following steps:

[0050] Step 1: The image semantic segmentation network receives the original handwriting image as input, preprocesses the original handwriting image, and outputs a multi-channel feature map, including a skeletal binary map, an eight-direction pen movement map, a grayscale map of writing time, and other handwriting feature maps.

[0051] Step 2: Construct a directed mesh graph by using the foreground of the skeletal image as graph nodes, the eight directions of pen strokes as graph edges, and writing time and other handwriting features as graph node features. This mesh graph is then input into a directed graph convolutional network (DCNN) to process the node features and output key point information, relative writing time within strokes, and whether nodes belong to the same stroke. The DCNN allows for smooth flow of node information within strokes, learning complete stroke features and outputting new node features to predict whether a stroke sequence contains key points, relative writing time within strokes, and whether pairs of nodes belong to the same stroke by calculating differences in their features.

[0052] Step 3: The key node features are then combined with other special points and input into the next encoder layer to learn the relationships between global nodes. The network outputs encoded node features, and by comparing pairwise encoded features, it can predict whether a pair of nodes are connected in the sequence. These special points include stroke start points, stroke segmentation points, and end points.

[0053] Step 4: Using a decoder network, the encoded node features are input into the decoder as memory, and the predicted sequence points are input into the decoder network as targets. The features of the next predicted point are decoded and compared with the features of each point in memory. The point with the closest similarity is output as the next predicted point, until the complete handwriting sequence is restored.

[0054] By applying the technical solution of this embodiment, the stroke order of the original handwriting image is effectively restored through deep neural network processing. Specifically, the multi-channel feature map generated by the image semantic segmentation network, including the skeletal binary map, the eight-direction stroke map, and the grayscale map of writing time, provides key handwriting morphology and time information for subsequent processing. The directed graph convolutional network further analyzes these features. By constructing and processing the directed mesh graph, it can identify and output key point information, the relative writing time within strokes, and determine whether nodes belong to the same stroke. This effectively improves the understanding and parsing ability of complex handwriting structures. The learning mechanism of the encoder network allows the model to capture the global relationship between nodes. By comparing the encoded features, it predicts whether nodes are connected in the sequence, thereby obtaining an overall view of the handwriting structure. The decoder network then reconstructs the sequence based on the encoded node features, outputs the attributes of the next predicted point, and determines the next optimal point in the sequence by comparing the feature differences with existing points. This method solves the problem that existing offline handwriting image stroke order restoration techniques struggle to achieve satisfactory results when faced with a large number of strokes and intersections. It ensures accurate, efficient, and logically consistent stroke order restoration even in complex handwriting scenarios, improving the reliability and readability of the restoration results.

[0055] To better understand the technical solution of this application, the following detailed description of each step and each layer of the network in the model is provided through further embodiments:

[0056] In one embodiment of this application, step 1 specifically includes: an image semantic segmentation network receives the original handwriting image as input, preprocesses the original handwriting image by concatenating the height and width images of the original image as additional channels with the original image, captures the local positional features of the handwriting, and outputs a multi-channel feature map, including a skeletal binary map, an eight-direction pen movement map, a writing time grayscale map, and other handwriting feature maps. The height and width images are obtained through normalization of the height and width coordinates of the corresponding pixels; the output skeletal binary map is used to identify the handwriting skeleton, the pen movement direction map indicates the pen movement trend, and the writing time grayscale map estimates the handwriting timing.

[0057] Specifically, the image semantic segmentation network serves as the initial layer of the entire model. This network is responsible for extracting initial features from the original image, refining the skeleton of the original image, and predicting the writing time and pen direction of each pixel for the next neighboring point sequence. U-Net can be used as the basic backbone network. Assume the input image has a height and width of [missing information]. See also Figure 2 The original image can be represented as In addition, to make the network more obviously obtain the positional information of image pixels, the height and width image can be concatenated with the original image on the channel and used as the input of U-Net.

[0058]

[0059] in Input to the model, For splicing characters, For height images, This is a width image.

[0060]

[0061]

[0062] Height image and width image A grayscale image with values ​​ranging from 0 to 1 represents the relative width and height of each pixel's location. As a supplementary feature to the original image, it helps the model to understand local location information more intuitively. This location information is particularly important prior information because most Chinese characters follow the writing habit of from left to right and from top to bottom.

[0063] The model outputs multi-channel image features with the same width and height as the input image. C represents the number of feature channels. The first channel feature represents the skeletal binary map. ( Figure 3 The second to ninth channel features represent an eight-directional binary image. ( Figure 4 The tenth channel feature represents the grayscale image of the writing time. ( Figure 5 The remaining channels may represent other image features. The eight channels represent the eight directions: top left, left, bottom left, top, bottom, top right, right, and bottom right, respectively.

[0064] In this embodiment, an image semantic segmentation network is used to preprocess the handwriting image. This preprocessing stage provides rich feature information for subsequent network layers, ensuring the integrity and accuracy of information in the process of transforming from image to graph and then to sequence modality. Especially when processing complex handwriting, the skeletal binary map and the pen direction map help to clearly define the stroke boundaries and directions, while the writing time grayscale map provides key clues for understanding the dynamic writing process of handwriting, effectively supporting the subsequent network's task of restoring the handwriting sequence, and achieving efficient and accurate stroke order restoration of handwriting images.

[0065] In some embodiments of this application, step 2 includes:

[0066] Constructing a directed mesh graph: Define the foreground pixels in the skeletalized image of the handwriting as graph nodes. Based on the pen movement direction graph of the handwriting, establish directed edges with writing direction orientation between adjacent nodes, and assign each node a feature containing writing time information, thereby forming a directed mesh graph.

[0067] Graph Node Feature Learning and Prediction: The directed mesh graph is input into a directed graph convolutional network. The network transmits and aggregates node feature information according to the direction of the directed edges, learns stroke-level feature representations, and outputs a feature vector for each node. The feature vector is used to predict at least whether the node is a key point in handwriting and the relative writing time of the node within its stroke.

[0068] Stroke segmentation and sequence restoration: Based on the node features output by the directed graph convolutional network, the feature similarity between nodes is calculated to determine whether node pairs belong to the same stroke, and then stroke segmentation is performed on all nodes; and the writing order within each stroke and between strokes is determined according to the predicted relative writing time of the nodes.

[0069] In a further embodiment, the specific method of calculating the feature similarity between nodes to determine whether a node pair belongs to the same stroke is to calculate the difference in output feature vectors between all node pairs, and input the difference into a linear classification layer to obtain a node relationship matrix. The element values ​​in this matrix are used to predict whether the corresponding two nodes belong to the same stroke.

[0070] In a further embodiment, stroke segmentation is achieved by first filtering out key nodes based on key point prediction, and then performing hierarchical clustering of the key nodes and sampled nodes in combination with the node relationship matrix, with each clustering result corresponding to an independent stroke.

[0071] In a further embodiment, based on the skeletalized binary map And eight-direction binary map Directed mesh graphs can be constructed. .in Representation diagram The node, by The foreground value corresponds to the pixel composition; Representation diagram The edge, by The prospect value determines, such as The foreground value of the first channel will form a directed edge with the pixel to its upper left. Figure 6 As an example of a directed mesh graph, different colors represent the directions of different edges. In particular, based on the number and direction of the edges, the starting point of the stroke is marked in white, and the ending point of the stroke is marked in purple.

[0072] Directed graph convolutional networks can use multi-layer GCNs as the backbone network, and the input node features of this network are: It can be accessed through Obtained through a linear layer,

[0073]

[0074] According to the diagram The defined topological structure and multi-layered GCN aggregate the node features of intra-stroke connections, thereby learning the stroke features of each node. Compared with the upper-layer image semantic segmentation network, the directed graph convolutional network can filter out a large number of irrelevant background pixels, allowing information to flow in a specified direction between the thin and narrow stroke skeletons. Even if the target stroke is particularly long and has many connected strokes, the network can efficiently learn the corresponding pixel nodes and their stroke features within the stroke.

[0075] The network output is node features. , Let D be the number of nodes, and D be the feature dimension, where the first feature dimension is... Indicates whether the node is a key point (endpoint, inflection point, or intersection of a stroke); second feature dimension This represents the relative writing time within a stroke. Furthermore, the feature differences between pairs of nodes are calculated, and then processed through a linear layer to obtain the matrix. Determine whether any two nodes belong to the same stroke.

[0076]

[0077] like Figure 7 The image shown is a stroke diagram synthesized from the output information of a directed graph convolutional network. First, based on... Filter out key points and sample the remaining non-key points at equal intervals, then according to... Hierarchical clustering is performed on all selected points. The number of clusters represents the number of strokes, and the points belonging to each cluster form a stroke segment. Based on... Each stroke segment can be sorted by time to determine its sequence number, and finally... All points within the stroke are colored according to their lightness or darkness, and the stroke image is finally rendered.

[0078] In step 2 above, the constructed directed mesh graph is used to learn and predict node features through a directed graph convolutional network. This network aggregates information based on the handwriting direction to predict whether a node is a key point and its relative writing time. Based on these prediction results, the feature similarity between nodes is calculated to determine whether nodes belong to the same stroke, thereby achieving stroke segmentation. The writing order of strokes and internal nodes is determined based on the relative writing time. The entire process not only efficiently extracts important handwriting features but also overcomes the limitations of traditional algorithms in processing complex handwriting, ensuring the complete preservation of handwriting style features and ultimately achieving accurate stroke order restoration of the handwriting image.

[0079] In some embodiments, step 3 includes: node sequence construction: selecting all nodes predicted as key nodes by the graph convolutional network and supplementing non-key nodes to form a candidate node sequence; temporal relationship encoding: fusing the features of each node in the candidate node sequence with special point embedding features representing writing stage information and inputting them into the encoder network; the encoder network models the global dependency relationship between all candidate nodes through a self-attention mechanism and outputs the encoded node features; temporal connection prediction: based on the encoded node features, calculating and predicting whether there is a direct writing successor relationship between any two nodes in the candidate node sequence.

[0080] The special points include the starting point of the handwriting, the stroke dividing point, and the ending point of the handwriting. These special points do not have specific physical coordinates in the handwriting image. These key points only serve as virtual points to tell the model the current decoding state. For example, the starting point of the handwriting indicates that no decoding has been performed yet, the stroke dividing point indicates that the decoding of the current stroke point sequence has been completed, and the ending point of the handwriting indicates that the decoding of all handwriting point sequences has been completed.

[0081] In a preferred embodiment, when supplementing sampling of non-critical nodes, the minimum spatial distance between the newly added node and any node in the existing candidate node set is constrained to be greater than a preset distance threshold; the fusion method is to concatenate the features of each node with the special point embedding features respectively.

[0082] In a preferred embodiment, the prediction of writing successor relationships is achieved by calculating the differences between each pair of the encoded node features and inputting the differences into a classification layer to obtain a node relationship matrix, wherein the value of an element in the matrix represents the probability that the node in the corresponding column is the next writing node of the node in the corresponding row.

[0083] The following embodiments illustrate one implementation method of step 3:

[0084] All nodes predicted as key nodes by the graph convolutional network are selected, and a portion of the remaining non-key nodes are randomly sampled (wherein, each random selection must ensure that the minimum Euclidean distance with the current node set is greater than a certain preset distance threshold) to form candidate sequence points. The node features also need to be processed through a linear layer to obtain new node features, and then concatenated with the embedding features of several special nodes to obtain the input node features of the encoder. . This represents the new number of nodes, which can also be understood as the length of the sequence points.

[0085]

[0086] These represent the embedding features of the stroke start point, stroke segmentation point, and stroke end point, respectively.

[0087] The backbone network of the encoder is a transformer encoder, which can fully learn the global relationships between key nodes and determine whether nodes are temporally connected. Compared to the previous layer's graph convolutional network, each layer's encoder network needs to learn the relationships between all node pairs, not just the connected node pairs defined in the edges. Therefore, the computational complexity is the square of the number of nodes. However, since the input nodes are selected nodes, the number of nodes is greatly reduced, and the overall computational cost is roughly equivalent to the previous layer. The output of the encoder network is... Furthermore, the feature differences between each pair of nodes are calculated, and then a linear layer is used to obtain the matrix. Determine the next possible node for each node.

[0088]

[0089] like Figure 8 The diagram shows the directed graph predicted by the encoder, with arrows indicating all the graphs. The predicted nodes are connected to each other.

[0090] In this embodiment, when specifically implementing the node sequence construction and temporal relationship encoding process in step 3, firstly, all points determined to be key nodes are selected from the output of the graph convolutional network. Based on this, additional non-key nodes are uniformly sampled by setting a certain distance threshold to enrich the node sequence and ensure information integrity. These key nodes include, but are not limited to, the endpoints, inflection points, and intersections of strokes, which are crucial for reconstructing the handwriting sequence. Subsequently, the features of these nodes are fused with the features of special embedding points—the start point, stroke segmentation point, and end point of the handwriting—to jointly constitute the input of the encoder network. The encoder, through a self-attention mechanism, not only enhances the representational ability of key nodes but also effectively captures the global correlation between nodes. The encoded node features output contain rich handwriting sequence information. Based on this, the encoded node features are used to predict the direct temporal connection relationship between any two nodes. This prediction is achieved by calculating the differences between node features, effectively identifying the temporal adjacency in the handwriting sequence and ensuring the coherence and accuracy of the reconstructed handwriting sequence. Through this process, this embodiment can accurately identify and process key points in handwriting images, thereby efficiently restoring the writing order of the handwriting and realizing the automation and intelligence of handwriting image analysis.

[0091] Furthermore, in the above embodiments, when supplementing non-critical nodes, special attention is paid to ensuring that the minimum spatial distance between the newly added node and any node in the existing candidate node set is greater than a preset distance threshold. This approach ensures a balanced distribution of sampling points, avoids information redundancy caused by local dense sampling, and is beneficial for improving the processing efficiency of subsequent network layers and the prediction accuracy of the model. The fusion method directly concatenates the features of each node with the embedded features of special points. In this way, each node not only contains its own handwriting features but also embeds its role information in the entire handwriting sequence, such as the starting point, stroke segmentation point, and ending point. This helps the encoder network better understand and capture the global relationships between nodes, thereby generating a handwriting sequence that is more logical and consistent with real writing habits during the decoding process. In this way, the model can gradually learn the entire process from handwriting image to stroke order restoration during training. Even when faced with complex and varied handwriting forms, it can effectively identify key points, predict stroke order and pen movement direction, and ultimately achieve accurate stroke order restoration of offline handwriting images. Of course, this design also means that the training process of the network is more controllable, reduces the dependence on traditional unsupervised algorithms, and makes the entire model architecture more concise and efficient.

[0092] In some embodiments, step 4 includes using the candidate node feature set output by the encoding network, which contains global temporal relationships, as a memory, and the previously decoded point sequence as a generated sequence; using the memory and the currently generated sequence as input, predicting the features of the next writing node through the decoder network; calculating the matching degree between the predicted features and all node features in the memory, selecting the node with the highest matching degree as the predicted next writing node, and adding it to the generated sequence; repeating the above steps iteratively until a marker representing the end point of the handwriting is predicted; during the iteration process, if a marker representing a stroke division point is predicted, a pen lifting event is recorded, and finally, a complete handwriting sequence consisting of the predicted node sequence and the pen lifting events identified therein is output. In this application, the initial generated sequence is a handwriting starting point, and then with each iteration, the next point is predicted one by one, so the generated sequence will also have one more point. That is, the length of the generated sequence is directly linked to the number of iterations.

[0093] Step 4 involves utilizing a decoder network. In a preferred embodiment, the specific implementation is as follows:

[0094] The backbone of the decoder network is the transformer decoder, which receives... As memory, based on the previously decoded point sequence The corresponding feature sequence in memory is then coupled with the embedded positional encoding. Obtain the feature input of the network ,

[0095]

[0096] The network will output the features of the next predicted location. , The feature differences between the data and all candidate points in the memory are calculated and passed through a linear layer to obtain the result. The point with the highest similarity is selected as the next prediction point. .

[0097]

[0098]

[0099] The decoder initially takes special points as input to represent the starting point of the stroke. It iteratively adds the output of the previous model to the decoded sequence of points as the input for the next model, until a special point representing the ending point of the stroke is decoded. If a special point representing a stroke division point is decoded along the way, it indicates that the stroke was lifted at that point. The final output is a complete sequence of points and their lifting and lowering states. Figure 9The image shown is a complete sequence diagram decoded by the decoder. Each stroke is colored and sorted using different colors and sequential numbers.

[0100] In this embodiment, step 4 utilizes the decoder network to calculate the matching degree between the predicted features and the features of all nodes in the memory bank, selecting the node with the highest matching degree as the next predicted writing node and adding it to the generated sequence. This process is iterated until a marker representing the end point of the handwriting is predicted. During this process, if a predicted node is identified as a stroke segmentation point, a pen lifting event is recorded. Finally, a complete handwriting sequence consisting of the predicted node sequence and the identified pen lifting events is output. This node-by-node prediction and sequence construction method ensures that the generation of the handwriting sequence conforms to the global temporal relationship and accurately captures the stroke segmentation details, achieving a high degree of restoration from handwriting image to stroke order. Furthermore, since each iteration only processes a single node, the overall computational efficiency is significantly improved, meeting the need for fast and accurate stroke order restoration of complex digital handwriting images.

[0101] In some embodiments, the method further includes a model training process. Model training requires a batch of handwriting point sequence data. This method uses the batched handwriting point sequence data to generate training supervision data for each layer of the network, including: performing multi-task supervised training on the skeleton map, time map, direction map, key points, relative time within strokes, stroke attribution, subsequent relationship of nodes, and sequence target; calculating the loss function with the supervision data and the output of each layer of the model network, and guiding the model parameters to perform iterative optimization to minimize the total loss function.

[0102] In some embodiments, the specific model training is as follows:

[0103] First, based on the point sequence data, images with different stroke thicknesses can be drawn as input images for the model. The sequence data is plotted as a skeletal image with strokes of thickness and width of 1. The writing time of the sequence points corresponding to each pixel in the skeletal image is used as the pixel value to draw a time grayscale image. The direction of the next point corresponding to each pixel in the skeletal image is plotted as an eight-direction binary map. .

[0104] Secondly, based on the lifting and lowering states of the strokes, determine which points are the start and end points of the strokes (the start and end points are collectively referred to as endpoints); calculate the range of motion angle change for each point, and points with a range of motion angle change greater than a certain threshold are considered inflection points; if multiple strokes pass through a point, it is considered an intersection point. Endpoints, inflection points, and intersection points are all marked as key points to obtain a marker mask. Reset the timer within the stroke starting from the beginning of the stroke; Furthermore, a symmetric matrix is ​​used to record whether any two nodes belong to the same stroke. .

[0105] The remaining non-critical points are sampled at equal intervals and combined with the critical points to obtain the sampled sequence points. The preceding and following relationships of the points in the sequence are marked. There is only one next point for each non-critical point; however, there may be multiple intersection points. Each stroke endpoint will have one end connected to a special point (stroke start point, stroke separator point, stroke end point). An asymmetric matrix is ​​generated based on this connection relationship. Finally, the point indices in the complete sequence are recorded as follows: .

[0106] The above data is used as supervision data, and the outputs of each layer of the model are used to calculate the loss function. This guides iterative optimization of the model parameters to minimize the total loss function. The total loss function is a linear weighted sum of the sub-loss functions.

[0107]

[0108]

[0109]

[0110] Weights of each sub-loss function The sub-loss functions are set based on human experience and calculated using the following formula:

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119] Where BCE is the binary cross-entropy loss, MSE is the mean squared error loss, and CE is the cross-entropy loss.

[0120] In some embodiments, a handwriting image stroke order restoration system is also provided, comprising:

[0121] The image semantic segmentation module is equipped with an image semantic segmentation network, which is used to receive the original handwriting image and output a multi-channel feature map containing skeletal, orientation and temporal features.

[0122] The graph construction and convolution processing unit includes a directed graph convolutional network, which is used to construct a directed mesh graph based on the skeletal binary map and the eight-direction map of the pen stroke in the multi-channel feature map, and process the graph node features to output key points and stroke information.

[0123] The encoding and decoding processing unit integrates an encoder network and a decoder network. The encoder network is used to learn the global relationships of key points, and the decoder network is used to generate subsequent handwriting sequences based on the decoded sequence points. The learning of the encoder network depends on the information of the key points, and the generation process of the decoder network depends on the output features of the encoder network.

[0124] In this embodiment, the handwriting image stroke order restoration system integrates an image semantic segmentation module, a graph construction and convolution processing unit, and an encoding and decoding processing unit, all working together to restore the writing order of complex handwriting images. The image semantic segmentation module uses an image semantic segmentation network to process the original handwriting image, generating a multi-channel feature map containing skeletal, directional, and temporal features, providing crucial information for subsequent processing. The graph construction and convolution processing unit first constructs a directed mesh graph based on the skeletal binary map and the eight-directional stroke map, then uses a directed graph convolutional network to analyze graph node features, identify key points, and parse stroke information, achieving the transformation from image features to graph structure information. The encoding and decoding processing unit further integrates the functions of an encoder network and a decoder network. The encoder network captures the global correlation between key points, while the decoder network generates subsequent handwriting sequences based on the decoded sequence points, ensuring the accuracy and coherence of the restored stroke order. In this embodiment, through a layered network design and data modality conversion, not only is the accuracy of feature extraction enhanced, but the efficiency and quality of stroke order restoration are also optimized, providing a powerful tool for offline handwriting analysis.

[0125] In some embodiments, a computer-readable storage medium is also provided, storing program instructions that, when executed by a processor, cause the processor to perform the online handwritten text segmentation method based on multimodal technology described above.

[0126] In some embodiments, an online handwritten text segmentation device based on multimodal technology is also provided, comprising: one or more processors; a memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for performing the online handwritten text segmentation method based on multimodal technology described above.

[0127] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0128] Obviously, those skilled in the art should understand that the modules or steps of the above embodiments can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular hardware and software combination.

[0129] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for restoring the stroke order of handwriting images, characterized in that, include: Step 1: Use an image semantic segmentation network to preprocess the original handwriting image and output a multi-channel feature map, including a skeletonized binary map, an eight-direction pen movement map, and a grayscale map of writing time. Step 2: Construct a directed mesh graph based on the feature map using a directed graph convolutional network, process the node features of the graph, and output key point information, relative writing time within strokes, and judgment on whether nodes belong to the same stroke. The directed mesh graph is constructed by using the foreground of the skeletal binary map as graph nodes, the eight-direction graph of pen strokes as the connecting edges of the graph, and the grayscale image of writing time as graph node features. Step 3: Use an encoder network to learn the embedding features of key nodes and special points, including the starting point of the handwriting, the stroke segmentation point, and the ending point of the handwriting. The network outputs the encoded node features, and predicts whether the pair of nodes are connected in the sequence by comparing the encoded features pairwise, so as to obtain the global relationship between the nodes. Step 4: Using a decoder network, the encoded node features are input into the decoder as memory, and the predicted sequence points are input into the decoder network as targets. The features of the next predicted point are decoded and compared with the features of each point in memory. The point with the closest similarity is output as the next predicted point, until the complete handwriting sequence is restored.

2. The handwriting image stroke sequence restoration method according to claim 1, characterized in that, Step 1, which uses an image semantic segmentation network to preprocess the handwriting image, involves concatenating the pixel position information of the original image, namely the height image and the width image, as additional channels with the original image as input to capture the local position features of the handwriting. The height image and the width image are obtained by normalizing the height and width coordinates of the corresponding pixels. The output skeletal binary map is used to identify the skeleton structure of the handwriting, the pen direction map is used to indicate the pen movement trend, and the writing time grayscale map is used to estimate the writing time sequence of the handwriting.

3. The handwriting image stroke sequence restoration method according to claim 1, characterized in that, Step 2 specifically includes: Constructing a directed mesh graph: Define the foreground pixels in the skeletalized image of the handwriting as graph nodes. Based on the pen movement direction graph of the handwriting, establish directed edges with writing direction orientation between adjacent nodes, and assign each node a feature containing writing time information, thereby forming a directed mesh graph. Graph Node Feature Learning and Prediction: The directed mesh graph is input into a directed graph convolutional network. The network transmits and aggregates node feature information according to the direction of the directed edges, learns stroke-level feature representations, and outputs a feature vector for each node. The feature vector is used to predict at least whether the node is a key point in handwriting and the relative writing time of the node within its stroke. Stroke segmentation and order determination: Based on the node features output by the directed graph convolutional network, the feature similarity between nodes is calculated to determine whether node pairs belong to the same stroke, and then stroke segmentation is performed on all nodes; and the writing order within each stroke and between strokes is determined according to the predicted relative writing time of the nodes.

4. The handwriting image stroke sequence restoration method according to claim 3, characterized in that, The specific method for calculating the feature similarity between nodes to determine whether a node pair belongs to the same stroke is to calculate the difference in the output feature vectors between all node pairs, and input the difference into a linear classification layer to obtain a node relationship matrix. The element values ​​in this matrix are used to predict whether the corresponding two nodes belong to the same stroke.

5. The handwriting image stroke sequence restoration method according to claim 3, characterized in that, The stroke segmentation is achieved by first filtering out key nodes based on key point prediction, and then performing hierarchical clustering of the key nodes and sampled nodes in combination with the node relationship matrix. Each clustering result corresponds to an independent stroke.

6. The handwriting image stroke sequence restoration method according to claim 3, characterized in that, The key points of handwriting refer to the endpoints, inflection points, or intersections of strokes.

7. The method for restoring the stroke order of a handwriting image according to any one of claims 1-6, characterized in that, Step 3 specifically includes: Node sequence construction: Select all nodes predicted as key nodes by the graph convolutional network, and supplement with sampling of non-key nodes to form a candidate node sequence; Temporal relation encoding: The features of each node in the candidate node sequence are fused with special point embedding features representing writing stage information and input into the encoder network; the encoder network models the global dependency relationship between all candidate nodes through a self-attention mechanism and outputs the encoded node features; Temporal connectivity prediction: Based on the encoded node features, calculate and predict whether there is a direct written connection between any two nodes in the candidate node sequence.

8. The handwriting image stroke sequence restoration method according to claim 7, characterized in that, When supplementing sampling of non-critical nodes, the minimum spatial distance between the newly added node and any node in the existing candidate node set is constrained to be greater than a preset distance threshold; the fusion method is to concatenate the features of each node with the embedded features of the special point respectively.

9. The handwriting image stroke sequence restoration method according to claim 7, characterized in that, The predicted writing connection relationship is achieved by calculating the differences between each pair of the encoded node features and inputting the differences into the classification layer to obtain a node relationship matrix. The value of each element in the matrix represents the probability that the node in the corresponding column is the next writing node of the node in the corresponding row.

10. The handwriting image stroke sequence restoration method according to any one of claims 1-6, characterized in that, Step 4 specifically includes: The set of candidate node features containing global temporal relationships output by the encoder network is used as a memory. The previously decoded point sequence is used as the generated sequence. The memory and the currently generated sequence are used as inputs to predict the features of the next writing node through the decoder network. The predicted features are matched with the node features in the memory. The node with the highest matching degree is selected as the predicted next writing node and added to the generated sequence. Repeat the above steps iteratively until a marker representing the end point of the handwriting is predicted. During the iteration process, if a marker representing the stroke division point is predicted, a pen lifting event is recorded. Finally, the complete handwriting sequence consisting of the predicted node sequence and the pen lifting events marked therein is output.

11. The handwriting image stroke sequence restoration method according to claim 10, characterized in that, The specific steps of using the memory bank and the currently generated sequence as input are as follows: extracting features corresponding to each node in the currently generated sequence from the memory bank, and superimposing positional encoding on these features to form the input features of the decoder network.

12. The handwriting image stroke sequence restoration method according to claim 10, characterized in that, The matching degree calculation specifically involves: calculating the difference vector between the predicted feature and the feature of each node in the memory bank, inputting the difference vector into a linear layer to obtain a matching score, and selecting the node with the highest matching score.

13. The method for restoring the stroke order of a handwriting image according to any one of claims 1 to 6, characterized in that, It also includes a model training process, which uses batch handwriting point sequence data to generate training supervision data for each layer of the network. The model training process includes steps 1-4, including: performing multi-task supervised training on the skeleton map, the time map, the direction map, the key points, the relative time within the stroke, the stroke attribution, the subsequent relationship of the node, and the sequence target; calculating the loss function by combining the supervision data with the output of each layer of the model network, and guiding the iterative optimization of the model parameters to minimize the total loss function.

14. The deep learning-based handwriting image stroke order restoration method according to claim 13, characterized in that, The loss function includes binary cross-entropy loss, mean squared error loss, and cross-entropy loss.

15. A handwriting image stroke sequence restoration system, characterized in that, The system implements the method according to any one of claims 1-14, comprising: The image semantic segmentation module is equipped with an image semantic segmentation network, which is used to receive the original handwriting image and output a multi-channel feature map containing skeletal, orientation and temporal features. The graph construction and convolution processing unit includes a directed graph convolutional network, which is used to construct a directed mesh graph based on the skeletal binary map and the eight-direction map of the pen stroke in the multi-channel feature map, and process the graph node features to output key points and stroke information. The encoding and decoding processing unit integrates an encoder network and a decoder network. The encoder network is used to learn the global relationships of the key points, and the decoder network is used to generate subsequent handwriting sequences based on the decoded sequence points. The learning of the encoder network depends on the information of the key points, and the generation process of the decoder network depends on the output features of the encoder network.

16. The handwriting image stroke sequence restoration system according to claim 15, characterized in that, The image semantic segmentation module is configured with a U-Net basic backbone network, which is used to receive the original handwriting image and output a multi-channel feature map containing skeletal, orientation and temporal features.

17. The handwriting image stroke sequence restoration system according to claim 15, characterized in that, The graph construction and convolution processing unit includes a multi-layer GCN directed graph convolutional network, which is particularly suitable for processing long strokes or multiple connected strokes.

18. The handwriting image stroke sequence restoration system according to claim 15, characterized in that, The encoder network input node features in the encoding and decoding processing unit also include the embedded features of the handwriting start point, stroke segmentation point and handwriting end point, which are used to enhance the understanding of the global relationship of the handwriting.

Images