An online hand-drawn table cell type identification method and device

Abstract

The application provides an online hand-drawn table cell type identification method and device, belongs to the field of computer pattern recognition, and improves the accuracy of identification by identifying the line graph and text features specific to a hand-drawn table; the stroke nodes and edges are classified by using a graph attention network, the strokes are then divided into different cells according to rules, and the cell types are identified according to the time-space relationship between the cells by using a time-space-graph attention network, so that the time-space relationship information of the strokes is fully utilized.

Description

Technical Field

[0001] This invention belongs to the field of computer pattern recognition, specifically relating to an online hand-drawn table cell type recognition method and apparatus. Background Technology

[0002] As an efficient way to express structured information, tables not only store highly condensed and concentrated information, but also transform scattered and disordered input information into logical and orderly information through their own structure, resulting in highly structured output information. Therefore, tables are widely used in creative interactive scenarios such as thought processing, knowledge organization, and discussion, helping users to classify and integrate scattered information. Users often use hand-drawing when performing such creative work to easily capture inspiration and ideas.

[0003] Online hand-drawn table cell type recognition aims to assign a label to each table cell, indicating its functional role in the data layout, facilitating users' intuitive understanding of table information during creation. In many scenarios, users demand real-time feedback on their hand-drawn tables, hoping for immediate recognition results. To achieve this, dynamic stroke classification is necessary—that is, strokes need to be categorized during the writing process. However, previous methods were largely static, requiring the user to complete the entire sketch before making a prediction. In practice, a more human-centered approach is to perform real-time prediction as the user writes.

[0004] Currently, there are roughly four main solutions for classifying table cell types: end-to-end symbolic neural network-based classification methods, tree-based Transformer network classification methods, online handwritten document and sketch stroke classification methods based on DyGAT, and image or PDF format recognition methods. Among these, the end-to-end symbolic neural network-based method leverages the advantages of embedded cell representations, learning high-dimensional table embeddings from thousands of tables and combining them with a probabilistic graphical model (Probabilistic Soft Logic (PSL)) to capture header position invariants and enforce data type homogeneity, ultimately yielding the classification result. The tree-based Transformer network classification method uses a two-dimensional tree to define cell coordinates and cell distances in a general structured table, using tree position embeddings and tree-based attention to integrate spatial and structural information for cell type classification. The DyGAT-based online handwritten document and sketch stroke classification method formalizes the document / sketch as a multi-feature graph, where nodes represent strokes and edges represent relationships between strokes. Multiple nodes are applied to a single stroke to control information flow, enabling sketch semantic segmentation, document layout analysis, and chart recognition. Image or PDF recognition methods employ techniques such as neural network methods based on textual and tree-structured information, and graph attention mechanisms.

[0005] However, while current technologies for classifying table cell types have made progress, they still suffer from limitations in meeting real-time user needs and failing to fully utilize interactive information generated during the writing process. Whether it's end-to-end symbolic neural network classification methods, tree-based Transformer network classification methods, or image or PDF-based recognition methods, these technologies all use document images or semi-structured data as input to their algorithms. They don't utilize information such as handwriting pressure or speed, and they heavily rely on the quantity and quality of the dataset, making it difficult to interpret the model. Furthermore, inputting image pixel information requires a much larger computational scale and longer recognition time compared to handwriting information. Moreover, patents related to image or PDF-based recognition methods only address table structure recognition, not cell type recognition. Simultaneously, neither method addresses real-time classification during handwriting; they must wait for the user to complete the entire draft before obtaining the document image or semi-structured data for prediction. In practice, a more user-friendly approach would be to perform real-time prediction while the user is writing. While the online handwritten document and sketch stroke classification method based on DyGAT can achieve dynamic recognition, it still has the problem of low accuracy in recognizing table cell types. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention proposes a method and device for identifying cell types in online hand-drawn tables. By recognizing the unique line graph and text features of watch tables, the accuracy of recognition is improved. By using a graph attention network to classify the nodes and edges of strokes, and then dividing the strokes into different cells according to rules, the spatiotemporal graph attention network is used to identify cell types based on the spatiotemporal relationships between cells, making full use of the spatiotemporal relationship information of strokes.

[0007] The technical solution adopted in this invention is as follows:

[0008] A method for identifying the type of hand-drawn table cells online includes the following steps:

[0009] 1) Extract strokes from hand-drawn table lines and text, and construct a structure diagram based on the extracted strokes. Each node in the diagram represents a stroke, and each edge in the structure diagram represents a spatiotemporal relationship between two strokes.

[0010] 2) Extract node and edge features based on the structure graph and input them into a multi-task learning graph attention network for node and edge classification;

[0011] 3) Based on the classification results of nodes and edges, group the strokes into each cell according to the rules;

[0012] 4) Using the above cells as nodes, connect the edges between cells that have spatiotemporal relationships to construct a relationship graph;

[0013] 5) Extract node and edge features of cells based on the relationship graph and input them into the spatiotemporal graph attention network for cell type recognition. Train and optimize the network by calculating the cross-entropy loss function and gradient descent.

[0014] 6) Utilize a trained spatiotemporal graph attention network to recognize user-drawn table strokes in real time and identify cell types.

[0015] Furthermore, in step 1), when constructing the structure graph, two strokes that are temporally adjacent are connected to their corresponding nodes by an edge in the graph, and two strokes that are spatially adjacent and whose minimum distance is less than a threshold are connected to their corresponding nodes by an edge in the graph.

[0016] Further, in step 2), the geometric and local context features of each stroke are extracted as node features, and the paired geometric features of stroke pairs are extracted as edge features.

[0017] Further, in step 2), the graph attention network consists of a shared layer block and two branches. The shared layer block includes three shared layers composed of graph attention layers, and the two branches include a node classification branch and an edge classification branch. Each branch consists of stacked attention layers plus a softmax layer.

[0018] Furthermore, the rules for grouping strokes into individual cells in step 3) include:

[0019] (1) If a pair of strokes are adjacent in time and the edge classification result between them belongs to the same cell, then group the pair of strokes into the same cell; otherwise, group them into different cells.

[0020] (2) Based on rule (1), if the time distance between two adjacent strokes in a cell is greater than a threshold, then the two strokes are grouped into two cells.

[0021] Furthermore, step 4) involves connecting edges between cells that have a spatiotemporal relationship, including:

[0022] The strokes within a cell are sampled uniformly, and the average time of each stroke is calculated. If the average time interval between two cells is less than a set threshold, then they are connected by an edge.

[0023] The strokes within a cell are sampled uniformly, and the centroid coordinates of the strokes within the cell are calculated. If the distance between the centroids of two cells is less than a set threshold, then they are connected by an edge.

[0024] Furthermore, step 5) involves extracting node features about cells based on the relationship graph, which includes:

[0025] By converting the absolute coordinates of a cell to relative coordinates, the positional characteristics of the cell can be obtained;

[0026] By using text recognition and long short-term memory networks to identify text features in cell content, the text features of cells can be obtained.

[0027] Extract the geometric features of a cell by drawing strokes within the cell.

[0028] The node features of a cell are composed of the acquired location features, text features, and geometric features;

[0029] The steps for extracting edge features of cells based on a relation graph include:

[0030] Obtaining spatial relationship edge features: Extracting the centroid vectors of node pairs with spatial relationships in the relationship graph as edge features;

[0031] Obtain edge features related to time: Obtain edge features by subtracting the average time values ​​of two cells.

[0032] Furthermore, in step 5), the spatiotemporal graph attention network is composed of stacked spatiotemporal graph attention layers, each of which includes a temporal channel and a spatial channel; the first layer performs graph convolution operation on the input node features and edge features to obtain new node features and edge features; each subsequent layer performs graph convolution operation on the new node features and edge features obtained from the previous layer to obtain updated node features and edge features again.

[0033] Furthermore, step 5) involves calculating the features of each layer of the spatiotemporal-graph attention network, including:

[0034] For connected neighbor node u and current node v, convert the edge features to the same size as the node features on each channel;

[0035] The attention values ​​of u and v in the current channel are calculated using a feedforward neural network based on the edge features of the current channel k, the node features of u and v, and the shared attention mechanism.

[0036] Calculate the attention value After processing with the softmax function, the result is... As the new node of u to v on the current channel k The contribution of; merging edge features and merging with the node features of u. Multiplication is used to aggregate information from u to the k-th channel of v, resulting in new node features of v.

[0037] By computing the node features of v in parallel for each channel and connecting all channels, new node features of v are obtained.

[0038] An online hand-drawn table cell type recognition device includes a hand-drawing panel, a memory, and a processor. The hand-drawing panel is used for users to draw tables, the memory stores table data and a computer program, and the processor executes the computer program to implement the steps of the above-described method.

[0039] The technical effects achieved by this invention are as follows:

[0040] 1. This invention first segments cells and then classifies them by type, thereby improving the accuracy of cell type recognition.

[0041] 2. This invention utilizes information such as stroke pressure, speed, and trajectory, making full use of handwriting information.

[0042] 3. This invention improves graph attention networks by mapping nodes and edges with temporal and spatial relationships to the latent space, so that the structural and semantic attributes of networks with different relationships can be encoded and saved.

[0043] 4. This invention differs from static classification in that it can provide real-time feedback. Attached Figure Description

[0044] Figure 1 This is a flowchart of an online hand-drawn table cell type recognition method according to an embodiment.

[0045] Figure 2 This is a flowchart of cell segmentation in an example.

[0046] Figure 3 This is a schematic diagram illustrating the construction of a diagram in an embodiment.

[0047] Figure 4 This is a schematic diagram of the training and prediction process of the spatiotemporal graph attention network in an embodiment.

[0048] Figure 5 This is a schematic diagram of the process for obtaining node features in an embodiment. Detailed Implementation

[0049] To make the various technical features, advantages, or effects of the present invention more apparent and understandable, a detailed description is provided below in conjunction with the accompanying drawings.

[0050] This embodiment specifically provides an online method for recognizing the cell type of a hand-drawn table. It enables real-time cell type recognition, unlike existing offline recognition methods, as it does not require waiting until the drawing is complete before performing the recognition algorithm. Figure 1As shown, the inkblot table is represented as a graph, with nodes representing strokes and edges representing the temporal and spatial relationships between strokes. Therefore, cell segmentation can be viewed as the classification of nodes and edges in the graph. After obtaining the cell segmentation results, cells are treated as nodes, edges are generated based on spatiotemporal relationships, and then node features and edge features are extracted separately and fed into a spatiotemporal graph attention network for table cell type classification. The main processing methods of this approach include cell segmentation and spatiotemporal graph attention network cell type classification, specifically including the following steps:

[0051] 1. Cell splitting

[0052] Univariate and binary geometric features extracted from strokes are used as node and edge features, respectively. These features are input into a graph attention network model in a multi-task learning mode for node and edge classification in the graph. Strokes are divided into two categories: table lines and table text. Stroke pairs are further categorized as belonging to the same cell, different cells, or having at least one stroke that is either non-table or a table line. The table is then segmented based on the stroke classification results and stroke pair classification results. The cell segmentation process is as follows: Figure 2 As shown.

[0053] 1.1 Construction of the Structure Diagram

[0054] A structural graph G(V,E) consists of nodes and edges, where a node i∈V represents a stroke, and an edge (i,j)∈E represents the relationship between a pair of strokes (i,j). Whether two nodes are connected by an edge requires consideration of the spatiotemporal relationship between the two strokes. Since the online nature of the data allows a document to be viewed as a series of strokes, temporal neighborhood captures the relationships between consecutively written strokes in the document. To utilize this temporal relationship, temporally adjacent strokes are connected by edges. On the other hand, spatial relationships are also an important source of knowledge because spatially adjacent strokes often share the same label. If two strokes x... i and x j Their minimum distance d(x) i ,x j If the value is below the set threshold T1, then the two nodes are connected. The graph construction process is as follows: Figure 3 As shown in the figure, the numbers in the figure represent the sequence of strokes in the sketch.

[0055] 1.2 Node Features and Edge Features

[0056] The geometric and local context features of each stroke are extracted as node features, and the paired geometric features of stroke pairs are extracted as edge features. Tables 1 and 2 list the node features and edge features, respectively.

[0057] Table 1 Node Characteristics

[0058]

[0059]

[0060] Table 2 Edge Features

[0061]

[0062]

[0063] 1.3 Graph Attention Network

[0064] This invention utilizes the multi-task learning graph attention network proposed by the National Key Laboratory of Multimodal Artificial Intelligence Systems at the Institute of Automation. It consists of a shared layer block and two branches: node classification and edge classification, as shown below. Figure 2 As shown, the shared layer block consists of three shared layers, each of which is a graph attention layer. Within each shared layer, only node features are updated. The node classification branch is constructed by stacking attention layers; the input is a set of node features and edge features, and the output is a new set of node features, which are then passed through a softmax layer to obtain the classification result. Similarly, the edge classification branch is also constructed by stacking attention layers; the input is a set of node features and edge features, and the output is a new set of node features, which are then passed through a softmax layer to obtain the classification result.

[0065] 1.4 Table Splitting

[0066] After obtaining the node and edge classification results, strokes are grouped into cells according to the following rule: if a pair of strokes are temporally adjacent and their edge class (i.e., the edge classification result) belongs to the same cell, then they are grouped into the same cell; otherwise, they are grouped into different cells. However, there are some obvious errors in the cell segmentation results, so a post-processing rule is proposed: if the temporal distance between two adjacent strokes in a cell is greater than a threshold δ1, then these two strokes are considered to come from different cells, and are separated into two parts from the stroke pair, grouped into two cells. The threshold δ1 is selected from the validation set.

[0067] 2. Cell type classification

[0068] A spatiotemporal-graph attention network was constructed using the cell-segmentation results. Nodes in the network represent the segmented cells, and edges represent spatiotemporal and spatial relationships, thus enabling the downstream task of cell type classification. The entire training and prediction process of the spatiotemporal-graph attention network is as follows: Figure 4 As shown.

[0069] 2.1 Relationship Diagram Construction

[0070] A relationship diagram contains two important elements: nodes and the relationships between them. The extracted cells are used as nodes in the diagram. There are two important relationships between the nodes: temporal relationships and spatial relationships. V = {v1, v2, ..., v...} m} is a set of cells, and R = {t, s} is a set of relationships between cells, where t represents time relationships and s represents spatial relationships.

[0071] To determine if nodes are connected by a time relationship: uniformly sample the strokes within a cell, then calculate the average time of each stroke. If the average time interval between two cells is less than a set threshold, then they are connected by an edge.

[0072] Whether there is a spatial relationship between nodes: uniformly sample the strokes in the cell, and then calculate the centroid coordinates of the strokes in the cell. If the distance between the centroids of two cells is less than the set threshold, then there is an edge connecting them.

[0073] 2.2 Node Features and Edge Features

[0074] 1) Node characteristics:

[0075] To better represent the characteristics of table cell types, cell coordinates are converted to relative coordinates to obtain positional features. Simultaneously, OCR is performed on the cell content to obtain text, which is then converted into vectors and processed through a Long Short-Term Memory (LSTM) network to obtain text features. Furthermore, strokes in each cell are extracted to obtain geometric features such as the number of strokes, the area, length, and width of the minimum bounding rectangle. The entire processing is as follows: Figure 5 As shown.

[0076] 2) Edge features:

[0077] To assess the varying importance of the contextual information carried by different neighbors, edge features were extracted between spatiotemporally adjacent strokes in the graph. These features can be considered as indicators of the similarity between two strokes. Therefore, they can be used to learn gates that control message passing routines between nodes, facilitating nodes' collection of useful contextual information.

[0078] Spatial relationship edge features: Centroid vectors are extracted from pairs of nodes with spatial relationships in the graph to represent their positional relationship.

[0079] Time-related edge characteristics: obtained by subtracting the average time values ​​of two cells.

[0080] 3. Spatiotemporal Graph Attention Network

[0081] Spatiotemporal graph attention networks utilize edge features with spatiotemporal relationships in a relational graph. In the graph, these relationships provide crucial information about why two nodes are connected. Both node and edge features are projected into independent spaces (channels), each channel identifying the contribution of neighborhood information to the node's current aspect.

[0082] The spatiotemporal graph attention network is composed of stacked attention layers of a spatiotemporal relation graph. The input of the l-th layer of the spatiotemporal graph attention network is a set of node features. Sum of edge features After one layer of graph convolution, a new set of node features is obtained. Sum of edge features Where N e and N r It is the number of nodes and edges. and It is the dimension of the input node and edge features. It is the dimension of the output feature.

[0083] Each layer consists of two channels: a temporal channel and a spatial channel. The goal is to extract different semantic features by projecting node v onto these two different subspaces. Considering the edge (v, i, u), which represents the edge relationship i between node v and node u, for each channel... This represents the features of node v associated with channel k. The edge features need to be converted to the same size as the nodes. This represents the characteristics of edge i in channel k. and It is the weight matrix of the k-th channel parameters.

[0084] When measuring the importance of neighbor node u to the k-th component of learning node v, edge features are considered and a shared attention mechanism is implemented:

[0085]

[0086] Where || denotes series connection, f k It is a feedforward neural network; att is an abbreviation for attention, indicating that the node u in the k-th channel should learn The weighted node v features This represents the characteristics of node u in channel k.

[0087] To make the importance more comparable, a softmax function was applied to the attention values ​​of channel k:

[0088]

[0089] Where N v R represents all first-order neighbors of node v. vzIt is node v and its neighbor node z (z∈N) v The relationship between ).

[0090] It can be regarded as a pair of neighbor nodes u The larger the value, the greater the contribution. Merged edge features and with Multiplication is used to aggregate information from neighbor node u to the k-th channel of node v:

[0091]

[0092] Where σ1 represents nonlinearity, and * denotes multiplication.

[0093] Finally, by computing the features of each channel in parallel and connecting all channels, the new features of node v are obtained:

[0094]

[0095] Where || represents series connection. In this setting, Depend on It consists of several features. By stacking multiple relational graph attention layers, high-order neighborhood information can be extracted for classification operations.

[0096] 4. Network training

[0097] Node features, edge features, and the graph are input into the spatiotemporal-graph attention network. The cross-entropy loss function is used to evaluate the difference between the spatiotemporal-graph attention network model's predictions and the actual labels. The model parameters are updated using the gradient information of the loss function, and gradient descent is used to gradually adjust the model to reduce the loss. The model's performance on unseen data is validated using a validation set. Hyperparameters such as the learning rate, number of layers, and number of attention heads are adjusted to optimize model performance, resulting in a trained spatiotemporal-graph attention network, which serves as a cell type classification model.

[0098] 5. Model Application

[0099] As the user continues to write, a node is added to the graph after each stroke is completed, the graph is segmented into cells, and then fed into a trained spatiotemporal graph attention network to obtain the results for display.

[0100] Although the present invention has been disclosed above with reference to embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions made by those skilled in the art to the technical solutions of the present invention should be covered within the protection scope of the present invention, which is defined by the claims.

Claims

1. A method for recognizing the type of hand-drawn table cells online, characterized in that, Includes the following steps: 1) Extract strokes from hand-drawn table lines and text, and construct a structure diagram based on the extracted strokes. Each node in the diagram represents a stroke, and each edge in the structure diagram represents a spatiotemporal relationship between two strokes. 2) Extract node and edge features based on the structure graph and input them into a multi-task learning graph attention network for node and edge classification; 3) Based on the classification results of nodes and edges, group the strokes into each cell according to the rules; The rules for grouping strokes into cells include: (1) If a pair of strokes are adjacent in time and the edge classification result between them belongs to the same cell, then the pair of strokes are grouped into the same cell; otherwise, they are grouped into different cells; (2) Based on rule (1), if the time distance between two adjacent strokes in a cell is greater than a threshold, then the two strokes are grouped into two cells. 4) Using the above cells as nodes, connect the cells with spatiotemporal relationships to construct a relationship graph; the steps of connecting the cells with spatiotemporal relationships include: uniformly sampling the strokes in the cell, calculating the average time of the strokes, and if the average time interval between two cells is lower than a set threshold, then an edge is connected; uniformly sampling the strokes in the cell, calculating the centroid coordinates of the strokes in the cell, and if the centroid distance between two cells is lower than a set threshold, then an edge is connected. 5) Extract node and edge features of cells based on the relationship graph and input them into the spatiotemporal graph attention network for cell type recognition. Train and optimize the network by calculating the cross-entropy loss function and gradient descent. 6) Utilize a trained spatiotemporal graph attention network to recognize user-drawn table strokes in real time and identify cell types.

2. The method as described in claim 1, characterized in that, In step 1), when constructing the structure graph, two strokes that are temporally adjacent are connected to their corresponding nodes by an edge in the graph, and two strokes that are spatially adjacent and whose minimum distance is less than a threshold are connected to their corresponding nodes by an edge in the graph.

3. The method as described in claim 1, characterized in that, In step 2), the geometric and local context features of each stroke are extracted as node features, and the paired geometric features of stroke pairs are extracted as edge features.

4. The method as described in claim 1, characterized in that, In step 2), the graph attention network consists of a shared layer block and two branches. The shared layer block includes three shared layers composed of graph attention layers. The two branches include a node classification branch and an edge classification branch. Each branch consists of stacked attention layers plus a softmax layer.

5. The method as described in claim 1, characterized in that, Step 5) involves extracting node features of cells based on the relationship graph, including: By converting the absolute coordinates of a cell to relative coordinates, the positional characteristics of the cell can be obtained; By using text recognition and long short-term memory networks to identify text features in cell content, the text features of cells can be obtained. Extract the geometric features of a cell by drawing strokes within the cell. The node features of a cell are composed of the acquired location features, text features, and geometric features; The steps for extracting edge features of cells based on a relation graph include: Obtaining spatial relationship edge features: Extracting the centroid vectors of node pairs with spatial relationships in the relationship graph as edge features; Obtain edge features related to time: Obtain edge features by subtracting the average time values ​​of two cells.

6. The method as described in claim 1, characterized in that, In step 5), the spatiotemporal graph attention network is composed of stacked spatiotemporal relationship graph attention layers, each layer including a temporal channel and a spatial channel; The first layer performs graph convolution on the input node features and edge features to obtain new node features and edge features; each subsequent layer performs graph convolution on the new node features and edge features obtained from the previous layer to obtain updated node features and edge features.

7. The method as described in claim 6, characterized in that, Step 5) involves calculating features for each layer of the spatiotemporal graph attention network, including: For connected neighbor node u and current node v, convert the edge features to the same size as the node features on each channel; The attention values ​​of u and v in the current channel are calculated using a feedforward neural network based on the edge features of the current channel k, the node features of u and v, and the shared attention mechanism. ; Calculate the attention value After processing with the softmax function, the result is... As the new node of u to v on the current channel k The contribution of; merging edge features and combining them with the node features of u. Multiplication is used to aggregate information from u to the k-th channel of v, resulting in new node features of v. ; By computing the node features of v in parallel for each channel and connecting all channels, new node features of v are obtained. .

8. An online hand-drawn table cell type recognition device, characterized in that, The method includes a drawing panel, a memory, and a processor. The drawing panel is used for a user to draw a table, the memory stores table data and a computer program, and the processor executes the computer program to implement the steps of the method according to any one of claims 1-7.

Images