ONLINE HANDWRITTEN DOCUMENT LAYOUT ANALYSIS SYSTEM.

Abstract

An online handwriting document layout analysis system (1) comprises a preprocessing unit for receiving a document composed of a plurality of strokes and generating an undirected graph comprising a plurality of nodes and a plurality of edges for representing relationships among the various strokes. A bidirectional recursive neural network unit for initializing a feature vector of each of the nodes and initializing a feature vector of each of the edges. A graph neural network unit for updating the feature vectors of the nodes and edges to obtain updated feature vectors. A fully connected neural network performs coarse-grained object classification and fine-grained object classification for each of the nodes and edges based on the updated feature vectors.A document restoration unit is used to restore the document tree structure. Figure to be published with the abstract: Fig. 2.

Description

Title of the invention: SYSTEM FOR ANALYZING THE LAYOUT OF HANDWRITTEN DOCUMENTS ONLINE. Field of invention

[0001] The invention relates to a layout analysis system, and in particular to a system for analyzing the layout of handwritten documents online.

[0002] Background of the invention.

[0003] Analyzing the layout of online handwriting documents allows segmenting strokes into different sets and determining the content type of each set of strokes, such as tables, annotations, etc. There are already relatively advanced techniques for recognizing text lines, mathematical formulas, and shapes, respectively. In the prior art, traditional algorithms for analyzing the layout of online handwriting documents are generally based on artificial neural networks.

[0004] However, artificial neural networks have difficulty in effectively utilizing the two-dimensional spatial information contained in the document and are unable to fully utilize information feature engineering. In addition, for various layout analysis works such as text or non-text classification, text line segmentation, graph detection and identification, table detection and identification, and mathematical formula detection, etc., each method can only identify a few types of document content, which does not meet the requirements of actual use. Summary of the invention

[0005] Accordingly, to improve the above-mentioned defects of the prior art, the object of the invention is to provide an online handwriting document layout analysis system, capable of simultaneously performing various fine-grained object segmentations and classifications, dividing strokes into several coarse-grained objects, and determining whether it is a paragraph, a list, a table, a diagram, or an annotation. The invention also makes it possible to subdivide the strokes of each coarse-grained object into several fine-grained objects and determine whether it is a text line, a formula, a basic shape, graffiti, or a set of mistyped strokes. Therefore, text / non-text classification, text line segmentation, graphics detection and identification, detection and identification of tables and detection of mathematical formulas can all be analyzed using the system of the invention. Brief description of the drawings

[0006] [Fig.l] is a structured block diagram showing the principal elements of the invention.

[0007] [Fig.2] is a structured block diagram showing the elements of the invention.

[0008] [Fig.3] is a structured block diagram showing the elements of the unit central to the invention.

[0009] [Fig.4] is a flow diagram of steps showing the processing of the document of the invention.

[0010] [Fig.5] is a schematic view showing the tree structure of the invention document.

[0011] [Fig.6] is a schematic view showing the processing of the invention document.

[0012] [Fig.7] is a schematic view showing the updating and classification of the feature vectors of the nodes and edges of the invention. Description of the embodiments

[0013] The invention provides an online handwriting document layout analysis system 1 for performing segmentation and classification on a range of multi-granularity objects. Referring to [Fig. 5], the system 1 is used to segment a document 300 into a plurality of coarse-grained objects to determine whether a type of each of the coarse-grained objects is a paragraph, a list, a table, a graph, or an annotation. Each of the coarse-grained objects is divided into a plurality of fine-grained objects. The system 1 is also used to determine whether each of the fine-grained objects is a line of text, a formula, a basic shape, graffiti, or a set of mistyped strokes. The coarse-grained objects and the fine-grained objects form a tree structure shown in [Fig. 5].

[0014] With reference to Figures 1 to 3, the system 1 comprises a central unit 200 which comprises a processor 2 and a memory 6 (see [Fig.3]). The processor 2 is used to execute the required operations of the system 1. The memory 6 is used to store the data, programs and results of the operations relating to the system 1.

[0015] With reference to Figures 1 and 2, the system 1 further comprises the following elements.

[0016] A preprocessing unit 10 receives the document 300 to perform preprocessing on the document 300. The document 300 is an online handwriting document composed of a plurality of strokes. The plurality of strokes consists of strokes close in time or space. The m strokes are represented by v_l, v_2, • • - , v_ m and each of the traits is represented by v_ i, in which l <i<m. Chaque trait v_ i des traits est composé de n_ i points représentés par (x_(i,l), y_(i,l)), • • (x_ (i, n_ i), y_ (i, n_ i)). Les m traits forment un ensemble KV = {v3 _1, v_2, , v_ m}.

[0017] In the preprocessing of the preprocessing unit 10, the preprocessing unit 10 is used to generate an undirected graph G = (V, E) to represent the relationships between the different strokes of the document 300 (step 100 of [Fig.4]), in which the undirected graph G comprises a plurality of nodes and a plurality of edges, V is the set defined above (Kv={v3 _i,v_2, • • - , v_ m}), and E is a set formed by the edges. Each of the nodes corresponds to a respective stroke, which is a directed sequence composed of the points of the stroke according to a writing order. Each of the edges corresponds to a pair of respective strokes that are close in time or space. The pair of respective strokes close in time or space corresponds to a respective edge. Each of the m strokes and itself form a set to form a respective edge which is called a loop. In other words, the loop is an edge connecting a line to itself.

[0018] In the undirected graph G, each of the m lines is considered to be close in time to at most N_T lines written after it. For each of the m lines, a distance d, between the line and the other lines is calculated to determine at most N_S lines spatially closest to the line. Each of N_T and N_S is a predetermined value. The distance of the closest point d to each of the two lines v_ i, v_(iA') of the m lines is represented by d (v _i, v_(iA')) = min -|- (j, jA')#V ( K (x_ (i, j) -x_ (iA', jA')) 3 A2+ K (y_ (i, j) -y_ (iA', jA')) 3 A2). The distance of the closest point d is the minimum distance between the two lines vi, v (iA'). The min_(j, jA) is a function allowing to take the minimum distance as the distance of the nearest point d from the distances between each of the points of the line v_i and all the points of the line v_(iA').

[0019] In the undirected graph G = (V, E), E=E_TUE_S. E_T= {{v _i, v_ j] ll <i<j<min {i + N_T, m]}. E_T est un ensemble composé des m traits v_l, v_2, • • - , v m et des N_T traits correspondant à chacun des m traits v_l, v_2, • • - , v m. Chacun des traits et les traits N_T correspondants qui lui sont proches dans le temps sont reliés par une arête correspondante.

[0020] E_S= {{vi, v_ (jk)} I (j_l, •••) =argsort J d (vi, vj),l <k<min { Km, N3 _S}}. E_S est un ensemble composé des m traits v_l, v_2, • • - , v _m et des N_S traits correspondant à chacun des m traits v_l, v_2, • • - , v _ m. Chacun des traits et les traits N_S correspondants les plus proches dans l'espace sont reliés par une arête correspondante. Le argsort j est une fonction permettant de trier les valeurs de d (v i, v j) dans un ordre décroissant et de renvoyer une séquence correspondante de j_ k (en référence au module NumPy du langage de programmation Python).

[0021] A bidirectional recursive neural network unit 500 is connected to the preprocessing unit 10 to initialize a feature vector of each of the nodes of the undirected graph G and initialize a feature vector of each of the edges of the undirected graph G by using recursive neural networks (RNNs). The feature vector is a vector representing the features of a corresponding paragraph, list, table, graph, and annotation, etc. In the bidirectional recursive neural network unit 500, the feature vector of each of the edges is initialized with zero values.

[0022] Referring to [Fig.2], the bidirectional recursive neural network unit 500 comprises the following elements:

[0023] A first BLSTM (Bidirectional Long Short Term Memory) unit 15 is connected to the preprocessing unit 10 to initialize the feature vector of each of the nodes and edges of the undirected graph G (step 110 of [Fig.4]). Since a spatial position relationship between a pair of features vdi, v_ (iA') can be obtained from the pair of features vi, v (iA') itself, the feature vector of each of the edges is initialized to a zero vector and is represented by E_ (i, iA') A ((0)). The feature vector of each of the edges corresponds to a set corresponding to the pair of features vi, v (iA').

[0024] The first BLSTM unit 15 is further used to extract a point feature P that incorporates contextual information, the point feature P being represented by P = (p_ (i, j ; k)) _ (i=l, • • -, m, j= 1, • • -, n_ i ; k=l, • • -, N_ F) =BLSTM(I), in which

[0025] I = (■ (x_l,l& --&x_ (1, n_l-l) @y_l,l&---&y_ (1, n_l-l) @0& - &0)B( K x3 _ (1, n_l)&---&x_(m,l) &---&x_ (m, n_m-l) &x_ (m, nm) @ K y3 _(l,n_l) &---&y_(m,l) &---&y_ (m, n_m-l) &y_ (m, nm) @1&---&0&---&0&1)), as shown in [Fig.6]

[0026] A pooling unit 20 is connected to the first BLSTM unit 15 to aggregate the point feature P into a stroke feature using average pooling (step 120 of [Fig.4]). The stroke feature is formed by the relationships between the different strokes. The stroke feature is represented by S= (s_ (i ; k)) _ (i=l, •••, m ; k=l, •••, N_F), in which s_ (i ; k) =l / n= i E_(j=l) A (ni) (i, j ; k) (as shown in [Fig.6]).

[0027] A second BLSTM (Bidirectional Long Short Term Memory) unit 25 is connected to the pooling unit 20 and is used to receive the feature of the feature from the pooling unit 20 to obtain the initialized feature vectors of the nodes corresponding to the features v_l, v_2, • • -, v_ m (step 130 of [Fig.4]). The initialized feature vectors of the nodes are represented by V_1A ((0)), V_2A ((0)), • • - , V_ mA ((0)) (as shown in [Fig.6]), in which (■ (V_1A ((0)) &V_2A ((0)) &•■•&¥_ mA ((0)))) =BLSTM(S).

[0028] A graph neural network unit 30 is connected to the second BLSTM unit 25 of the bidirectional recursive neural network unit 500. The graph neural network unit 30 is used to update the feature vector of each of the nodes and the feature vector of each of the edges to obtain an updated feature vector of each of the nodes and an updated feature vector of each of the edges by using a message-passing-based graph neural network (GNN) (step 140 of [Fig. 4]). For each of the edges, the feature vector of the edge is updated by using a current feature vector of the edge, the feature vectors of the nodes corresponding to the features of the edge, and the graph neural network.Similarly, for each of the nodes, the feature vector of the node is updated using the feature vector of the edge corresponding to the node, the feature vectors of the nodes close in time or adjacent in space and the graph neural network. The initialized feature vectors V_1A ((0)), ¥_2A ((0)), •••, ¥ mA ((0)) of the nodes obtained in step 130 are input into a graph attention network (GAT) 311 to obtain the updated feature vectors of the nodes. The initialized feature vector E_ (i, iA') A ((0)) of each of the edges is input into a forward propagation neural network (FNN) 312 to obtain the updated feature vectors of the edges. Referring to [Fig.7], the feature vectors of nodes and edges are alternately updated a predetermined number of times to obtain the updated feature vectors of nodes and edges.Updated feature vectors of nodes and edges incorporate contextual information for classification.

[0029] Referring to [Fig.7], the graph neural network of the graph neural network unit 30 is an L-layer graph neural network. The updated feature vector of each of the nodes is represented by V iA((l)) =U_VA((1)) ({V_(iA') A ((1-1)) | (vi, v_ (iA')) eE}, {E_ (i, iA') A ((1-1)) I (vi, v_ (iA')) eE}), wherein the graph attention network 311 is represented by U_VA((1)) and 1<1 <L. Le vecteur de caractéristiques mis à jour de chacune des arêtes est représenté par E_ (i, iA’) a (Q) =u_EA((l)) (E_ (i, iA’) A ((1-1)), ¥_ iA((l-1)), V_(iA’) A((l-1))), dans lequel le réseau de neurones à propagation avant 312 est représenté par U_EA((1)) et 1<1<L.

[0030] A fully connected neural network unit 35 is connected to the graph neural network unit 30. The fully connected neural network unit 35 is used to predict the type of coarse-grained object and fine-grained object for the strokes corresponding to each of the nodes and to predict whether the pair of strokes corresponding to each of the edges belongs to the same coarse-grained object or the same fine-grained object. When prediction of the fully connected neural network unit 35, coarse-grained object classification and fine-grained object classification are performed for each of the nodes and edges using fully connected neural networks based on the updated feature vectors of the nodes and edges from the graph neural network unit 30 (step 150 of [Fig.4]).

[0031] Referring to Figures 2 and 7, the fully connected neural network unit 35 comprises the following elements:

[0032] A first classifier 351 and a second classifier 352 receive the updated feature vectors of the nodes to predict the coarse-grained object type and the fine-grained object type to which the features corresponding to each of the nodes respectively belong.

[0033] A third classifier 353 and a fourth classifier 354 receive the updated feature vectors of the edges to respectively predict whether the pair of features v_ i, v_ (iA') corresponding to each of the edges belongs to the same coarse-grained object or to the same fine-grained object.

[0034] The first classifier 351, the second classifier 352, the third classifier 353 and the fourth classifier 354 are all composed of a fully connected neural network and an activation function. The activation function may be a softmax function or a sigmoid function.

[0035] A document restoration unit 40 is connected to the fully connected neural network unit 35 to restore a tree structure of the document 300 (step 160 of [Fig. 4]), as shown in [Fig. 5]. The document restoration unit 40 serves to group all the features to obtain the corresponding fine-grained objects by using connected component analysis according to the prediction results of the features belonging to the same fine-grained object in the fully connected neural network unit 35. The type of each of the corresponding fine-grained objects is determined by a confidence sum that the features of the fine-grained object are predicted to belong to a specific type of the fine-grained object in the fully connected neural network unit 35.

[0036] The document restoration unit 40 further serves to group the corresponding fine-grained objects to obtain the corresponding coarse-grained objects by using connected component analysis according to prediction results of the features belonging to the same coarse-grained object in the fully connected neural network unit 35. The type of each of the corresponding coarse-grained objects is determined by the sum of confidences that the features of the coarse-grained object are predicted to belong to a specific type of the coarse-grained object in the fully connected neural network unit 35.

[0037] The invention is thus described, but it is obvious that it can be modified in many ways. Such variations are not to be construed as a departure from the spirit and scope of the invention, and all modifications which would be obvious to a person skilled in the art are intended to be included within the scope of the following claims.

Claims

1. Claims An online handwriting document layout analysis system (1); the system is for segmenting a document into a plurality of coarse-grained objects to determine whether the type of each of the coarse-grained objects is a paragraph, a list, a table, a graph, or an annotation; each of the coarse-grained objects is divided into a plurality of fine-grained objects; the system is for determining whether the type of each of the fine-grained objects is a line of text, a formula, a basic shape, graffiti, or a set of mistyped strokes; the coarse-grained objects and the fine-grained objects form a tree structure; the system comprises: - a central unit (200) comprising a processor (2) and a memory (6); the processor serving to execute the required operations of the system; and the memory serving to store the data, - programs and results of operations relating to the system; - a preprocessing unit (10) for receiving the document (300) to perform preprocessing on the document; the document is a line handwriting document and is composed of a plurality of strokes which are close in time or space; - wherein the preprocessing unit is used to generate an undirected graph to represent the relationships between the different features of the document; the undirected graph comprises several nodes and several edges; each of the nodes corresponds to a respective feature, which is an oriented sequence composed of the points of the feature according to a writing order; each of the edges corresponds to a pair of respective features close in time or space; the pair of respective features close in time or space corresponds to a respective edge; each of the features and itself form a set to form a respective edge called a loop; that is, the loop is an edge connecting a feature to itself; - in which, in the undirected graph, each stroke is considered to be close in time to at most N_T strokes written after it; for each of the strokes, a distance to the closest point between the stroke and the other strokes is calculated to determine at most N_S strokes spatially closest to the stroke; each of N_T and N_S is a predetermined value; - a bidirectional recursive neural network unit connected to the preprocessing unit for initializing a feature vector of each of the nodes of the undirected graph and initializing a feature vector of each of the edges of the undirected graph using recursive neural networks (RNNs); in the bidirectional recursive neural network unit, the feature vector of each of the edges is initialized with zero values; - a graph neural network unit connected to the bidirectional recursive neural network unit; the graph neural network unit is used to update the feature vector of each of the nodes and the feature vector of each of the edges to obtain an updated feature vector of each of the nodes and an updated feature vector of each of the edges by using a message-passing-based graph neural network (GNN); - a neural network unit fully connected to the graph neural network unit; the fully connected neural network unit is used to predict the type of coarse-grained object and fine-grained object for the strokes corresponding to each of the nodes and to predict whether the pair of strokes corresponding to each of the edges belongs to the same coarse-grained object or the same fine-grained object; wherein, in the prediction of the fully connected neural network unit, coarse-grained object classification and fine-grained object classification are performed for each of the nodes and edges using fully connected neural networks based on the updated feature vectors of the nodes and edges from the graph neural network unit; - a document restoration unit connected to the fully connected neural network unit to restore the tree structure of the document; the document restoration unit is used to group all the features to obtain the corresponding fine-grained objects using connected component analysis

2. based on the prediction results of the features belonging to the same fine-grained object in the fully connected neural network unit; wherein, in the document restoration unit, the type of each of the corresponding fine-grained objects is determined by the sum of confidences that the features of the fine-grained object are predicted to belong to a specific type of the fine-grained object in the fully connected neural network unit; and - the document restoration unit further serves to group the corresponding fine-grained objects to obtain the corresponding coarse-grained objects by using connected component analysis based on the prediction results of the features belonging to the same coarse-grained object in the fully connected neural network unit; wherein, in the document restoration unit, the type of each of the corresponding coarse-grained objects is determined by a sum of confidences that the features of the coarse-grained object are predicted to belong to a specific type of the coarse-grained object in the fully connected neural network unit. A system The system of claim 1, wherein the bidirectional recursive neural network unit comprises: - a first BLSTM (Bidirectional Long Short Term Memory) unit connected to the preprocessing unit for initializing the feature vector of each of the nodes and each of the edges; the feature vector is a vector representing the features of a corresponding paragraph, list, table, graph and annotation; a spatial position relationship between the stroke pair is obtained from the stroke pair itself and the feature vector of each of the edges is initialized to a zero vector; the feature vector of each of the edges corresponds to a set corresponding to the stroke pair; the first BLSTM unit is further used to extract a point feature that incorporates contextual information;- a pooling unit connected to the first BLSTM unit for aggregating the point feature into a stroke feature using average pooling; the stroke feature is formed by relationships between different strokes; and; - a second BLSTM (Bidirectional Long Short Term Memory) unit connected to the pooling unit and serving to receive the feature of the stroke from the pooling unit to obtain the initialized feature vectors of the nodes corresponding to the strokes.

3. The system of claim 2, wherein, in the graph neural network unit, for each of the edges, the feature vector of the edge is updated using a current feature vector of the edge, the feature vectors of the nodes that correspond to the strokes of the edge, and the graph neural network; for each of the nodes, the feature vector of the node is updated using the feature vector of the edge that corresponds to the node, the feature vectors of the nodes that are close in time or adjacent in space, and the graph neural network; the feature vectors of the nodes and edges are alternately updated a predetermined number of times to obtain the updated feature vectors of the nodes and edges; and the updated feature vectors of the nodes and edges incorporate the contextual information for classification.

4. The system of claim 3, wherein the fully connected neural network unit comprises: - a first classifier and a second classifier for receiving the updated feature vectors of the nodes to predict the coarse-grained object type and the fine-grained object type to which each of the strokes corresponding to each of the nodes respectively belongs; - a third classifier and a fourth classifier for receiving the updated feature vectors of the edges to respectively predict whether the pair of strokes corresponding to each of the edges belongs to the same coarse-grained object or the same fine-grained object; and - wherein the first classifier, the second classifier, the third classifier and the fourth classifier are all composed of a fully connected neural network and an activation function.

Images