Dynamic graph representation table recognition method based on variational autoencoder

Abstract

This invention relates to the field of document intelligent processing technology, and more particularly to a dynamic graph representation table recognition method based on variational autoencoders (VAEs). This method is used to read implicit content relationships from complex table images. It is characterized by the following steps: S1, preprocessing step; S2, VAE encoding step; S3, VAE decoding step; S4, optimization step; S5, output step. This invention captures the distribution characteristics and uncertainties of table relationships through VAE encoding latent space, and the decoding module infers the dynamic relationships between nodes. It effectively handles implicit relationships across rows or columns in multi-level headers and blank cells, such as hierarchical dependencies, causal associations, and numerical calculation relationships, solving the problem of existing technologies neglecting non-local relationships.

Description

Technical Field

[0001] This invention relates to the field of intelligent document processing technology, and in particular to a dynamic graph representation table recognition method based on variational autoencoders. Background Technology

[0002] Tables, as an important form of information presentation, are widely used in finance, scientific research, government affairs, and many other fields. With the advancement of digitalization, a large number of tables exist in the form of images or scanned documents, making the need for automated parsing increasingly urgent. Complex tables often contain multiple layers of headers, blank cells, and structures spanning rows or columns, and may suffer from scanning distortion, noise interference, and other problems, posing a significant challenge to accurate recognition.

[0003] Existing table recognition technologies have many limitations: traditional table recognition methods rely on manually designed rules or features such as font and alignment, making it difficult to capture implicit relationships across rows or columns, and they are poorly adaptable to irregular layouts;

[0004] Although deep learning methods have introduced semantic embedding techniques, they tend to focus on explicit structures and ignore non-local relationships. Their accuracy drops significantly in scenarios with multiple headers or blank cells, and they are also susceptible to geometric offsets that can cause cells to stick together.

[0005] Traditional table recognition methods are inadequate in handling uncertainty. Existing technologies are not robust to interference such as scanning distortion and noise. Rule-based methods require manual parameter adjustment, and end-to-end models lack an effective uncertainty quantification mechanism. At the same time, the aggregation computation cost of graph neural networks (GNNs) is high, which limits real-time performance.

[0006] Poor generalization and privacy risks limit its applicability in professional fields such as finance and scientific research. The lack of masking mechanisms during the extraction of sensitive content can easily lead to data leakage. Some PDF table extraction technologies only focus on format conversion and ignore the mining of implicit relationships, which affects the efficiency of downstream tasks.

[0007] Therefore, we propose a dynamic graph representation table recognition method based on variational autoencoder (VAE) to solve the above problems. Summary of the Invention

[0008] In view of the deficiencies mentioned in the background art of the prior art, the present invention provides a dynamic graph representation table recognition method based on variational autoencoder.

[0009] This invention is achieved through the following technical solution:

[0010] A dynamic graph representation table recognition method based on variational autoencoder, used to read implicit content relationships from complex table images, includes the following steps:

[0011] S1. Preprocessing steps: Extract cell boundaries and text content from the input table image, use improved Canny edge detection combined with extended neighborhood flood fill tracking to generate cell location information pos_i and semantic vector sem_i, fuse them to form node feature vector V_i, and construct an initial graph G=(V,E) based on Euclidean distance threshold, where E represents geometric adjacency relationship;

[0012] S2, VAE encoding steps: Use graph attention network to aggregate node features of the initial graph G, map the feature vector to the latent space Z, generate the mean vector μ_z and standard deviation vector σ_z of the latent distribution, and obtain the latent vector z through reparameterized sampling to capture the uncertainty of table relationships;

[0013] S3, VAE decoding steps: Pairwise concatenation of latent vector z, prediction of the inter-node relationship probability matrix E^ using multilayer perceptron, and calculation of reconstruction loss L_rec and KL divergence loss L_KL based on binary cross-entropy.

[0014] S4. Optimization steps: Introduce a contrastive learning mechanism, generate positive and negative sample pairs by perturbing the latent vector, calculate the InfoNCE contrastive loss L_con, and train the model using the total loss L=L_rec+γ L_con, where β and γ are weight parameters.

[0015] S5. Output steps: Apply threshold filtering to the relation probability matrix E^ to generate a weighted relation graph G_rel, which supports querying and visualization of implicit relations.

[0016] In a preferred embodiment of the present invention: in the preprocessing step, the improved Canny edge detection uses a Gaussian filter with a kernel size of 5×5 and a standard deviation σ=1.0. The gradient magnitude and direction are calculated by the Sobel operator, and a binary edge map is generated by non-maximum suppression and hysteresis edge connection.

[0017] In a preferred embodiment of the present invention: the extended neighborhood flood filling tracking is extended from 8 neighborhoods to 12 neighborhoods, with priority given to horizontal or vertical tracking, the best tracking direction is selected based on the gradient intensity difference, and a rectangular box is fitted by the minimum bounding rectangle algorithm, with the constraint box angle < 5° and the aspect ratio > 0.1.

[0018] In a preferred embodiment of the present invention: the node feature vector V_i=[pos_i;sem_i], where pos_i is a 4-dimensional position vector containing row index, column index, width and height information; sem_i is a 512-dimensional semantic vector, generated by encoding cell text through a pre-trained language model PLM, and V_i has a total dimension of 516.

[0019] In a preferred embodiment of the present invention: the construction of edge E in the initial graph G satisfies E_ij=1 if and only if dist(pos_i,pos_j)<τ, otherwise E_ij=0, where dist is the normalized Euclidean distance and τ is the set threshold.

[0020] In a preferred embodiment of the present invention: in the VAE encoding step, the latent space dimension is set to 128 dimensions, and the standard deviation vector σ_z is ensured to be positive by the Softplus activation function to quantify uncertainties such as the inheritance of blank cells.

[0021] In a preferred embodiment of the present invention, the relation probability prediction in the VAE decoding step adopts the Sigmoid activation function, which maps the multilayer perceptron output to the [0,1] interval, supporting the inference of hierarchical dependencies or causal relationships.

[0022] In a preferred embodiment of the present invention, in the optimization step, the temperature parameter τ of the comparison loss is 0.07 to control the discriminative learning intensity and improve the robustness to noise variation. Positive and negative sample pairs are generated by simulating table variation, and the discriminative loss L_con quantifies the discriminativeness of the latent representation. The weight parameter γ is adjusted according to the training requirements.

[0023] In a preferred embodiment of the present invention, in the output step, the filtering threshold φ is 0.5, and edges with E^_ij>φ are retained to construct a weighted relation graph G_rel.

[0024] In a preferred embodiment of the present invention, the complex table includes tables with multi-layered headers, blank cells, geometric distortions, scanning noise, or irregular layouts, and is suitable for automated analysis of scanned documents, PDFs, or image-based tables, covering scenarios such as financial statements and scientific document analysis.

[0025] The beneficial effects of this invention are:

[0026] 1. This invention captures the distribution characteristics and uncertainties of table relationships through VAE encoding latent space, and the decoding module infers the dynamic relationships between nodes, effectively handling implicit relationships across rows or columns in multi-level headers and blank cells, such as hierarchical dependencies, causal associations, and numerical calculation relationships, and solving the problem of existing technologies ignoring non-local relationships;

[0027] 2. In this invention, the preprocessing module adopts an improved edge detection and extended neighborhood flood filling tracking algorithm to adapt to complex tables with no borders, distortion, and noise interference; the optimization module introduces a contrastive learning mechanism to improve the robustness of the model to interference such as scanning distortion and layout variation, and reduce the risk of overfitting. Detailed Implementation

[0028] A dynamic graph representation-based table recognition method based on variational autoencoder (VAE) is proposed to extract implicit content relationships from complex table images. Employing an end-to-end design, the method converts complex tables into dynamic graph representations and captures the latent relationship space through a VAE to achieve automatic recognition of content relationships. The input is a document image or PDF table data, and the output is a relationship graph that supports querying implicit associations (such as "cell A is the parent of cell B").

[0029] Complex tables, including those with multi-layered headers, blank cells, geometric distortions, scanning noise, or irregular layouts, are suitable for automated analysis of scanned documents, PDFs, or image-based tables, covering scenarios such as financial statements and scientific document analysis.

[0030] The architecture comprises five modules: a preprocessing module extracts cell boundaries and text, processes image data using OCR, and constructs an initial graph G (nodes fuse positional and semantic information, edges are connected based on adjacency relationships), providing standardized input for subsequent modules. The VAE encoding module receives G, aggregates features using a graph neural network, maps them to a latent space Z, captures the mean and uncertainty of relation distributions, generates continuous vectors through sampling, and supports table mutation processing. The VAE decoding module reconstructs graph edges from Z, predicts relation probabilities, calculates reconstruction loss, and ensures accurate dynamic relations. The optimization module introduces contrastive learning, generates positive and negative sample pairs, optimizes the discriminative power of the latent representation, and balances reconstruction, regularization, and contrastive losses to improve robustness. The output module generates a relation graph based on the reconstructed edges, supports visualization and query functions, and integrates results to provide interpretable output.

[0031] Specifically, the following steps are included:

[0032] S1. Input table data from document images, PDFs, or scanned copies via the preprocessing module. Convert to image format (using the rasterize function of open libraries such as PyMuPDF) to uniformly process scanned PDFs or digital PDFs, ensuring compatibility, rather than directly relying on the readable metadata of the PDF electronic document. This input represents the original table source to be read.

[0033] The output is an initial graph G, where V is the set of nodes, each node represents a cell, embedding location and semantic information, and E is the set of edges, representing preliminary adjacency relationships. This output is used as the basis for VAE inference.

[0034] This module performs cell boundary reading and text extraction, feature fusion to enhance representation, and structured data graph construction. It handles irregular layouts (such as multi-level headers and blank cells), with a computational complexity of O(m×n) for a m-row, n-listed cell, and supports parallel processing. This module ensures accurate "reading" and is consistent with subsequent relation parsing.

[0035] The preprocessing module includes the following specific steps:

[0036] 1.1 Cell Extraction: The core objective is to extract cell bounding boxes from an image, handling distorted, noisy, and borderless scenes. This involves using Optical Character Recognition (OCR) technology to detect cell boundaries and extract text content from the input image (or converted PDF image). A custom boundary detection algorithm (see below) is used to identify cell bounding boxes. For each cell, location information (row, column, width, height) and the text string are extracted. This step focuses on image-based processing, ensuring compatibility with scanned documents or images embedded in PDFs.

[0037] A custom boundary detection algorithm is based on the publicly available Canny edge detection algorithm for the cell extraction step. It combines custom floodfill tracking and table optimization rules to achieve precise localization. The algorithm avoids existing deep learning methods (such as TabCellNet's CNN cell detection), focusing on rule-based optimization and adapting to table features (such as line dominance and grid structure).

[0038] Edge extraction: A modified Canny edge detection algorithm is used to generate a binary edge map. The image is smoothed using a Gaussian filter with a kernel size of 5×5 and a standard deviation σ=1.0 (set empirically based on the thickness of the table lines). The gradient magnitude and direction are calculated using the Sobel operator, and non-maximum suppression is performed to preserve local maxima. Hysteresis edges are connected to form continuous edges. The output is a binary edge map E (1 represents an edge, 0 represents no edge).

[0039] Contour tracing (custom floodfill variant): Starting from the initial pixel E_link (e.g., top left corner) of the edge map E, a custom floodfill algorithm is used to trace the closed boundaries of the cells, where the custom rule prioritizes horizontal / vertical tracing (based on the grid characteristics of the table), and a dynamic neighborhood search is used to expand from 8 neighborhoods to 12 neighborhoods (including diagonal and secondary offsets such as (±2,0), (0,±2) and diagonal) to capture the complete cell region, terminating at the flooded pixel EN;

[0040] For the current pixel p=(x,y), the tracking direction d is:

[0041] d=argmax_{dir}(gradient(E_link,dir)),dir∈{horizontal,vertical}

[0042] gradient(E_link,dir) is the edge intensity difference, calculated based on the edge map E and the direction dir. It is defined as the gradient value in the dir direction and is used to quantify the local edge continuity. It provides a value to select the best tracking direction and ensures that the algorithm prioritizes the horizontal or vertical lines of the table.

[0043] If dir=horizontal, then ∑{k=-r to r} E_link(x,y+k)-E_link(x,yk) (r=3, empirical radius). The intensity difference is calculated from the pixel intensity, ensuring tracking is biased towards a table structure and supporting complex boundary detection.

[0044] Where K represents the summation index, which is used to iterate through the summation range. The value changes from -r to r (when r=3, k=-3,-2,-1,0,1,2,3).

[0045] E_link(x,y+k) represents the positive offset pixel value, which is the pixel value of the edge map pixel E_link at coordinates (x,y+k) (0 or 1, where 0 represents non-edge and 1 represents edge).

[0046] E_link(x,yk) represents the negative offset pixel value. It represents the pixel value (0 or 1, where 0 indicates a non-edge and 1 indicates an edge) of the edge map pixel at coordinates (x,yk).

[0047] r defines the expanded window size, supports secondary offsets (such as ±2), and matches a 12-neighborhood of a pixel.

[0048] `horizontal` indicates the horizontal direction, expanding the search along the x-axis (left and right) to capture horizontal lines or boundaries in the table. This option helps the algorithm prioritize processing neighboring pixels on the left and right, ensuring that horizontal edges (such as table rows) are effectively tracked;

[0049] "Vertical" indicates the vertical direction, extending the search along the y-axis (up and down) to capture vertical lines or boundaries in the table. This option helps the algorithm prioritize processing neighboring pixels above and below, ensuring that vertical edges (such as table columns) are effectively tracked.

[0050] Define d as the preferred direction (0: horizontal, 1: vertical), and calculate based on local gradients from existing visual recognition algorithms to ensure straight lines are prioritized in the table. Output a list of contours C, where each C is a sequence of points. Flood fill is a public variant that customizes the bias towards the table geometry and expands the neighborhood to capture a wider range of potential boundaries (such as lines across cells), avoiding circular contours. Edge map E is the pixel intensity.

[0051] The minimum bounding rectangle algorithm (existing) is used to perform table optimization on the fitted rectangle of the contour C. Custom constraints can be set: frame angle < 5° (to ensure a rectangle), aspect ratio > 0.1 (to avoid thin lines).

[0052] 1.2 Feature Fusion: Based on the extracted cell information, an embedding vector is generated for each cell. The positional information is encoded into a 4-dimensional vector, and the text uses a pre-trained language model (PLM, non-BERT or RoBERTa variant) to generate semantic vectors. These vectors are then concatenated to form a node representation. This step captures the geometric layout of the positional information and the meaning of the content semantically, avoiding unimodal dependence. The formula is V_i=[pos_i;sem_i];

[0053] Where pos_i represents the position vector (defined in 4 dimensions: [row_i, col_i, width_i, height_i]). Calculated through boundary detection, it is defined as geometric coordinates (row_i is the row index, col_i is the column index, and width_i and height_i are the pixel dimensions), and the geometric coordinates are extracted.

[0054] `sem_i` represents the semantic vector (defined as 512 dimensions). PLM encodes the text and defines it as content embedding (average pooling hidden states) to capture the meaning of the text. This parameter provides content semantics, avoiding the limitations of relying solely on position; concatenating `pos_i` and `sem_i` is defined as dimension concatenation; `V_i` is the vector of the i-th node, defined as 516 (4+512) dimensions. The cell-level comprehensive representation provides the basis for VAE input. This parameter is the final embedding of node i, fusing geometric and semantic information to ensure that downstream modules can capture multi-dimensional features.

[0055] 1.3 Graph Construction: A graph G(V,E) is created based on the extracted cells. Node V corresponds to embedded cells, and edges E connect adjacent cells using an Euclidean distance threshold τ (considering only geometric adjacency, not semantic similarity or key-value relationships). An excessively large threshold leads to too many edges and a dense graph, resulting in computational complexity; an excessively small threshold leads to too few edges and a sparse graph, potentially losing important relationships. Therefore, based on experience, τ is generally set to 0.5. This step ensures graph sparsity, consistent with VAE input requirements. Node V is embedded, and edges E connect geometric adjacencies using a distance threshold, with the formula: E_ij=1 if dist(pos_i,pos_j)<τ, else 0;

[0056] Where E_ij represents the edge label (0 / 1), and is defined as an adjacency indicator by determining whether there is an edge between node i and node j;

[0057] dist(pos_i,pos_j) represents the Euclidean distance, which calculates the distance between the position vectors of two nodes using pos, defining geometric similarity. Normalized to the interval [0,1], where 0 indicates that two cells overlap and 1 indicates the maximum possible distance. The normalized value reflects the physical proximity of the cells, captures local layout features, and avoids global computation overhead.

[0058] S2, the VAE encoding module, is one of the core components of this invention. It receives the initial graph G generated by the preprocessing module, maps it to the latent space Z, and captures the mean and uncertainty of the relation distribution, thereby achieving "intelligent reading" of the table content—that is, inferring implicit relationships (such as cross-row dependencies or blank inheritance) by aggregating explicit content (such as positional and semantic embeddings). This module aggregates node features through a graph attention network (GAT) to generate a compressed representation, supporting the processing of implicit relationships in complex tables. Based on the preprocessing output: taking G as input, this module generates latent parameters μ_z and σ_z, as well as a sampling vector z, providing the foundation for the decoding module. In table recognition, this module bridges the gap between basic "reading" (explicit content extracted by preprocessing) and "intelligent parsing" (capturing uncertainty in the latent space), improving the accuracy of recognizing complex structures and avoiding the limitations of existing methods (such as simple GNNs) that ignore variations.

[0059] The specific steps are as follows: the module takes an initial graph G=(V,E) as input, where V is the node embedding and E is the edge set. It outputs latent space parameters μ_z and σ_z, as well as a sampling vector z, to achieve graph aggregation, latent parameter generation, and sampling.

[0060] The specific steps in the VAE encoding module are as follows: A Graph Attention Network (GAT) encoder is used to aggregate neighborhood features. For each node v, an updated representation h_v = GAT(V,E) is computed, fusing local and global information. V is aggregated based on edges E of G, ensuring the capture of multi-hop relationships. The latent parameters are calculated from the aggregated representation, generating the mean μ_z and standard deviation σ_z. This step bridges the aggregation to sampling, providing distribution parameters. Sampling uses reparameterization to generate z. Continuous vectors are sampled from the parameters to avoid gradient problems.

[0061] The formula is A linear transformation maps the 516-dimensional node representation to a 128-dimensional latent space.

[0062] in This represents the latent mean vector (dimension h=128), sourced from the GNN output, defined as the center of the relation distribution, capturing the average latent associations. This parameter quantifies the distribution mean, aiding in handling deterministic relationships;

[0063] in The latent standard deviation vector (dimension h=128) originates from the GNN output and is defined as the distribution variance, quantifying uncertainty (such as the effect of blanks). This parameter captures the variation.

[0064] in This represents the GNN encoder function, derived from a variant of the GAT implementation, and is defined as an aggregation function. G is the output parameter. This function captures neighborhood information and supports complex graphs;

[0065] G represents the graph structure (V,E), providing the encoding basis; z represents the sampled latent vector (dimension h=128), which is derived from reparameterization and defined as a random representation to ensure continuity. This parameter is used to decode the input and avoid gradient vanishing. This represents a sampling operation, defined as sampling from a distribution; N represents a normal distribution; where express The variance; I represents the identity matrix, defined as covariance scaling to ensure independent dimensions.

[0066] S3, the VAE decoding module, receives the latent space vector z generated by the VAE encoding module, reconstructs graph edges from it, and predicts the dynamic relationships between nodes. This module ensures that the reconstructed edges accurately reflect the implicit content associations (such as hierarchical dependencies or causal relationships) of the table through nonlinear mapping and loss optimization, achieving "intelligent reading" of the table content, i.e., inferring implicit relationships from the compressed representation. This module is based on the encoded output: using z as input, it generates predicted edges and provides reconstruction loss to the optimization module. The role of this module in table recognition is to bridge the latent space to relationship prediction, improve the ability to "read" implicit content, and avoid the limitations of existing methods (such as simple MLP) that ignore distribution uncertainties.

[0067] The specific steps are as follows: take the latent space vector z obtained by the VAE encoding module as input, output the predicted edge probability matrix E^, and the reconstruction loss L_rec;

[0068] Edge probability prediction specifically involves concatenating the latent vectors z_i and z_j corresponding to nodes i and j in pairs to generate a concatenated vector with dimension 2h=256. (h=128 is the dimension of the latent space). As input to a multilayer perceptron (MLP), the concatenated vector is nonlinearly mapped using the MLP, then activated by a sigmoid activation function σ. This compresses the output value to the [0,1] interval, yielding the predicted edge probabilities E^_ij for nodes i and j, as shown in the formula. ;

[0069] Where E^_ij represents the predicted edge probability (range [0,1]), the confidence level of the relationship between nodes i and j. This parameter quantifies the strength of the relationship and helps identify implicit associations, such as the prediction of hierarchical dependencies;

[0070] σ represents the Sigmoid activation function, which mathematically maps the MLP output to the [0,1] probability interval, ensuring output normalization. This function compresses the value range and avoids extreme predictions;

[0071] MLP stands for Multi-Layer Perceptron. The concatenated vector (dimension 2h=256) is generated by z-sampling to obtain the latent vectors of nodes i and j, which are then concatenated into pairs to provide the input basis for the MLP.

[0072] The loss calculation specifically involves calculating the reconstruction loss and combining it with KL divergence regularization to ensure distribution consistency. This step bridges the prediction and optimization process, providing feedback signals.

[0073] In the VAE decoding step, the general binary cross-entropy (BCE) loss function is used to quantify the difference between the predicted edge probability E^_ij and the true edge label E_ij, thus obtaining the reconstruction loss. The specific formula is as follows

[0074]

[0075] in, The reconstruction loss is represented by the binary cross-entropy, which quantifies the difference between the predicted and the true edge. This loss guides model optimization.

[0076] BCE indicates the selection of the generalized binary cross-entropy loss function.

[0077] The true edge label (0 / 1) is represented by the data from the preprocessing module graph G, which is defined as a supervision signal indicating whether ij is connected.

[0078] S4. The optimization module receives the reconstruction loss L_rec from the VAE decoding module and optimizes the latent space representation through a contrastive learning mechanism, improving the model's robustness and generalization ability to complex table relationships. This module, based on the decoding output, generates a contrastive loss L_con using L_rec as input, which, along with the total loss L, is fed back to the VAE encoding and decoding modules to adjust parameters.

[0079] The role of this module in table recognition is to enhance the model's robustness to complex tables through contrastive learning (such as handling errors caused by noise or variable layouts), improve the accuracy of "reading" implicit relationships (such as more reliably inferring inheritance or hierarchical dependencies of blank cells), and avoid the overfitting limitations of existing methods (such as simple gradient descent).

[0080] This module serves as a feedback loop for the overall invention, ensuring stable training of the VAE dynamic graph representation, supporting closed-loop optimization from basic "reading" to "intelligent parsing", and enabling the model to "learn better" through "positive and negative sample comparison training", ultimately improving the practical value of table recognition.

[0081] The specific steps are as follows: the optimization module takes the reconstruction loss L_rec and the latent space vector z sampled by the VAE encoding module as input, and outputs the contrastive loss L_con and the total loss L to achieve contrastive learning and total loss optimization. This solves the problem of overfitting in existing methods (such as simple gradient descent), enhances the model's adaptability to complex table scenarios such as noise and mutated layouts, and improves the accuracy of "reading" implicit relationships such as blank cell inheritance and hierarchical dependence.

[0082] Generate positive and negative sample pairs from the latent vector z, and simulate table distortion and layout mutation by adding noise to the latent vector z_i of node i, thereby generating positive sample vectors from the same source mutation. _i, and simultaneously select the heterogeneous potential vector as the negative sample vector. _k, this step provides the data foundation for the comparative loss calculation, directly supporting the improvement of the model's "reading" robustness;

[0083] The contrastive loss L_con, which measures the discriminative power between positive and negative samples, is calculated using the following formula:

[0084]

[0085] in, This represents the negative logarithmic operation, maximizing the loss form of the positive sample probability. This operation transforms the ratio into a loss value, ensuring the direction of optimization;

[0086] exp represents the exponential function, softmax numerator / denominator, which softens the similarity. This function avoids hard classification and improves gradient flow.

[0087] This represents a similarity function that measures the similarity between a target vector and positive sample vectors.

[0088] z_i represents the latent vector of node i. _i represents a positive sample vector. _k negative sample vectors.

[0089] τ represents the temperature parameter (0.07), which can be customized and is a softening factor that controls the contrast intensity. This parameter adjusts the loss sensitivity;

[0090] The contrastive loss is calculated by plotting L_con based on positive and negative samples, quantifying the discriminative power of the latent representation. This step bridges the sample to optimization, providing feedback signals and improving the inference of latent relationships.

[0091] Combined with reconstruction loss The comparison loss L_con generates the total loss L, and the formula is as follows: γ is the contrast weight, which can be dynamically adjusted according to the performance feedback during model training (such as loss convergence and validation set effect) to balance reconstruction accuracy and contrast learning effect; the total loss L is output as the optimization objective to the gradient update stage, and fed back to the VAE encoding / decoding module to complete parameter iteration. Through the closed-loop process from sample generation to loss calculation to total loss integration, stable training convergence is ensured.

[0092] The total loss L is output as the optimization objective to the gradient update stage. This step self-consistently integrates the loss and feeds it back to the encoding / decoding module to ensure training convergence. A general divergence regularization method can be used to calculate the difference between the latent distribution and the standard normal distribution, which is used to regularize the latent space and prevent overfitting or distribution collapse.

[0093] S5. The output module receives the predicted edge probability matrix E^ from the VAE decoding module and generates the final relation graph, supporting downstream tasks such as semantic querying and inference. This module ensures the interpretable output of implicit content relationships (such as hierarchical dependencies or causal associations) through graph construction and visualization, achieving the final "intelligent reading" of table content and transforming the inferred relationships into queryable results. This module is based on the decoding output and optimization module: using E^ as input, it generates the relation graph G_rel and provides a query interface;

[0094] In table recognition, this module transforms the implicit relationships inferred by VAE into practical outputs (such as knowledge graphs), improving the interpretability and application value of the recognition results and avoiding the limitations of existing methods (such as simple edge prediction) that lack downstream support. The module's role in the overall invention is as a endpoint module, presenting the process results from basic "reading" to "intelligent parsing."

[0095] The specific steps are as follows: the output module constructs a weighted relation graph G_rel using the predicted edge probability matrix E^ and the initial nodes V generated in the preprocessing stage. E^_ij (the original confidence level output by the VAE decoding module) is directly used as the initial weight of the edges, achieving a connection from the decoding result to the output graph and ensuring accurate transmission of relation strength. Simultaneously, high-confidence relations are retained through threshold filtering, with the filtering formula being... ;

[0096] Where φ=0.5 is a custom preset filtering threshold. The If else threshold judgment logic retains only edges with a predicted probability greater than the threshold, ensuring that the graph contains only high-confidence relationships.

[0097] Meanwhile, based on threshold filtering, the retained edges are classified into relation types according to probability values ​​(e.g., probability E^_ij > 0.8 is defined as "strong association"), further refining the graph structure; the weight range of the filtered relation edges is [0,1], representing the final relation strength between nodes i and j, providing the core data foundation for subsequent queries and connecting the graph construction and query functions;

[0098] Where E^_ij represents the predicted probability, obtained from the VAE decoding module, and is defined as the original confidence level, which is used as the filtering input; If else is defined as a threshold judgment to ensure that only high-confidence edges are retained.

[0099] The final output graph uses a visual format such as JSON or graphics to intuitively present the implicit relationships between the table nodes;

[0100] This provides the core foundation for querying, and for a given node i, it involves traversing the graph to check... The values ​​are selected and filtered to meet the criteria. The nodes form a query result set. It supports querying implicit relationships such as "the parent relationship of node i".

[0101] The following are detailed explanations regarding missing credit entries for debit accounts (accounting logic), missing cumulative data (to be calculated), and missing remarks information (not required).

[0102]

[0103] I. Basic Background and Business Logic

[0104] This example is based on an accounting chart, and the core business logic includes:

[0105] 1. Accounting rules: The credit side of a debit account is empty; missing cumulative data needs to be calculated; and remarks are optional.

[0106] 2. Coding Rules: Hierarchical subject codes follow inheritance, and sub-subject codes adopt the coding standards of parent subjects;

[0107] 3. Numerical relationships: Gross profit = Operating revenue - Operating cost. The sum of the values ​​of sub-items equals the parent item (e.g., cumulative main business revenue = cumulative product A + cumulative product B).

[0108] 4. Inheritance Relationship: Blank fields such as budget completion rate and remarks indicate that they are to be filled or inherited from the parent attribute;

[0109] 5. Verification relationship: Operating cost = direct materials + direct labor (in the example 1,500 = 900 + 600).

[0110] II. Cell Extraction and Key Node Filtering

[0111] 1. Edge detection and tracking

[0112] An improved Canny edge detection method is used to extract cell boundaries: Gaussian filter σ=1.0, kernel size 5×5, outputting a binary edge map E(x,y)∈{0,1};

[0113] Flood filling tracking: Priority direction is selected based on gradient strength, using the following formula:

[0114] d=argmaxdir∈{horizontal,vertical} gradient_strength(E,p,dir);

[0115] For example, in the "Operating Revenue" cell p=(120,45):

[0116] The horizontal intensity (gradient intensity in the horizontal direction) is Σ|E(120,48)-E(120,42)|+...=0.85;

[0117] The vertical intensity (gradient intensity in the vertical direction) is Σ|E(123,45)-E(117,45)|+...=0.62;

[0118] By prioritizing horizontal tracking, 63 cells were identified, and 10 key nodes were selected for subsequent VAE processing.

[0119] 2. List of key nodes

[0120]

[0121] III. Feature Fusion and Initial Graph Construction

[0122] 1. Node Feature Vector Generation

[0123] Node feature vector V_i=[pos_i;sem_i]∈R516

[0124] pos_i∈R4, the row, column, width, and height coordinates of the cell;

[0125] sem_i∈R512, semantic features extracted by a pre-trained language model (PLM);

[0126] Example calculation:

[0127] C1 (Operating Revenue): pos_1=[2,0,80,25];

[0128] sem_1=[0.23,-0.45,0.78,...,0.12]∈R512;

[0129] Finally, V_1=[2,0,80,25,0.23,-0.45,0.78,...,0.12]∈R516.

[0130] C2 (25 million): pos_2=[2,3,60,25]∈R4;

[0131] sem_2=[0.89,0.12,-0.34,...,0.67];

[0132] Finally, V_2=[2,3,60,25,0.89,0.12,-0.34,...,0.67]∈R516.

[0133] 2. Initial Adjacency Calculation

[0134] Adjacency determination formula: E_ij=1 if dist(pos_i,pos_j)<τ,else 0, where, distance calculation: Euclidean distance (only row and column coordinates are taken), normalized to the [0,1] interval; threshold τ=0.35 (calculated from the average width and height of the cell / the maximum dimension).

[0135] Example calculation:

[0136] C1 (Operating Revenue) Position: [2,0] → C2 (25 million) Position: [2,3];

[0137] dist(pos_1,pos_2)= =3;

[0138] Normalized distance: 3 / max_dimension = 3 / 8 = 0.375;

[0139] τ=min(avg_width,avg_height) / max_dimension=70 / 8=0.35;

[0140] Since 0.375 > 0.35, E12 = 0 (not directly adjacent);

[0141] However, C1 is adjacent to other nodes:

[0142] C1(Operating Revenue)[2,0]→C3(Main Operating Revenue)[3,0]:dist= =1;

[0143] Normalized distance: 1 / 8 = 0.125 < 0.35 → E13 = 1;

[0144] C1(Operating Revenue)[2,0]→C9(Operating Costs)[6,0]:dist= =4;

[0145] Normalized distance: 4 / 8 = 0.5 > 0.35 → E19 = 0;

[0146] The adjacency relation of C2(2,500):

[0147] C2[2,3]→C4(2,000)[3,3]:dist= =1;

[0148] Normalized distance: 1 / 8 = 0.125 < 0.35 → E24 = 1;

[0149] C2[2,3]→C6(1,200)[5,3]:dist= =3;

[0150] Normalized distance: 3 / 8 = 0.375 > 0.35 → E26 = 0.

[0151] Complete initial adjacency relation:

[0152] E13=1: Operating Revenue Main operating revenue (same column, adjacent row);

[0153] E24=1:2,500 2,000 (same column, adjacent rows);

[0154] E35=1: Main operating revenue Product A (hierarchical relationship);

[0155] E37=1: Main operating revenue Product B (hierarchical relationship);

[0156] E57=1: Product A Product B (same level);

[0157] E68=1: 1,200 800 (same column, adjacent rows);

[0158] E9,10=1: Operating Costs 1,500 (adjacent to the same line);

[0159] Total: 8 initial adjacent edges.

[0160] IV. VAE Encoding and Decoding Calculation

[0161] 1. GAT graph aggregation and latent space sampling

[0162] Input an initial graph G(V,E) (V is a 516-dimensional node embedding, E is the edge set), and output the latent distribution parameters μ_z, σ_z∈R through a GNN encoder.128 The linear transformation maps the 516-node features to a 128-dimensional latent space;

[0163] in, : The latent mean vector, with a dimension of 128, is calculated through a linear layer Linearμ(516→128);

[0164] : The latent standard deviation vector, with a dimension of 128, is calculated through a linear layer Linearσ(516→128);

[0165] Specific calculation results:

[0166] =GNNEncμ(G)=[0.23,-0.45,0.12,...,0.78]∈R 128

[0167] =GNNEncσ(G)=[0.18,0.32,0.09,...,0.41]∈R 128

[0168] ;

[0169] 128 ;

[0170] 2. Edge probability prediction and reconstruction loss calculation

[0171] The edge probabilities E^_ij are predicted by pairwise concatenation of latent vectors and MLP mapping, and the binary cross-entropy (BCE) reconstruction loss is calculated:

[0172] Example of a single node pair (C1-C2):

[0173] The concatenated vector [z_1;z_2] = [0.169, 0.084, ..., 0.745, 0.231, ..., 0.887] ∈ R 256 The predicted value E^_12 = 0.95 (the relationship probability predicted by VAE), and the actual value E^_12 = 0 (C1-C2 are not directly adjacent in the initial graph).

[0174] BCE loss: BCE(E^_12,E_12)=-[E_12×log(E^_12)+(1-E_12)×log(1-E^_12)]

[0175] =-[0×log(0.95)+1×log(1-0.95)]

[0176] =-[0+log(0.05)]

[0177] =-log(0.05)=2.996

[0178] Total reconstruction loss:

[0179] Summing the matrix for all 100 elements (a 10×10 matrix):

[0180] L_rec=Σ 10 _i=1Σj=1 10 BCE(E^_ij,E_ij)

[0181] =BCE(E^_11,E_11)+BCE(E^_12,E_12)+...+BCE(E^_10,E_10)

[0182] =0+2.996+0.073+...+0

[0183] =0.156 (Average loss of all 100 elements)

[0184] V. Comparative Learning and Total Loss Optimization

[0185] 1. Comparative loss calculation

[0186] Contrast loss formula:

[0187]

[0188] The temperature parameter τ = 0.1.

[0189] Example calculation (using z_1 as an example):

[0190] Positive sample similarity sim(z_1, _1) = 128-dimensional vector dot product / (norm 1 × norm 2) = 0.94;

[0191] Negative sample similarity sim(z_1, _2)=0.23,sim(z_1, =0.19 for _3), ..., mean similarity of negative samples ≈ 0.21;

[0192] Temperature parameter: τ=0.1;

[0193] Numeric: exp(0.94 / 0.1) = exp(9.4) = 12,088;

[0194] Denominator: exp(9.4)+exp(2.3)+exp(1.9)+...=12,088+10+7+...=12,234;

[0195] L_con1=-log(12,088 / 12,234)=-log(0.988)=0.012;

[0196] Continue calculating the contrast loss from z_2 to z_10:

[0197] L_con2=-log(8,955 / 9,187)=-log(0.975)=0.025;

[0198] L_con3=0.034;L_con4=0.041;L_con5=0.058;L_con6=0.067;

[0199] L_con7=0.089;L_con8=0.134;L_con9=0.156;L_con10=0.254;

[0200] Average contrast loss for the entire sample: L_con=(1 / 10)×Σ 10 _i=1 L_coni=(1 / 10)×(0.012+0.025+0.034+0.041+0.058+0.067+0.089+0.134+0.156+0.254)=(1 / 10)×0.870=0.087.

[0201] 2. Overall Loss Consolidation

[0202] Total loss formula: (With a comparison weight γ=0.1), the final result is: L=0.156+0.1×0.087=0.1647.

[0203] VI. Output Module: Graph Construction and Intelligent Query

[0204] 1. Relationship filtering and classification

[0205] Set the filtering threshold φ=0.5 to filter the 100 original predicted edges:

[0206] Retain high-confidence edges: 32 (E^_ij>0.5), filter weak relations: 68 (E^_ij≤0.5).

[0207] Relationship Classification:

[0208] Extremely strong relationships (>0.85): 12, such as C1-C2 (0.95), C3-C4 (0.96), C5-C6 (0.94);

[0209] Strong ties (0.65-0.84): 8, such as C1-C9 (0.78), C4-C6 (0.82);

[0210] Medium-strength relationships (0.5-0.64): 12, such as C1-C5 (0.58), C2-C8 (0.52);

[0211] 2. Intelligent query results

[0212] Query the relevant nodes for "Operating Revenue (C1)" and scan. j=1,2,...,10

[0213] For nodes where Erel_12=0.95>0, Erel_13=0.93>0, and Erel_15=0.58>0, the result is Q1={C2(25 million), C3(main operating revenue), C5(product A), C9(operating cost)};

[0214] Query the relevant nodes for "Product A (C5)":

[0215] Q5 = {C3 (main operating revenue), C6 (12 million)} (strong relationship);

[0216] Relationship type: C5-C3 (hierarchical relationship 0.91), C5-C6 (numerical relationship 0.94).

[0217] The final output is a visual relationship graph, which intuitively presents the implicit relationships such as hierarchy, values, and connections between cells, as shown in the following example:

[0218]

[0219] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. A dynamic graph representation table recognition method based on variational autoencoder, used to read implicit content relationships from complex table images, characterized in that, Includes the following steps: S1. Preprocessing steps: Extract cell boundaries and text content from the input table image, use improved Canny edge detection combined with extended neighborhood flood fill tracking to generate cell location information pos_i and semantic vector sem_i, fuse them to form node feature vector V_i, and construct an initial graph G=(V,E) based on Euclidean distance threshold, where E represents geometric adjacency relationship; S2, VAE encoding steps: Use graph attention network to aggregate node features of the initial graph G, map the feature vector to the latent space Z, generate the mean vector μ_z and standard deviation vector σ_z of the latent distribution, and obtain the latent vector z through reparameterized sampling to capture the uncertainty of table relationships; S3, VAE decoding steps: Pairwise concatenation of latent vector z, prediction of the inter-node relationship probability matrix E^ using multilayer perceptron, and calculation of reconstruction loss L_rec and KL divergence loss L_KL based on binary cross-entropy. S4. Optimization steps: Introduce a contrastive learning mechanism, generate positive and negative sample pairs by perturbing the latent vector, calculate the InfoNCE contrastive loss L_con, and train the model using the total loss L=L_rec+γ L_con, where β and γ are weight parameters. S5. Output steps: Apply threshold filtering to the relation probability matrix E^ to generate a weighted relation graph G_rel, which supports querying and visualization of implicit relations.

2. The dynamic graph representation table recognition method based on variational autoencoder according to claim 1, characterized in that: In the preprocessing step, the improved Canny edge detection uses a Gaussian filter with a kernel size of 5×5 and a standard deviation of σ=1.

0. The gradient magnitude and direction are calculated by the Sobel operator, and a binary edge map is generated by non-maximum suppression and hysteresis edge connection.

3. The dynamic graph representation table recognition method based on variational autoencoder according to claim 2, characterized in that: The extended neighborhood flood-filled tracking expands from 8 neighborhoods to 12 neighborhoods, prioritizes horizontal or vertical tracking, selects the optimal tracking direction based on gradient intensity difference, and fits a rectangular box using the minimum bounding rectangle algorithm, with the constraint box angle < 5° and aspect ratio > 0.

1.

4. The dynamic graph representation table recognition method based on variational autoencoder according to claim 2, characterized in that: The node feature vector V_i=[pos_i;sem_i], where pos_i is a 4-dimensional position vector containing row index, column index, width and height information; sem_i is a 512-dimensional semantic vector, generated by encoding cell text through a pre-trained language model PLM, and V_i has a total dimension of 516.

5. The dynamic graph representation table recognition method based on variational autoencoder according to claim 2, characterized in that: The construction of edge E in the initial graph G satisfies E_ij=1 if and only if dist (pos_i,pos_j)<τ, otherwise E_ij=0, where dist is the normalized Euclidean distance and τ is the set threshold.

6. The dynamic graph representation table recognition method based on variational autoencoder according to claim 1, characterized in that: In the VAE encoding step, the latent space dimension is set to 128 dimensions, and the standard deviation vector σ_z is ensured to be positive through the Softplus activation function to quantify uncertainties such as the inheritance of blank cells.

7. The dynamic graph representation table recognition method based on variational autoencoder according to claim 6, characterized in that: The relation probability prediction in the VAE decoding step uses the Sigmoid activation function to map the multilayer perceptron output to the [0,1] interval, supporting the inference of hierarchical dependencies or causal relationships.

8. The dynamic graph representation table recognition method based on variational autoencoder according to claim 1, characterized in that: In the optimization step, the temperature parameter τ=0.07 of the comparison loss is used to generate positive and negative sample pairs by simulating table mutation, the comparison loss L_con quantifies the discriminativeness of the latent representation, and the weight parameter γ is adjusted according to the training requirements.

9. The dynamic graph representation table recognition method based on variational autoencoder according to claim 1, characterized in that: In the output step, the filtering threshold φ is 0.5, and edges with E^_ij > φ are retained to construct a weighted relation graph G_rel.

10. The dynamic graph representation table recognition method based on any one of claims 1-9, characterized in that: The complex tables include tables with multi-layered headers, blank cells, geometric distortions, scanning noise, or irregular layouts. They are suitable for automated analysis of scanned documents, PDFs, or image-based tables, covering scenarios such as financial statements and scientific document analysis.

Images