A table region positioning correction method based on image edge detection

Abstract

The present application relates to the technical field of scanning image processing, in particular to a table region positioning correction method based on image edge detection. The specific implementation steps include: obtaining a financial report original image, constructing a cooperative feedback link based on an instance segmentation network and an edge detection algorithm, extracting a semantic mask through the instance segmentation network, then using the edge detection algorithm for pattern recognition, refining the geometric structure, forming a cooperative feedback through an attention mechanism, and accurately identifying the table; establishing a table structure topology graph, and performing graph matching with a typical grid graph selected from a database, calculating pose correction parameters, and correcting the image; generating a multi-channel repair guide graph containing lines, content and defect positions, guiding the table defect repair of the table repair network.

Description

Technical Field

[0001] This invention relates to the field of scanned image processing technology, specifically to a table region localization and correction method based on image edge detection. Background Technology

[0002] With the global digitalization wave, the financial industry urgently needs to transform the vital information contained in massive amounts of paper documents into structured data that computers can understand and process. In core business processes such as bank loan approval, insurance claims, and corporate financial and tax automation, systems process tens of thousands of scanned financial statements, bank statements, and other documents daily. The tables in these documents are the carriers of critical data; therefore, accurately identifying the structure of these tables and extracting their content is a core prerequisite for achieving business process automation, big data risk control, and business intelligence decision-making.

[0003] However, in actual digital acquisition, physical defects and acquisition noise cause severe degradation of the visual pattern of the table, posing a serious challenge to automated recognition. Broken lines disrupt the continuity of the table pattern, leading to ambiguity in OCR recognition; missing small frames cause the loss of structural information, making it impossible to reason about the complete layout of the layout; document tilt distortion fundamentally challenges pose localization and pattern matching algorithms.

[0004] Existing methods are limited to the direct analysis of visible pixel features and lack the ability to infer and complete the inherent grid-like structural patterns of tables. This makes it difficult for existing methods to reliably identify the complete topological relationships and correct geometric orientation of tables from incomplete visual information, thus introducing errors and risks into automated data extraction processes.

[0005] To address this, a table region localization and correction method based on image edge detection is proposed. Summary of the Invention

[0006] The purpose of this invention is to provide a table region localization and correction method based on image edge detection to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution: a table region localization and correction method based on image edge detection, the specific implementation method including:

[0008] Edge detection algorithm for obtaining raw images of financial statements;

[0009] The original image is subjected to tabular semantic recognition by an instance segmentation network to generate a basic mask; within the range defined by the basic mask, geometric pattern recognition is performed to generate a geometric structure skeleton; the geometric structure skeleton is fed back to the instance segmentation network; a collaborative feedback link based on the instance segmentation network and the edge detection algorithm is formed, and the cell structure mask and edge structure information are iteratively converged to generate the cell structure mask and edge structure information.

[0010] A table structure topology graph is established based on cell structure mask and edge structure information; the table structure topology graph is then matched with the highest-fitting mesh graph in the financial statement database to calculate the attitude correction parameters.

[0011] Based on the posture correction parameters, geometric correction is performed on the original image and the structural mask to generate a posture-corrected image and a corrected mask.

[0012] A multi-channel repair guide map is generated based on the corrected mask and edge structure information; a table repair network is trained using a financial statement database; based on the multi-channel repair guide map, the table repair network is used to perform table repair on the pose-corrected image to generate the final corrected image.

[0013] Preferably, the collaborative feedback loop based on the instance segmentation network and the edge detection algorithm specifically comprises:

[0014] The original image is input into a Mask R-CNN-based instance segmentation network for initial forward inference. The Mask R-CNN-based instance segmentation network identifies and generates a base mask by understanding the contextual semantics of the original image. Based on the base mask, an edge detection algorithm is used to extract structural features from the original image to generate a geometric skeleton. The geometric skeleton is encoded into a spatial attention map using an attention encoder. The spatial attention map is used as a feedback signal and fused with the original image to generate the guidance parameters required for the next iteration, forming a collaborative feedback loop. The collaborative feedback loop iterates until the geometric skeleton converges, generating cell structure masks and edge structure information.

[0015] Preferably, the specific implementation process of generating a geometric structure skeleton by performing geometric pattern recognition on the original image based on a base mask and using an edge detection algorithm includes:

[0016] The image region is defined based on the base mask, and the Canny operator is used to extract candidate edge pixels that constitute the table outline within the defined region to generate an edge feature image.

[0017] The Hough transform is used to extract linear structures from the edge feature image, and all extracted linear structures are integrated to form the geometric skeleton.

[0018] Preferably, the specific implementation process of establishing the table structure topology graph based on the cell structure mask and edge structure information includes:

[0019] Instance analysis is performed on the cell structure mask to identify the cell structure mask boundary and geometric center point, and abstract them into topological graph nodes representing cell instances. The positioning coordinates of cell instances in the image are recorded in the topological graph nodes. Based on edge structure information, the connection relationship between the generated nodes is identified, and the table line segments connecting two adjacent nodes are extracted and abstracted into edges of the topological graph. All identified nodes and edges representing topological relationships together constitute the table structure topological graph.

[0020] Preferably, the step of performing graph matching between the table structure topology graph and the highest-fitting mesh graph in the financial statement database to calculate the attitude correction parameters specifically includes:

[0021] Collect various types of financial statements, classify and store them according to type, and establish a financial statement database;

[0022] Pattern analysis is performed on the table structure topology graph to extract key topological features, including the number of rows and columns and the relative spacing between cells. Based on these key topological features, the financial statement database is identified, and a typical financial statement grid graph is extracted using a maximum matching algorithm. A global optimization algorithm is then used to perform graph matching between the table structure topology graph and the typical financial statement grid graph, iteratively solving for an affine transformation matrix that minimizes the positional deviation of all corresponding nodes and the angular deviation of corresponding edges between the two graphs, which serves as the attitude correction parameter.

[0023] Preferably, the step of generating a multi-channel repair guidance map based on the corrected mask and edge structure information specifically involves:

[0024] Based on the edge structure information, a binary mask reflecting the complete table frame is generated as the line structure channel; a logical union operation is performed on all cell masks in the corrected mask to generate a binary mask that identifies all non-edge regions as the content region channel; the line structure channel is compared and analyzed with the pose-corrected image to identify specific areas with missing pixels and broken lines, and a binary mask marking the repair positions is generated as the defect marking channel; the line structure channel, content region channel, and defect marking channel are combined to form a multi-channel repair guidance map.

[0025] Preferably, the process of training the table repair network using a financial statement database includes:

[0026] Extract complete financial statements from the financial statement database; process the complete financial statements into paired training samples; randomly incomplete the complete financial statements to generate incomplete table images; extract line structure channels, content area channels, and defect marker channels based on the complete financial statements; generate a sample repair guide map by combining the three channels; construct paired training samples by combining the complete financial statements, incomplete table images, and sample repair guide maps.

[0027] The incomplete table image and the sample repair guide image are concatenated along the channel dimension and input into the table repair network to generate a predicted repair image. The overall error between the predicted repair image and the complete financial statement is calculated using an error function. Based on the overall error, the internal weight parameters of the table repair network are iteratively updated using a gradient descent-based backpropagation algorithm until the model converges, generating the trained table repair network.

[0028] Compared with the prior art, the beneficial effects of the present invention are:

[0029] 1. This invention designs a collaborative feedback loop, combining the powerful semantic understanding capability of deep learning models to identify cells with the precise geometric analysis capability of edge detection algorithms to extract linear skeletons. Through iterative feedback, the semantic recognition results provide the initial region for geometric extraction, while the extracted geometric structure, in turn, precisely guides and corrects the deep learning model's recognition in the form of a spatial attention map. The two work collaboratively and iteratively refine the model. This mechanism fundamentally overcomes the poor stability of traditional algorithms when facing image noise and distortion, and also solves the problem that existing deep learning models cannot understand and complete logical structures. Therefore, when processing financial statements of varying quality in the real world, it greatly improves the robustness of table localization and the accuracy of cell segmentation.

[0030] 2. Unlike existing technologies that rely on borders or corners for coarse correction, this invention constructs a table structure topology diagram that reflects the inherent logical relationships within the table and matches it with a typical financial statement grid diagram generated based on extensive data learning. This method, based on global topology optimization, can calculate precise pose parameters, i.e., the affine transformation matrix, effectively addressing complex nonlinear distortions caused by shooting angles, paper curling, etc., thus achieving table flattening. This provides an ideal, distortion-free image foundation for subsequent structural repair and OCR recognition.

[0031] 3. This invention proposes a scheme for generating a multi-channel repair guidance map, providing the repair network with unprecedentedly rich prior information. This guidance map not only marks the defect location and defect marking channels, but more importantly, it clarifies the complete line structure of the table, the line structure channels, and the content areas that must be protected, as well as the content area channels. This makes table repair no longer a blind pixel-level filling, but an intelligent reasoning based on the table's logical structure. The repair network can understand where to draw straight lines and where to retain text, thereby faithfully restoring broken frame lines, removing stains, and perfectly protecting critical financial data, outputting a table image with a complete structure and clear content.

[0032] 4. The generation of the typical financial statement grid diagram and the training of the table repair network in this invention both rely on large-scale and diverse financial statement datasets. This makes the method not limited to a specific report format, but rather has good adaptability and generalization ability for various types of financial statements. In the context of massive document processing in the financial industry, this method can stably and adaptively complete tasks, significantly reducing the reliance on manual proofreading and providing key technical support for achieving end-to-end automation of financial business processes. Attached Figure Description

[0033] Figure 1 The flowchart illustrates a table region localization and correction method based on image edge detection, as proposed in an embodiment of this invention.

[0034] Figure 2 This is a schematic diagram of the collaborative feedback link structure proposed in an embodiment of this invention application;

[0035] Figure 3 This is a schematic diagram of the structure of the multi-channel repair guidance diagram proposed in the embodiments of this invention. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0037] Please see Figures 1 to 3 This invention relates to a table region localization and correction method based on image edge detection, the specific implementation steps of which are as follows:

[0038] Obtain the raw image of the financial statements;

[0039] A base mask is extracted from the original image using an instance segmentation network based on Mask R-CNN; within the area indicated by the base mask, a geometric skeleton is extracted from the original image using an edge detection algorithm; the geometric skeleton is fed back to the instance segmentation network based on Mask R-CNN, forming a collaborative feedback loop based on the instance segmentation network and the edge detection algorithm, and iteratively converges to generate cell structure masks and edge structure information;

[0040] A table structure topology graph is established based on cell structure mask and edge structure information; the table structure topology graph is then matched with the highest-fitting financial statement grid graph in the financial statement database to calculate the attitude correction parameters.

[0041] Based on the posture correction parameters, an affine transformation is performed on the original image and the cell structure mask to obtain the posture-corrected image and the corrected mask.

[0042] A multi-channel repair guide map is generated based on the corrected mask and edge structure information; a table repair network is trained using a financial statement database; based on the multi-channel repair guide map, the table repair network is used to repair table defects and generate the final corrected image.

[0043] The technical solution of the present invention will be further described in detail below with reference to specific embodiments.

[0044] Example 1

[0045] This application discloses a table region localization and correction method based on image edge detection, which is used to perform table region localization and correction on balance sheet A. (See attached document.) Figure 1 The specific implementation steps of the method proposed in this invention include:

[0046] S1. Obtain the original image of the financial statements;

[0047] S2. Extract a base mask from the original image using an instance segmentation network based on MaskR-CNN; extract a geometric skeleton from the original image within the area indicated by the base mask using an edge detection algorithm; feed the geometric skeleton back to the instance segmentation network to form a collaborative feedback loop based on the instance segmentation network and the edge detection algorithm, and iteratively converge to generate cell structure masks and edge structure information.

[0048] S3. Establish a table structure topology graph based on cell structure mask and edge structure information; perform graph matching between the table structure topology graph and the highest-fitting financial statement grid graph in the financial statement database, and then calculate the attitude correction parameters;

[0049] S4. Based on the posture correction parameters, perform an affine transformation on the original image and the cell structure mask to obtain the posture-corrected image and the corrected mask;

[0050] S5. Generate a multi-channel repair guide map based on the corrected mask and edge structure information; train a table repair network using a financial statement database; based on the multi-channel repair guide map, use the table repair network to repair table defects and generate the final corrected image.

[0051] Further, obtain the original image of the uploaded balance sheet A; corresponding to step S1 above, specifically:

[0052] Specifically, the object being processed is a scanned image of an A4-sized paper balance sheet, with a size of [missing information]. The pixels are tilted counterclockwise by approximately 3 degrees. In the "Liabilities and Owner's Equity" section, there is a mark circled in red that covers the border of two cells.

[0053] Further, a base mask is extracted from the original image of balance sheet A using an instance segmentation network based on Mask R-CNN; within the area indicated by the base mask, a geometric skeleton is extracted from the original image using an edge detection algorithm; the geometric skeleton is fed back to the instance segmentation network to form a collaborative feedback loop based on the instance segmentation network and the edge detection algorithm, iteratively converging to generate cell structure masks and edge structure information; corresponding to step S2 above, the specific implementation process includes:

[0054] Specifically, the original image is input into a pre-trained instance segmentation network based on Mask R-CNN for initial recognition. Due to strong feature interference from the red pen markings, the model incorrectly identifies two covered cells as a large, irregular mask. The flawed mask set generated at this stage is called the base mask. This network, pre-trained on over ten thousand images of various financial statements, is capable of initially recognizing text regions and table cells in the image.

[0055] Using an edge detection algorithm, edge pixels are extracted within the image region defined by the base mask using the Canny operator to form a preliminary edge map, with a low threshold of 50 and a high threshold of 15. Then, a Hough transform is applied to this edge map to identify line segments longer than 50 pixels, which constitute the geometric framework.

[0056] The geometric skeleton, a dataset containing line coordinates and angles, is transformed into a spatial attention map with the same size as the feature map inside the Mask R-CNN-based instance segmentation network through an attention encoder. In this map, the positions of real table lines have higher weights, while non-table line regions have lower weights. This spatial attention map is then weighted and fused with the deep feature map of the MR-CNN to guide the network to pay more attention to these identified geometric structures in the next round of inference. In the second inference, the network focuses more on the regions enhanced by the spatial attention map, thus generating cell masks that more closely resemble the actual table lines. This process is repeated until the difference between the line segments of the old and new skeletons satisfies a multi-dimensional convergence criterion.

[0057] Specifically, the multi-dimensional convergence judgment mechanism is as follows:

[0058] After each iteration, four stability metrics are calculated: quantitative stability (the difference in the total number of line segments between the old and new structures); positional stability (the average deviation distance of line segments between the old and new structures); angular stability (the average angular change of line segments between the old and new structures); and topological stability (the change in the number of key intersections in the geometric framework, including cross and T-shaped intersections). Then, the state difference vector is calculated, and the convergence inertia, i.e., the rate of change of difference, is calculated based on the state difference vector, using the formula:

[0059] ;

[0060] in, Let the convergence inertia of the k-th round be... Let be the state difference vector in the k-th round. This indicates the calculation of Euclidean distance. When convergent inertia... If the value is less than the threshold of 0.05 for three consecutive cycles, the geometric structure skeleton is considered to have reached a stable state, i.e., the geometric structure skeleton has converged.

[0061] In this embodiment, after 5 iterations, the geometric skeleton extracted by the collaborative feedback link tends to be stable, and at this time the structural positioning information of the cell structure mask and each table line is obtained.

[0062] The collaborative feedback loop significantly improves the accuracy and reliability of cell segmentation. Traditional single-inference networks are prone to errors, while this solution uses a loop of semantic-guided geometry and geometric-refined semantics to enable the model to self-correct and optimize. This allows cell masks and structural localization information to converge to a better solution during iteration, solving the problem of inaccurate initial segmentation.

[0063] Edge detection algorithms ensure the stable and reliable extraction of the table's geometric skeleton from imperfect masks. The Canny algorithm and Hough transform effectively filter out noise, focusing on extracting straight-line features. The extracted structural localization information provides high-quality geometric feedback for collaborative feedback, ensuring the effective operation of the entire feedback system.

[0064] Furthermore, a table structure topology graph is established based on the cell structure mask and edge structure information; the table structure topology graph is then graph-matched with the highest-fitting financial statement grid graph in the financial statement database to calculate the attitude correction parameters; corresponding to step S3 above, the specific implementation process includes:

[0065] The algorithm iterates through each individual cell mask instance in the cell structure mask, calculating the first and zeroth moments for each mask based on its partition distribution, and further obtaining the precise coordinates of its geometric center point. The geometric center point is abstracted as a topology graph node. The coordinates of its cell boundary and geometric center point are used as the positioning coordinates of that cell instance and stored in the corresponding topology node attributes. After all nodes are generated, connection relationships are identified between the generated nodes based on edge structure information. Each line segment in the edge structure information is traversed, and the existence of two topology nodes is searched within a baseline distance threshold around it. In this embodiment, the baseline distance threshold is set to 5 pixels. After traversing all line segments and completing the above matching, all identified nodes and edges representing topological relationships together constitute a table structure topology graph that accurately and completely reflects the balance sheet structure.

[0066] The method described above for constructing a topological graph of a table structure achieves the abstraction and reconstruction of the table image structure. Compared to directly processing pixels or boundaries, abstracting a table into a topological graph containing nodes and edges can more fundamentally describe the row and column relationships and sparsity of the table. This representation method is more robust to subsequent global matching and local correction, and is less affected by local variations in contour thickness or small noise sparsity.

[0067] Furthermore, graph matching is performed between the table structure topology graph and the highest-fitting financial statement mesh graph in the financial statement database to calculate the attitude correction parameters; corresponding to step S3 above, the specific implementation process includes:

[0068] First, over five thousand standard asset inventory tables, profit and loss statements, and other financial statements were collected and categorized to establish a financial statement database. Then, pattern analysis was performed on the table structure topology diagram generated in this embodiment to extract key topological features, such as the number of rows and columns and the relative spacing between cells. Based on these key topological features, a maximum matching algorithm was used within the financial statement database to identify the typical financial statement with the highest matching degree.

[0069] Specifically, the maximum matching algorithm is as follows:

[0070] Key topological features are extracted, including the total number of rows and columns in the table and the cell merging pattern. Auxiliary features are extracted, including node adjacency relationships and cell span. Using these key topological features, a first round of screening and filtering is performed from the entire financial statement database to define a candidate set of highly similar financial statements. A matching degree function is constructed using the auxiliary features; the higher the overlap with the auxiliary features, the higher the matching degree score. The financial statement with the highest matching degree score is selected as the representative financial statement.

[0071] An affine transformation matrix that minimizes the topological error between balance sheet A and typical financial statements is iteratively solved using a random sampling consensus algorithm. This matrix accurately describes the 3-degree tilt of the original image.

[0072] The graph matching and parameter calculation methods described above provide a highly predictive and widely representative matching benchmark. By establishing a financial grid graph database and using algorithmic filtering, the graph matching can better adapt to various versions and specifications of reports, enhancing the method's versatility. Furthermore, by converting the table topology into a topological graph and performing graph matching with a standard template, it can go beyond simple four-corner perspective transformation to calculate a more accurate affine transformation matrix, thus successfully flattening deformed tables and creating ideal conditions for subsequent repair and OCR.

[0073] Further, based on the pose correction parameters, an affine transformation is performed on the original image and the cell structure mask to obtain the pose-corrected image and the corrected mask; corresponding to step S4 above, the specific implementation process includes:

[0074] Based on the solved affine transformation matrix, an affine transformation is performed on the original image and the cell structure mask to obtain a perfectly horizontal pose-corrected image and a corrected mask.

[0075] Furthermore, a multi-channel repair guidance diagram for repairing asset storage table A is generated based on the corrected mask and edge structure information; corresponding to step S5 above, the specific implementation process includes:

[0076] Specifically, based on the edge structure information, a binary mask reflecting the complete table frame is generated, namely the line structure channel. A logical union operation is performed on all cell masks in the corrected mask to generate a binary mask identifying all non-edge content-bearing areas, namely the content region channel. The line structure channel is compared and analyzed with the pose-corrected image to identify specific areas with pixel loss and cracks due to red pen markings, generating a binary mask marking the repair locations, namely the defect marking channel. Finally, the line structure channel, content region channel, and defect marking channel are combined to form a multi-channel repair guidance map.

[0077] The aforementioned multi-channel restoration guidance map provides comprehensive prior information for the table restoration network, not only indicating the image defects but also illustrating the ideal table structure and the content areas that need to be protected. By creating the multi-channel restoration guidance map, the restoration task becomes clearer, thus achieving accuracy and fidelity far exceeding traditional image restoration methods.

[0078] Furthermore, a table repair network is trained using a financial statement database; based on the multi-channel repair guidance map, the table repair network is used to repair table defects and generate a final corrected image; corresponding to step S5 above, the specific implementation process includes:

[0079] Over 10,000 high-resolution balance sheet images were selected from a financial statement database and processed into paired training samples. For each statement image, random defacement was applied to simulate various image defects that might be encountered in real-world financial business scenarios, generating a defaced image table. Based on the original complete financial statements, line structure channels, content region channels, and defect marker channels were extracted, and these three channels were combined to generate a sample restoration guide map. Paired training samples were constructed by combining the complete financial statement images, the defaced table images, and the sample restoration guide map.

[0080] U-Net with an attention mechanism is adopted as the basic architecture for the table inpainting network. This architecture includes an encoder path to capture the contextual features of the image and a symmetrical decoder path. Attention modules are incorporated into the skip connections between the encoder and decoder, enabling the network to pay more attention to important structural regions indicated by the guiding map.

[0081] In each training batch, the incomplete table image and the corresponding sample repair guide image are concatenated along the channel dimension to form a 4-channel tensor as the input to the repair network.

[0082] To ensure the accuracy and realism of the restoration, a composite loss function is used to calculate the overall error between the predicted restored image and the ideal complete table image. This composite loss function... It consists of three weighted parts:

[0083] Structural similarity loss measures the similarity between two images in terms of brightness, contrast, and structure. This loss function ensures that the repaired table maintains macroscopic structural consistency with the original table, avoiding structural distortion or blurring.

[0084] The mask-weighted reconstruction loss uses the content region channel and defect marker channel of the sample repair guide map as weight masks to weight the traditional L1 pixel loss, focusing the loss calculation on the most critical pixel subset, thus achieving strong protection of the content and key supervision of the repair area.

[0085] The structural gradient loss algorithm extracts image edges using the Laplacian operator and calculates the difference between the predicted and ground truth images on the edge map. By focusing on the rate of change between pixels, the algorithm effectively constrains the inpainting network to generate inpainting results that conform to the characteristics of the lines, thus improving the quality of line inpainting.

[0086] The three loss structures mentioned above are weighted to form a composite loss function. .

[0087] The Adam optimizer, based on gradient descent, is used to calculate the overall error via backpropagation. The gradients of the network's internal weight parameters are calculated, and the changes are iteratively updated in the reverse direction of the gradients. Training is performed on a high-performance GPU cluster with a batch size of 16 and an initial learning rate of 0.0002. The network model is considered converged when the loss on the independent validation set stops decreasing for several consecutive epochs, by monitoring the loss value. After training, the network model's weight parameters are saved. The final result is a table-based network repair tool.

[0088] Finally, the pose-corrected image of Balance Sheet A and the corresponding multi-channel repair guide image are input into the trained table repair network. The table repair network outputs a corrected image, in which the table lines below the red pen marks are completely repaired.

[0089] A deep learning model specifically designed for table restoration tasks was obtained. By constructing specific training sample pairs containing incomplete images, ideal images, and guide maps, the network learns how to perform intelligent restoration using structured prior information. Its restoration capabilities far surpass those of general image restoration networks, especially in restoring the continuity of table lines and protecting text content from damage, demonstrating significant advantages.

[0090] The overall technical solution of this invention proposes a comprehensive solution combining traditional image recognition and neural networks. By leveraging the synergistic feedback between an instance segmentation network and an edge detection algorithm, it overcomes the limitations of single methods in handling severely distorted or low-quality images, significantly improving the accuracy of table edge structure recognition. Pose calculation based on graph matching rather than simple edge detection yields globally optimal geometric correction parameters, effectively addressing complex nonlinear distortions. A multi-channel restoration guidance graph is designed, providing rich structural prior knowledge for the table restoration network. This ensures that restoration is no longer blind pixel filling, but rather targeted, structured restoration consistent with table logic. The restored image has clear lines and complete content, greatly facilitating subsequent OCR processing.

[0091] Example 2

[0092] This embodiment aims to detail the specific workflow of the present invention when processing a scanned copy of a bank statement where some table lines are broken due to printing quality issues, particularly demonstrating how the collaborative feedback link corrects initial identification errors. (See reference...) Figure 2 .

[0093] The object being processed is a partial image of a bank statement extracted from a company's loan application materials, with a size of [size missing]. The image contains columns such as "Transaction Date," "Summary," "Income," and "Expenses," but a key horizontal line separating two transactions has a broken section in the middle due to a printing quality issue.

[0094] Will The original image of pixels is input into the pre-trained instance segmentation network. Due to a 40-pixel break in the horizontal lines, the network cannot visually find a clear closed boundary. Therefore, it incorrectly merges two cells that should be independent (in this example, expenditure amount cells belonging to two separate transactions) into an irregular cell mask. Other cells with clear lines were correctly identified.

[0095] Based on the initial mask, the geometric skeleton is extracted. The edge detection algorithm then performs edge detection on the initially generated cell mask, including erroneous cells. Within the defined image region, the Canny operator is applied. A low threshold of 50 and a high threshold of 150 are set. The gradient intensity of the pixel neighborhood is calculated using this operator, and a double-threshold method is used to filter and connect edge points. A preliminary binary edge feature map is generated. In the image, all clear table lines are successfully detected as continuous edge pixel chains. For the broken horizontal line, its two unbroken segments are also detected, but a clear pixel interruption is formed in the middle. The Hough transform is applied to the above edge feature map to identify straight line segments. Because the gap at the break is larger than the maximum gap set by the Hough transform, the Hough transform identifies the broken horizontal line as two independent, discontinuous line segments.

[0096] All extracted line segments are processed by an attention encoder to generate an 800x600 spatial attention map. In this map, the pixel positions corresponding to the identified line segments are assigned high weight values ​​(1.0 in this example), while other areas are assigned low weight values ​​(0.1 in this example). This attention map is fed back to the instance segmentation network and fused with its backbone network on the deep feature map.

[0097] The instance segmentation network utilizes enhanced feature data fused with attention maps for a second inference. Despite visual breaks in the original image, the contextual information provided by the attention maps guides the network to tend to segment at those locations. The network successfully corrects the previously incorrect cell masks. The correction resulted in two independent cell masks with more regular boundaries, correctly separating the two transaction records. In the third iteration, a more complete attention map further guided the segmentation network to generate near-perfect cell masks. When the geometric skeleton was extracted again, the result was exactly the same as the skeleton data from the second iteration, indicating that the geometric skeleton had converged.

[0098] After the above collaborative feedback iteration process, this embodiment finally outputs two key pieces of information: cell mask, a set of masks that precisely correspond to the actual table cells and have clear boundaries; and structural positioning information, a geometric skeleton data containing all table lines.

[0099] Example 3

[0100] This implementation details the process of extracting the table structure topology graph from a tilted profit and loss statement image, matching it with a height-adapted financial statement graph, and performing graph matching to calculate the attitude correction parameters used for geometric correction.

[0101] The object being processed is a profit and loss statement image with an approximately 15-degree tilt and slight perspective distortion. The precise mask and edge information of all cells in the image have been extracted.

[0102] The original image underwent rapid preprocessing to extract its coarse topological features, including the number of rows and columns. Keyword-based OCR was then used to quickly locate the approximate positions of several key anchor cells, such as "Total Profit" and "Operating Revenue." The extracted coarse features were then compared one by one with typical financial statement images in the database to calculate their matching degree. The matching degree score reflects the degree of similarity between the image and the structure of typical financial statement images. Through comparison, image A in the database had the highest matching degree score, 0.93. Therefore, it was determined to be the financial statement image with the highest fit to this original image.

[0103] Iterate through the mask of each cell in the image and calculate its geometric center point coordinates. The center point of the mask for the "Revenue" cell is calculated as follows: The center point of the "Operating Costs" cell mask was calculated as Based on edge structure information, a nearly perpendicular table line connecting the "Operating Revenue" cell and the "Operating Cost" cell is identified, and its geometric attributes, such as direction and length, are recorded as the edge connecting the two nodes. A table structure topology graph containing the center points of all cells and the connecting lines is constructed. An affine transformation matrix M with minimum topological error is solved. An error function is defined; in this example, the mean square error of the coordinates of all matching node pairs after affine transformation is used. The least squares method is employed to calculate a 2x3 affine transformation matrix M.

[0104] ;

[0105] This matrix precisely describes the composite transformations, such as rotation, scaling, and translation, required to correct the image, i.e., the pose correction parameters.

[0106] Example 4

[0107] This embodiment describes in detail the process of generating a multi-channel repair guide map. (See attached document.) Figure 3 This document describes the training and application process of the table repair network. This example uses a corrected but still flawed cash flow statement image as an example.

[0108] An image of a company's cash flow statement after precise attitude correction in the preceding steps, with dimensions of [size missing]. Pixels. Although the image has a standard geometric pose, it still has the following defects: the horizontal dividing line below "Net Cash Flow from Operating Activities" has a 60-pixel break after scanning due to creases in the original document; there is a [missing information] on the amount figure corresponding to "Net Cash Flow from Investing Activities". The blemishes on the pixels obscure part of the numbers.

[0109] Create A completely black background for each pixel. Based on the existing complete geometric skeleton information, all the table lines that should exist on the background are drawn in white, with a line width of 2 pixels, generating a binary mask for the line structure channel.

[0110] The line structure channel is compared pixel-by-pixel with the pose-corrected image to identify pixels that exist on the ideal lines but are missing in the actual image. A blob detection algorithm is applied to the pose-corrected image to find connected regions that do not belong to text or lines but meet the characteristics of blemish size, such as connected regions with an area between 100 and 2000 pixels in this example. The two types of defect regions are merged to generate the final defect marker channel.

[0111] Logically combine all known cell masks into a single binary image. This image precisely tells the repair network which areas need to be protected, generating content region channels.

[0112] Stack the three channels mentioned above to form a sheet. The three-channel guide diagram.

[0113] The table inpainting network uses a U-Net neural network, which consists of an encoder part for feature extraction, a decoder part for upsampling and detail restoration, and an output part that finally outputs the inpainted image.

[0114] A training set containing 50,000 pairs of financial statement samples is established. The incomplete table image is concatenated with a multi-channel restoration guide image along the channel dimension to form a 4-channel tensor, which serves as the input to U-Net. The network outputs a predicted restored image. A composite loss function is used to calculate the overall error between the predicted image and the ideal complete table image. An Adam optimizer based on gradient descent is employed, and the internal weight parameters of U-Net are iteratively updated based on the overall error using a backpropagation algorithm.

[0115] After approximately 100 epochs of iterative training, the model converges when the network's loss on the validation set no longer decreases. This results in a well-trained table repair network capable of intelligently correcting table defects based on structured guidance information.

[0116] The posture-corrected image of the cash flow statement in this embodiment is stitched together with the corresponding multi-channel repair guidance image to form a... The input tensor is fed into a pre-trained table repair network for one forward inference iteration. The table repair network outputs a table... The final corrected image.

[0117] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for locating and correcting table regions based on image edge detection, characterized in that, Includes the following stages: Obtain the original image of the financial statement; perform table semantic recognition on the original image using an instance segmentation network to generate a basic mask; within the range defined by the basic mask, perform geometric pattern recognition using an edge detection algorithm to generate a geometric structure skeleton; The geometric skeleton is fed back to the instance segmentation network; A collaborative feedback loop based on instance segmentation network and edge detection algorithm is formed, and the cell structure mask and edge structure information are generated iteratively and converged. A table structure topology graph is established based on cell structure mask and edge structure information; the table structure topology graph is then matched with the highest-fitting mesh graph in the financial statement database to calculate the attitude correction parameters. Based on the posture correction parameters, geometric correction is performed on the original image and the structural mask to generate a posture-corrected image and a corrected mask. A multi-channel repair guidance map is generated based on the corrected mask and edge structure information. Specifically, a binary mask reflecting the complete table frame is generated based on the edge structure information, serving as the line structure channel; a logical union operation is performed on all cell masks in the corrected mask to generate a binary mask that identifies all non-edge regions, serving as the content region channel; the line structure channel is compared and analyzed with the pose-corrected image to identify specific areas where the table is missing, and a binary mask marking the location to be repaired is generated as the defect marking channel; the line structure channel, content region channel, and defect marking channel are combined to form a multi-channel repair guidance map. A table repair network is trained using a financial statement database; based on the multi-channel repair guidance map, the table repair network is used to perform table repair on the pose correction image to generate the final correction image.

2. The table region localization and correction method based on image edge detection according to claim 1, characterized in that, The collaborative feedback loop based on the instance segmentation network and edge detection algorithm is specifically as follows: The original image is input into an instance segmentation network based on Mask R-CNN for initial forward inference; the instance segmentation network identifies and generates a basic mask by understanding the contextual semantics of the original image. Based on the base mask, the edge detection algorithm is used to perform geometric pattern recognition on the original image to generate a geometric structure skeleton. The geometric skeleton is encoded into a spatial attention map using an attention encoder; The spatial attention map is used as a feedback signal and fused with the original image to generate the guidance parameters required for the next iteration, forming a collaborative feedback loop; The collaborative feedback loop iterates until the geometric skeleton converges, generating cell structure masks and edge structure information.

3. The table region localization and correction method based on image edge detection according to claim 2, characterized in that, The specific implementation process of generating a geometric structure skeleton by performing geometric pattern recognition on the original image based on a base mask and using an edge detection algorithm includes: The image region is defined based on the base mask. Within the defined region, the Canny operator is used to extract candidate edge pixels that constitute the table outline, generating an edge feature image. The Hough transform is used to extract linear structures from the edge feature image, and all extracted linear structures are integrated to form the geometric skeleton.

4. The table region localization and correction method based on image edge detection according to claim 1, characterized in that, The specific implementation process of establishing the table structure topology graph based on cell structure mask and edge structure information includes: Instance analysis is performed on the cell structure mask to identify the cell structure mask boundary and geometric center point, and abstract them into topological graph nodes representing cell instances. The positioning coordinates of cell instances in the image are recorded in the topological graph nodes. Based on edge structure information, the connection relationship between the generated nodes is identified, and the table line segments connecting two adjacent nodes are extracted and abstracted into edges of the topological graph. All identified nodes and edges representing topological relationships together constitute the table structure topological graph.

5. The table region localization and correction method based on image edge detection according to claim 4, characterized in that, The process of performing graph matching between the table structure topology graph and the highest-fitting mesh graph in the financial statement database to calculate the attitude correction parameters includes: Collect various types of financial statements, classify and store them according to type, and establish a financial statement database; Pattern analysis is performed on the table structure topology graph to extract key topological features, including the number of rows and columns and the relative spacing between cells. Based on these key topological features, the graph is identified in the financial statement database, and the highest-fitting financial statement grid graph is extracted using a maximum matching algorithm. A global optimization algorithm is then used to perform graph matching between the table structure topology graph and the highest-fitting financial statement grid graph, iteratively solving for an affine transformation matrix that minimizes the positional deviation of all corresponding nodes and the angular deviation of corresponding edges between the two graphs, which serves as the attitude correction parameter.

6. The table region localization and correction method based on image edge detection according to claim 1, characterized in that, The specific implementation process of using a financial statement database to train a table-based network for repair includes: Extract complete financial statements from the financial statement database; process the complete financial statements into paired training samples; randomly incomplete the complete financial statements to generate incomplete table images; extract line structure channels, content area channels, and defect marker channels based on the complete financial statements; generate a sample repair guide map by combining the three channels; construct paired training samples by combining the complete financial statements, incomplete table images, and sample repair guide maps. The incomplete table image and the sample repair guide image are concatenated along the channel dimension and input into the table repair network to generate a predicted repair image. The overall error between the predicted repair image and the complete financial statement is calculated using an error function. Based on the overall error, the internal weight parameters of the table repair network are iteratively updated using a gradient descent-based backpropagation algorithm until the model converges, generating the trained table repair network.

Images