A document table extraction method, device, equipment and medium

Abstract

The application discloses a document table extraction method, device and equipment and a medium, and relates to the technical field of computer information processing. The extraction method comprises the following steps: performing OCR recognition on a document table image to be processed to obtain a text block; performing visual feature coding on the document table image to obtain deep visual features; performing semantic feature coding on a text sequence of the text block to obtain a semantic feature vector; performing spatial feature coding on a boundary box of the text block to obtain a spatial feature vector; performing feature fusion processing on the deep visual features, the semantic feature vector and the spatial feature vector to obtain multi-modal guided features; and performing structured decoding processing on the multi-modal guided features to obtain a structured representation of the table. The application fuses the text and position information pre-recognized by OCR with the visual features of the document table, guides the re-expression of the visual features, and actively aligns to a logical structure defined by prior information, thereby improving the extraction accuracy of the logical structure of the table.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer information processing, and in particular to a document table extraction method, device, equipment and medium. BACKGROUND

[0002] Document table extraction is a key technology for detecting, structuring and extracting cell content (OCR recognition) of table regions in document images, and has wide applications in financial report analysis, bill processing, and archive digitization. Existing technologies usually follow a serial process of "image preprocessing -> table structure recognition -> cell content recognition" when processing document table recognition tasks. However, in the real physical world, document images obtained by scanning or shooting generally have defects such as geometric distortion (e.g. perspective, folding, bending), uneven illumination, low resolution and noise. These defects directly lead to misalignment of the row and column lines in the table in vision, seriously interfering with subsequent table structure recognition.

[0003] Although existing image correction algorithms can alleviate global geometric distortion to some extent, they often fail to deal with complex local deformation. When the structure recognition deviates, for example, two rows are incorrectly merged into one row, or one column is incorrectly split into two columns, the subsequent OCR recognition step may accurately recognize individual text blocks, but due to the incorrect structure attribution, the association between cell content and table header will be severely incorrect, i.e. "serial" or "misaligned row" problems. This error is structural and cannot be solved by simply improving the accuracy of the OCR engine.

[0004] Therefore, how to overcome the visual defects of document images and accurately recognize the logical structure of the table is a technical problem that needs to be solved in the current document table extraction field. SUMMARY

[0005] The present application provides a document table extraction method, device, equipment and medium to solve the above technical problems.

[0006] The present application is implemented by the following technical solutions:

[0007] In a first aspect of the present application, a document table extraction method is provided, comprising:

[0008] performing OCR recognition on the document table image to be processed to obtain text blocks;

[0009] performing visual feature encoding on the document table image to obtain deep visual features;

[0010] performing semantic feature encoding on the text sequence of the text blocks to obtain a semantic feature vector;

[0011] perform spatial feature coding on the bounding box of the text block to obtain a spatial feature vector;

[0012] perform feature fusion processing on the deep visual feature, the semantic feature vector and the spatial feature vector to obtain a multi-modal guided feature;

[0013] perform structured decoding processing on the multi-modal guided feature to obtain a structured representation of the table.

[0014] The above method of the present application separates visual features from text semantic information for the existing processing mode, and fails to fully utilize text priori knowledge to assist and calibrate table structure recognition. The present application innovatively takes the text content and its spatial position information recognized by OCR as semantic and spatial priori guided features, and deeply fuses them with the document table visual features to form a fusion feature representation with multi-modal priori information, guides the re-expression of the visual features, so that the model no longer relies on the traditional pixel-level image correction with limited effect, but forces the visual features to actively align to the logical structure defined by the priori information in the high-dimensional feature space, so that the decoding model can deal with complex local deformations such as wrinkles and bends that are difficult to cope with by the traditional method, and improves the decoding robustness.

[0015] Further, before performing OCR recognition on the document table image to be processed, the method further comprises:

[0016] obtaining a document image to be processed, recognizing and extracting a table region in the document image to obtain a document table image to be processed;

[0017] The document image is converted from a document in a real scene, and the conversion mode includes photographing and scanning.

[0018] Further, performing OCR recognition on the document table image to be processed to obtain text blocks, comprising:

[0019] calling at least two heterogeneous OCR recognition engines to perform OCR recognition on the document table image to be processed to obtain recognition results output by each of the OCR recognition engines; the recognition results include text blocks and confidence scores of the text blocks;

[0020] obtaining a credibility level of each of the text regions according to the confidence scores of the text blocks of each of the text regions;

[0021] dividing each of the text regions into a first text region and a second text region according to the credibility level;

[0022] selecting a text block with the largest confidence score among the text blocks of the first text region as a final corresponding text block of the first text region;

[0023] correcting the text sequence of the second text region to obtain a corrected text sequence;

[0024] correcting the text block of the second text region according to the corrected text sequence to obtain a corrected text block, and taking the corrected text block as a final corresponding text block of the second text region.

[0025] Further, the correcting the text sequence of the second text region to obtain a corrected text sequence comprises:

[0026] obtaining a local image of the second text region, and extracting a visual feature of the local image;

[0027] obtaining a preliminary text sequence of the second text region, and extracting a text feature of the preliminary text sequence; the preliminary text sequence is obtained by an existing text sequence in the text block of the second text region;

[0028] inputting the visual feature and the text feature into a cross-attention mechanism to obtain a cross-attention fusion feature vector;

[0029] correcting the preliminary text sequence based on the cross-attention fusion feature vector to obtain a corrected text sequence.

[0030] Further, the correcting the text block of the second text region according to the corrected text sequence to obtain a corrected text block comprises:

[0031] obtaining a confidence score of the corrected text sequence;

[0032] if the confidence score of the corrected text sequence is not lower than a confidence threshold, replacing the existing text sequence in the text block of the second text region with the corrected text sequence to obtain a corrected text block;

[0033] if the confidence score of the corrected text sequence is lower than the confidence threshold, replacing the existing text sequence in the text block of the second text region with a predefined semantic placeholder to obtain a corrected text block.

[0034] Further, the method further comprises: replacing the text sequence of the text block with a null value with a predefined semantic placeholder.

[0035] Further, the feature fusion processing of the deep visual feature, the semantic feature vector and the spatial feature vector to obtain a multi-modal guided feature comprises:

[0036] The deep visual feature, the semantic feature vector and the spatial feature vector are projected into the same feature dimension respectively, and the deep visual feature, the semantic feature vector and the spatial feature vector are spliced along the feature dimension, to obtain a multi-modal guided feature.

[0037] In a second aspect, the present application provides a document table extraction device, comprising:

[0038] An OCR recognition module is configured to perform OCR recognition on the document table image to be processed to obtain a text block.

[0039] A visual feature encoding module is configured to perform visual feature encoding on the document table image to obtain a deep visual feature.

[0040] A semantic prior extraction model is configured to perform semantic feature encoding on a text sequence of the text block to obtain a semantic feature vector.

[0041] A spatial prior extraction module is configured to perform spatial feature encoding on a bounding box of the text block to obtain a spatial feature vector.

[0042] A multi-modal fusion module is configured to perform feature fusion processing on the deep visual feature, the semantic feature vector and the spatial feature vector to obtain a multi-modal guided feature.

[0043] A structured decoding module is configured to perform structured decoding processing on the multi-modal guided feature to obtain a structured representation of the table.

[0044] In a third aspect, the present application provides an electronic device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the document table extraction method of any one of the first aspect.

[0045] In a fourth aspect, the present application provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executable by a processor to implement the document table extraction method of any one of the first aspect.

[0046] Compared with the prior art, the present application has the following advantages and beneficial effects:

[0047] The understanding of the visual feature is guided and calibrated by the text and spatial prior information, so that the model can correctly infer the logical structure of the table by means of the logical association of the text content and the spatial arrangement rules even if the visual lines are blurred or broken, and fundamentally solves the problem of structure misjudgment caused by unreliable vision.

[0048] Through multi-source OCR collaborative decision-making and adaptive correction mechanism, the application can generate high-quality and high-credibility text priori information, effectively counteracts the noise and distortion of real scene images, and improves the quality of input information from the source, thereby laying a solid foundation for subsequent identification.

[0049] By introducing semantic placeholders, the complete topological structure of the table is preserved, and through an end-to-end structured decoder, the structure and content of the entire table can be globally and one-time generated, which avoids error accumulation in intermediate steps, and can effectively prevent serious structural errors such as'serial' and'misaligned' cell content, thereby ensuring the logical correctness of the output result.

[0050] The framework design of the application has modular characteristics, and the OCR engine, visual backbone network and language model can be flexibly replaced or upgraded to adapt to the needs of technology development and different application scenarios, and has good universality and scalability. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the example embodiments of the application, the drawings needed in the examples will be briefly introduced as follows, and it should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor. In the drawings:

[0052] Figure 1 is a flowchart of a document table extraction method according to an embodiment of the application;

[0053] Figure 2 is a flowchart of a priori information extraction method based on collaborative decision-making and credibility grading according to an embodiment of the application;

[0054] Figure 3 is a table extraction flowchart of a financial statement image according to an embodiment of the application;

[0055] Figure 4 is a table extraction result diagram according to an embodiment of the application. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solutions and advantages of the application more clear and explicit, the application will be further described in detail below in combination with examples and drawings, and the illustrative embodiments of the application and their descriptions are only used to explain the application, and not as a limitation on the application.

[0057] It has to be explained that the terms "include" and "have" and their any variants in the specification and claims of the present application and the above-mentioned drawings are intended to cover the inclusions that are not exclusive, for example, the processes, methods, systems, products or devices that include a series of steps or units do not have to be limited to or inherent to other steps or units.

[0058] The terms used in various embodiments of the present application are only for the purpose of describing specific embodiments and are not intended to limit various embodiments of the present application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly dictates otherwise. Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which various embodiments of the present application belong. The terms (such as those defined in a generally used dictionary) will be interpreted to have the same meaning as the contextual meaning in the related art and will not be interpreted to have an idealized meaning or an overly formal meaning, unless clearly defined in various embodiments of the present application.

[0059] Due to the existence of geometric distortion, visual noise and other non-ideal factors in the document image, the accuracy of table structure recognition depending on the pure visual model is low, and the existing table extraction method based on optical character recognition (OCR) usually takes OCR as an isolated post-processing link, so that the valuable text semantic information and the inherent spatial layout information in the table structure cannot be effectively utilized in the structure recognition stage, causing the disconnection of visual information and semantic information.

[0060] Based on this, the present application proposes a document table recognition method based on multi-modal prior information guidance. The core idea of the method is to construct the text content and its spatial position information recognized by OCR as a kind of strong semantic and spatial prior information, which is used to actively guide and modulate the expression of visual features, forming a deep fusion multi-modal feature representation. Based on the feature representation, the decoding model can understand the logical layout of the table more deeply, and finally realize the end-to-end and high-precision joint recognition of the table logical structure and cell content.

[0061] Referring to Figure 1 which is a document table extraction method flowchart of an embodiment of the present application, comprising the following steps.

[0062] S1, respectively, the document table image to be processed is subjected to OCR recognition processing and visual feature coding processing, to obtain text blocks and deep visual features.

[0063] First, a document table image to be processed is acquired, which includes a table object to be extracted. The original form of the document table image is a document file, which can be obtained by converting various formats of document files, such as word / PDF documents into common image formats, or directly obtained by photographing or scanning the document in the real scene. The document image obtained by scanning or photographing is more likely to have visual defects such as geometric distortion, uneven illumination, low resolution, and noise. It can be understood that before step S1 of the present application, the acquired document image has been subjected to basic preprocessing, such as document structure recognition, table region recognition and extraction, etc.

[0064] The document table image is subjected to OCR recognition processing, and each text block can be obtained. This step can call an existing OCR engine to realize. The OCR engine outputs the text sequence, the boundary box and the confidence score recognized from the document table image. The text sequence is the recognized text content, such as "revenue" and "¥1,500,000.00". The boundary box is stored in the form of a position vector, such as (x_min, y_min, x_max, y_max). x_min and x_max correspond to the minimum and maximum values of the boundary box in the x direction, and y_min and y_max correspond to the minimum and maximum values of the boundary box in the y direction. The confidence score is the degree of confidence of the computer on the recognition result, represented by a number between 0 and 1. For example, 0.98 represents 98% certainty that the recognition result is "revenue", and 0.6 represents only 60% certainty about the recognition result, reflecting the confidence of the text sequence.

[0065] The document table image is subjected to visual feature encoding processing, and the deep visual features of the document table image can be obtained. This step can be realized by a visual backbone network, such as Swin Transformer. The original, uncorrected complete document table image is input into the visual backbone network, which analyzes the image region by region and extracts deep visual feature maps that can represent global and local texture, edge and layout information of the image.

[0066] S2, the text sequence of the text block is subjected to semantic feature encoding to obtain a semantic feature vector, and the boundary box of the text block is subjected to spatial feature encoding to obtain a spatial feature vector.

[0067] The position vector of the bounding box of each text block obtained in S1 is two-dimensionally positionally encoded, projected into a high-dimensional space, and a spatial feature vector capable of accurately describing the relative position and size of each text block in the global image is generated. For example, by normalizing each bounding box and then projecting it into a high-dimensional space through a position encoding function, a spatial feature vector is obtained. The normalization of the bounding box can be achieved by dividing the width and height of the bounding box by the width and height of the image, respectively, to obtain the relative height and relative width, and converting the absolute coordinates to uniform relative coordinates. The position encoding function, such as the sine position encoding or its variant function, converts the four normalized coordinates into a high-dimensional vector (e.g., 768 dimensions), and each coordinate corresponds to multiple numbers, finally forming a vector representation of 768 numbers, and a high-dimensional space vector is generated for each text block.

[0068] The semantic feature encoding obtains the semantic feature vector of each text sequence through a semantic model. The semantic features of the same text block have a corresponding relationship with the spatial features. Based on this corresponding relationship, the semantic features and the spatial features can be fused to obtain the semantic and spatial prior features at the text block level. By splicing the semantic and spatial prior features of all text blocks, the spatial prior features at the document level can be obtained.

[0069] S3, the deep visual feature, the semantic feature vector and the spatial feature vector are fused to obtain a multi-modal guidance feature.

[0070] The multi-modal guidance feature can be obtained by first projecting each feature into the same feature dimension and then splicing along the feature dimension. Alternatively, the semantic feature vector and the spatial feature vector can be spliced to obtain prior features, and then spliced with the deep visual feature in the same feature dimension. For example, two independent and trainable projection networks are used to linearly project the above-mentioned visual features and prior features into a unified feature dimension, and then the two projected feature vectors are spliced to generate the final multi-modal guidance feature.

[0071] This step integrates the prior information containing semantics and positions into the visual features, so that each element in the fused multi-modal guidance feature contains three-dimensional information of "what can be seen" (visual), "where is it" (spatial), and "what is written" (semantic). The essence of the guidance fusion mechanism is to use the prior information generated in S2 as a calibration anchor to guide the re-expression of the visual features in the feature space, so that it can overcome the geometric distortion at the pixel level and directly align to the internal logical structure of the table.

[0072] S4, the multi-modal guidance feature is subjected to structured decoding processing to obtain a structured representation of the table.

[0073] With the multi-modal guidance features generated by S3, the final structured parsing of the table is completed, and the "position-text" table logic is accurately restored. This step can be implemented by a structured decoder based on the Transformer architecture, which is specifically designed to understand the row-column logical relationship of the table. The multi-modal guidance features are input into the structured decoder based on the Transformer architecture, and under the action of the self-attention mechanism, the model can determine which text blocks belong to the same row or column according to the spatial prior information and semantic prior information in the fusion features, even if they are visually misaligned in the original image, thereby significantly improving the robustness of table structure parsing.

[0074] The structured decoder generates a Markdown structured data file that describes the entire table logical structure and content in an end-to-end manner in an autoregressive manner. This output form includes row-column relationships, cross-row and cross-column information, and the final text within the cell.

[0075] The scheme of the present application is different from the traditional serial processing paradigm, but innovatively uses a multi-modal guidance fusion mechanism to use text and spatial prior information to "guide" and "calibrate" the model's understanding of visual features. This deep fusion allows the model to see the structural essence through the visual phenomenon, and even in the case of blurred or broken visual lines, it can correctly infer the logical structure of the table by relying on the logical association of the text content (such as similar data should be in the same column) and the spatial arrangement rules, fundamentally solving the problem of structural misjudgment caused by unreliable vision.

[0076] In a further improved embodiment, when performing OCR recognition on the document table image to be processed, at least two heterogeneous OCR recognition engines are called to perform OCR recognition on the document table image to be processed, and recognition results output by each recognition engine are obtained. Each recognition result output by each recognition engine includes text blocks corresponding to each text region of the document table image and confidence scores of the text blocks.

[0077] Heterogeneity is reflected in differences in model architecture, training data set, or recognition paradigm to ensure that the capabilities of each engine are complementary and reduce the risk of systematic bias of a single model. This embodiment aims to robustly and accurately extract text content and its bounding box position from the document table image to be processed, which may be distorted, and based on collaborative decision-making and confidence grading, to provide more accurate prior information for subsequent steps.

[0078] Prior information is extracted and corrected based on collaborative decision-making and confidence grading to ensure the credibility of the prior information. The specific operation is as shown in Figure 2 .

[0079] S2-1, based on the confidence scores of the text blocks of each text region, obtain the confidence level of each text region.

[0080] S2-2, divide each text region into a first text region and a second text region according to the confidence level. The division rule can be: the text region with a confidence level higher than a preset level is regarded as the first text region, otherwise as the second text region.

[0081] S2-3, the first text region and the second text region are processed differently:

[0082] S2-31, for the first text region: select the text block with the highest confidence score in the text blocks of the first text region as the final corresponding text block of the first text region;

[0083] S2-32, for the second text region: (1) correct the text sequence of the second text region to obtain a corrected text sequence; (2) correct the text block of the second text region according to the corrected text sequence to obtain a corrected text block; (3) take the corrected text block as the final corresponding text block of the second text region.

[0084] For example, the confidence level is divided into three levels: high, medium and low. For text regions with medium and high levels, select the text block with the highest confidence score corresponding to the text region to construct prior information. The recognition result of the text block with high confidence score has higher credibility, while the recognition result of the text block with low confidence score has higher probability of error. When the confidence scores of the text blocks corresponding to a text region are all low (such as lower than a preset score threshold 0.6), the overall confidence of the region is low, and the recognition results of each engine are not reliable, which needs to be corrected before being included in the construction of prior information to ensure the accuracy of the prior information.

[0085] For example, if the confidence of the recognition result of all engines for a certain region is higher than a first preset threshold (for example, 0.95), the text blocks of each recognition result are highly consistent within a preset string similarity threshold (for example, the normalized edit distance is less than 0.1) and an Intersection over Union (IoU) threshold (for example, greater than 0.9), the region is divided into a high-confidence region, the result with the highest confidence is adopted, and is marked as a high-confidence prior, or the consistent result (such as the voted text and the fused bounding box) is adopted; if the recognition results of each engine are inconsistent, but the confidence score of a certain engine is significantly higher than that of other engines and is higher than a second preset threshold (for example, 0.8), the region is divided into a medium-confidence region, the result with the highest confidence is adopted, and is marked as a medium-confidence prior; if there is a significant conflict between the results of each engine, and the confidence of all engines is lower than the second preset threshold, the region is marked as a low-confidence region, and is waiting for further correction processing.

[0086] Through the multi-source OCR collaborative decision-making and adaptive correction mechanism, the embodiment can generate high-quality and high-confidence text prior information, effectively resisting the noise and distortion of real scene images. This improves the quality of input information from the source, laying a solid foundation for subsequent recognition.

[0087] The confidence grading mechanism enables the subsequent fusion model to learn to "distinguish" and "dynamically balance" different labeled prior information. The model can intelligently switch between reliable text prior and noisy visual features, avoiding the interference of unreliable information on the overall decision, and making the model perform better in real complex scenes.

[0088] In a further improved embodiment, the text sequence of the second text region with a confidence level not higher than a preset level is corrected, specifically as follows.

[0089] S3-1, acquire a local image of the second text region, and extract visual features of the local image.

[0090] S3-2, acquire a preliminary text sequence of the second text region, and extract text features of the preliminary text sequence.

[0091] S3-3, input the visual features and the text features into a cross-attention mechanism to obtain a cross-attention fusion feature vector.

[0092] S3-4, correct the preliminary text sequence based on the cross-attention fusion feature vector to obtain a corrected text sequence.

[0093] The embodiment is directed to a text region with a credibility level that does not meet the standard. A lightweight visual-language correction module is instantiated, and a local image slice and a preliminary text sequence of the text region are input into the visual-language correction module for adaptive correction. The preliminary text sequence is obtained from the existing text sequence in the text block of the second text region, i.e., the text sequence obtained by preliminary recognition of the text region by at least two recognition engines. Through a cross-attention mechanism, the model can focus on the local image texture features, re-evaluate and correct the preliminary text sequence character by character or word by word, and finally output the corrected text sequence and a new confidence score.

[0094] For example, the three OCR results of a certain second text region are "management fee", "management fee", and "management fee". The most frequently occurring character (or word) is taken as the initial text sequence. For example, the character "guan" is consistent in the three results, so it is retained. The characters "ya, jia, and li" are inconsistent, so all possibilities are temporarily retained (as candidates). The character "fei" is consistent, so it is retained. The final initial text sequence is "guan [ya / jia / li] fee". Alternatively, the confidence score of the character is used to select the character with the highest confidence score from the candidate characters to form the initial text sequence. For example, the confidence score of "li" is the highest among "ya, jia, and li", and the initial text sequence "guan li fee" is formed.

[0095] The visual features of "guan li fee" and the local image are input into the cross-attention mechanism model, and the initial text sequence is corrected one by one based on the cross-attention mechanism. The character "li" is corrected to "li", and the initial text sequence "guan li fee" is obtained, and the confidence score of "guan li fee" is updated to 0.92.

[0096] Further, the second text block is corrected according to the corrected text sequence in S2-32, and a corrected text block is obtained. Specifically, if the confidence score of the corrected text sequence is not lower than the confidence threshold (e.g., 0.92>0.9), the existing text sequence in the text block of the second text region is replaced by the corrected text sequence, and a corrected text block is obtained. If the confidence score of the corrected text sequence is lower than the confidence threshold, the existing text sequence in the text block of the second text region is replaced by a predefined semantic placeholder, and a corrected text block is obtained.

[0097] For example, for the regions whose corrected confidence is still below a preset threshold (e.g. 0.5), instead of keeping the unreliable text at the location, a predefined placeholder with specific semantics, such as <UNCERTAIN_TEXT>, is generated at the original bounding box location of the cell. The innovation of this is that, while discarding the unreliable content information, the fact that the cell exists and its precise spatial location information in the whole table are kept intact, thus guaranteeing the integrity of the table topology and providing key layout clues for the subsequent structure recognition.

[0098] As for the updating of the bounding box of the corrected text block, one of the three or the average of the coordinates of the three bounding boxes can be selected as the new bounding box as the bounding box of the corrected text block.

[0099] Further, the text sequence of the text block with an empty text sequence is replaced by a predefined semantic placeholder.

[0100] Similar to the placeholder processing of semantic uncertainty, the key clue of converting "recognition failure" into "structure success" through the semantic placeholder converts the failure of content recognition (negative event) into a powerful structure signal (positive clue) that retains the precise spatial position, effectively blocking the propagation of errors. The semantic placeholder ensures that even if part of the cell content cannot be recognized, the overall topology skeleton of the table remains intact, fundamentally solving the stubborn problem of "serial" and "misaligned" global and catastrophic structure errors caused by local errors in the existing serial method, and ensuring the logical correctness of the output result.

[0101] The technical effects of the improved scheme of the present application will be further illustrated below with actual cases. The case is to process a financial statement image taken by a mobile phone, which has slight perspective distortion and slightly dark light on the right side, and contains column items such as project name and amount. The key process of the extraction case is shown in Figure 3 .

[0102] I. Prior information acquisition and correction.

[0103] 1.1. OCR recognition of text blocks.

[0104] The system calls three OCR engines (engine A: Tesseract-based; engine B: PaddleOCR-based; engine C: proprietary CNN+CTC model) in parallel to process the image.

[0105] 1.2. Extraction of reliable prior information according to the collaborative decision mechanism.

[0106] For the "Operating Income" cell in the report, all three engines clearly identify it as "Operating Income", with the IoU of the bounding box > 0.95 and the confidence levels of all > 0.98. This result is determined to be a high-confidence prior;

[0107] For a monetary cell "¥1,500,000.00", Engine A identifies it as "¥1,500,00(J.00" (confidence level 0.75) due to lighting issues, Engine B identifies it as "¥1,500,000.00" (confidence level 0.96), and Engine C identifies it as "¥1,500,000.OO" (confidence level 0.88). According to the collaborative decision-making mechanism, the result of Engine B is adopted due to the highest confidence level and is marked as a medium-confidence prior.

[0108] For a cell blurred due to paper wrinkles, the three engines give conflicting results with low confidence levels (all < 0.6) such as "Administrative Expenses", "Detention Expenses", "Administrative Expenses", etc., and this area is marked as a low-confidence text area.

[0109] 1.3. Prior information correction.

[0110] The system sends the image slice of the "Administrative Expenses" area and the initially recognized text "Administrative Expenses" to the vision-language correction module. Through the cross-attention mechanism, this module discovers that the visual features of the "理" character side in the image do not match "押" or "赍" better, and finally corrects the text to "Administrative Expenses" and gives an updated confidence level of 0.92.

[0111] 1.4. Introducing semantic placeholders.

[0112] For a completely blank cell, the system generates the **<EMPTY_CELL>** placeholder at its bounding box position.

[0113] II. Multimodal feature encoding and fusion.

[0114] 2.1. Visual feature encoding.

[0115] Input the entire financial statement image into the Swin Transformer-Base model to obtain a deep visual feature.

[0116] 2.2. Spatial feature encoding.

[0117] Obtain the normalized bounding box coordinates of each text / placeholder, such as (0.1, 0.2, 0.3, 0.25) for "Operating Income", and generate its spatial feature vector.

[0118] 2.3. Multimodal prior information fusion.

[0119] Two MLPs (Multi-Layer Perceptron) are used as projection networks to project the two feature vectors into 768 dimensions, respectively.

[0120] III. Structured decoding based on prior guidance.

[0121] The multi-modal guidance feature sequence is concatenated with a task prompt word (such as "generate Markdown table") and input into a Transformer-based table recognition large model (similar to the decoder part of Donut or Pix2Struct).

[0122] The decoder finally generates the output in Markdown format as shown in Figure 4 , accurately restoring the logical structure and content of the table. As shown in the figure, due to the fusion of the features, "revenue" and "¥1,500,000.00" are close in vertical position, and semantically one is the project name and the other is the amount. The model can correctly infer that they belong to different columns but belong to the same row, even though in the original image, due to perspective distortion, the vertical center of "¥1,500,000.00" is slightly lower than "revenue". The model can be calibrated through prior information.

[0123] During model training, to efficiently train the model used in the above cases of the present application, a phased fine-tuning strategy is used for training.

[0124] (1) First phase: freeze the main parameters of the visual backbone network and the language model, and only train the two projection networks in step 2.3. This phase aims to let the model learn how to align and preliminarily fuse information from different modalities.

[0125] (2) Second phase: on the basis of the first phase, unfreeze all model parameters (or set a lower learning rate), and fine-tune the entire model end-to-end. This phase enables the modules to be optimized collaboratively to maximize the accuracy of the final table recognition. This training strategy takes advantage of the powerful capabilities of pre-trained large models while ensuring the efficiency of adaptation to specific tasks.

[0126] The above technical points of the present application work together to achieve efficient, intelligent, and accurate matching of supply and demand in scientific and technological services, significantly improving the service capabilities and user experience of the platform, and having wide practical application value.

[0127] Embodiments of the present application also provide a document table extraction device, comprising:

[0128] An OCR recognition module is configured to perform OCR recognition on the document table image to be processed to obtain text blocks.

[0129] a visual feature encoding module configured to encode visual features of the document table image to obtain deep visual features;

[0130] a semantic prior extraction model configured to encode semantic features of the text sequence of the text block to obtain a semantic feature vector;

[0131] a spatial prior extraction module configured to encode spatial features of the bounding box of the text block to obtain a spatial feature vector;

[0132] a multi-modal fusion module configured to fuse the deep visual features, the semantic feature vector, and the spatial feature vector to obtain multi-modal guided features;

[0133] a structured decoding module configured to decode the multi-modal guided features to obtain a structured representation of the table.

[0134] Further, the OCR recognition module includes at least two heterogeneous optical character recognition engines, each of which performs OCR recognition on the document table image to be processed to obtain recognition results output by the recognition engines.

[0135] The above document table extraction device further includes a prior information acquisition module and a prior correction module. The prior information acquisition module extracts high-confidence prior information from the recognition results of the recognition engines based on a collaborative decision mechanism. The prior correction module corrects a text region whose confidence level does not meet the requirements and acquires corrected prior information.

[0136] Further, the prior correction module further includes a semantic placeholder configuration module. For prior information that is still not reliable after correction processing or a text block whose text sequence is empty, the semantic placeholder configuration module replaces the text sequence with a predefined semantic placeholder.

[0137] The semantic feature extraction module performs semantic feature extraction on each text sequence, including semantic feature extraction on the replaced semantic placeholder. The replaced semantic placeholder is used as an updated text sequence, and the result of the extraction indicates that the text sequence corresponding to the bounding box position is not reliable or is a blank cell.

[0138] Further, the visual encoding module adopts a Swin Transformer visual backbone network; the semantic feature extraction module adopts a language model; the spatial feature extraction module adopts a normalization layer to normalize the boundary box of each text block, and then projects it to a high-dimensional space through a position encoding function to generate a spatial feature vector; the multi-modal guiding module uses two independent and trainable projection networks to linearly project the visual features and prior features into a unified feature dimension, then splices the two projected feature vectors to generate the final multi-modal guiding features; and the structured decoding module adopts a structured decoder based on the Transformer architecture.

[0139] Further, the document table extraction device further comprises a fine-tuning training module configured to fine-tune the model structure in stages.

[0140] Embodiments of the present application also provide an electronic device comprising a processor and a memory, the number of processors can be one or more. The memory as a kind of computer readable storage medium, can be used to store software programs, computer executable programs and modules. The processor executes the various functions of the electronic device and data processing by running the software programs, instructions and modules stored in the memory, so as to realize the document table extraction method of any one of the above embodiments of the present application.

[0141] The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required by a function; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state memory device. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0142] Embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, when the computer program is executed by a processor, the document table extraction method of any one of the embodiments of the present application is realized.

[0143] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer readable storage medium include: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0144] The computer readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, in which a computer readable program code is carried. Such a propagated data signal can take multiple forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can transmit, propagate or transport a program for use by or in connection with an instruction execution system, apparatus or device.

[0145] The embodiments of the present application also provide a computer program product, which, when running on a computer, causes the computer to execute the document table extraction method of any of the above embodiments of the present application.

[0146] The above detailed description of the specific implementation of the present application further explains the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above detailed description is only a specific implementation of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of extracting a table from a document, characterized by, The method comprises the following steps: calling at least two heterogeneous OCR recognition engines to perform OCR recognition on a document table image to be processed respectively, to obtain recognition results, wherein the recognition results comprise text blocks output by the recognition engines and confidence scores of the text blocks; processing the recognition results as follows: S2-1, obtaining a confidence level of each text region according to the confidence scores of the text blocks of the text region; S2-2, regarding a text region with a confidence level higher than a preset level as a first text region, and otherwise as a second text region; S2-3, selecting a text block with the highest confidence score from the text blocks of the first text region as a final text block corresponding to the first text region; performing correction processing on a text sequence of the second text region to obtain a corrected text sequence, performing correction processing on a text block of the second text region according to the corrected text sequence to obtain a corrected text block, and regarding the corrected text block as a final text block corresponding to the second text region; wherein the correction processing on the text sequence of the second text region comprises: S3-1, obtaining a local image of the second text region and extracting visual features of the local image; S3-2, obtaining a preliminary text sequence of the second text region and extracting text features of the preliminary text sequence; the preliminary text sequence is obtained from an existing text sequence in the text block of the second text region; S3-3, inputting the visual features and the text features into a cross-attention mechanism to obtain a cross-attention fusion feature vector; S3-4, performing character-by-character correction on the preliminary text sequence based on the cross-attention fusion feature vector to obtain a corrected text sequence; performing visual feature encoding on the document table image to obtain deep visual features; performing semantic feature encoding on a text sequence of the processed text block to obtain a semantic feature vector; performing spatial feature encoding on a bounding box of the processed text block to obtain a spatial feature vector; performing feature fusion processing on the deep visual features, the semantic feature vector, and the spatial feature vector to obtain a multi-modal guidance feature; performing structured decoding processing on the multi-modal guidance feature to obtain a structured representation of the table.

2. The method of claim 1, wherein, Before performing OCR recognition on the document table image to be processed, the method further comprises: obtaining a document image to be processed, recognizing and extracting a table region in the document image, and obtaining a document table image to be processed; the document image is converted from a document in a real scene, and the conversion mode comprises photographing and scanning.

3. The method of claim 1, wherein, performing correction processing on the text block of the second text region according to the corrected text sequence to obtain a corrected text block, comprising: obtaining a confidence score of the corrected text sequence; if the confidence score of the corrected text sequence is not lower than a confidence threshold, replacing an existing text sequence in the text block of the second text region with the corrected text sequence to obtain a corrected text block; If the confidence score of the corrected text sequence is lower than the confidence threshold, replace the existing text sequence in the text block of the second text region with a predefined semantic placeholder to obtain a corrected text block.

4. The method of claim 3, wherein, The method further includes: replacing the text sequence of the text block of the second text region with a predefined semantic placeholder.

5. The method of claim 1, wherein, The deep visual feature, the semantic feature vector, and the spatial feature vector are subjected to feature fusion processing to obtain a multi-modal guidance feature, including: The deep visual feature, the semantic feature vector, and the spatial feature vector are respectively projected to the same feature dimension, and then the deep visual feature, the semantic feature vector, and the spatial feature vector are subjected to feature splicing along the feature dimension to obtain a multi-modal guidance feature.

6. An apparatus for extracting a table of a document, characterized by comprising: The method further includes: The OCR recognition module includes at least two heterogeneous optical character recognition engines, each of which performs OCR recognition on a document table image to be processed to obtain a recognition result, the recognition result including text blocks output by each recognition engine and confidence scores of the text blocks; The visual feature encoding module is configured to perform visual feature encoding on the document table image to obtain a deep visual feature; The semantic prior extraction model is configured to perform semantic feature encoding on a text sequence of the processed text block to obtain a semantic feature vector; The spatial prior extraction module is configured to perform spatial feature encoding on a bounding box of the processed text block to obtain a spatial feature vector; The multi-modal fusion module is configured to perform feature fusion processing on the deep visual feature, the semantic feature vector, and the spatial feature vector to obtain a multi-modal guidance feature; The structured decoding module is configured to perform structured decoding processing on the multi-modal guidance feature to obtain a structured representation of the table. The processing of the text block includes: S2-1, obtaining a confidence level of each text region according to the confidence scores of the text blocks of each text region of the text region; S2-2, regarding a text region with a confidence level higher than a preset level as a first text region, and otherwise as a second text region; S2-3, selecting a text block with the highest confidence score among the text blocks of the first text region as a final corresponding text block of the first text region; The text sequence of the second text region is subjected to correction processing to obtain a corrected text sequence, and the text block of the second text region is subjected to correction processing according to the corrected text sequence to obtain a corrected text block, which is taken as a final corresponding text block of the second text region. The correction processing of the text sequence of the second text region includes: S3-1, obtaining a local image of the second text region and extracting a visual feature of the local image; S3-2, obtaining a preliminary text sequence of the second text region and extracting a text feature of the preliminary text sequence; the preliminary text sequence is obtained from an existing text sequence in the text block of the second text region; S3-3, inputting the visual feature and the text feature into a cross-attention mechanism to obtain a cross-attention fusion feature vector; S3-4, performing character-by-character correction on the preliminary text sequence based on the cross-attention fusion feature vector, to obtain a corrected text sequence.

7. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The computer program is executed by the processor to implement the document table extraction method in any one of claims 1-5.

8. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the document table extraction method in any one of claims 1-5.

Images