Adaptive document processing method and system

Abstract

This invention discloses an adaptive document processing method and system. The adaptive document processing method includes: receiving a document image and preprocessing it; extracting the text of each part of the document image and the absolute spatial coordinates of the bounding boxes corresponding to each part of the text, and dynamically detecting the root bounding box of the document image; starting from the root bounding box, using a corresponding template to obtain the relative spatial relationships between all bounding boxes; generating structured output, and associating each part of the text with each bounding box. This invention can recognize non-standard document images, making document image recognition more flexible.

Description

Instruction Manual 1 Adaptive Document Processing Method and System Technical Field This invention relates to the technical field of document processing, and particularly to an adaptive document processing method and system that defines spatial relationships without code and handles multiple document types. Background Technology Document recognition is a widely used function. For example, in business cooperation, it is necessary to input important content from quotation forms sent by the other party into the computer, which requires the use of document recognition functions. Another example is that when finance departments are organizing paper invoices, they need to extract information from all invoices and organize corresponding reports. Besides these examples, there are many other occasions that require document recognition, but existing document recognition technologies have the following problems. Traditional OCR (Optical Character Recognition) systems typically rely on fixed coordinates. However, in practical applications, documents often undergo dynamic layout changes due to the following reasons: Physical deformation: scaling, rotation, and perspective distortion during scanning / capturing (e.g., tilted documents captured by mobile phones); Format diversity: multiple different formats of text content exist within the same type of document (e.g., invoice formats from different companies). This leads to inefficiencies in existing OCR systems when processing non-standard document images, unstructured data, or multi-page documents with different formats. Users need to programmatically solve these problems, requiring them to possess programming skills. Therefore, providing an adaptive document processing method and system capable of handling non-standard document images or unstructured data is a pressing technical problem. Summary of the Invention To address the inefficiency of processing non-standard document images in existing technologies, this invention proposes an adaptive document processing method and system. The adaptive document processing method of this invention includes: receiving a document image and preprocessing it; extracting the text of each part of the document image and the absolute spatial coordinates of the bounding boxes of each part of the text, and dynamically detecting the root bounding box of the document image; starting from the root bounding box, using the corresponding template and the absolute spatial coordinates of the bounding boxes of each part of the text to obtain the relative spatial relationships between all bounding boxes; generating structured output and associating each part of the text with its corresponding bounding box. Furthermore, the preprocessing includes at least one of the following: skew correction, noise reduction, contrast optimization, and binarization of the document image. Furthermore, the absolute spatial coordinates include the coordinates of the upper left and lower right corners of the bounding box. Furthermore, the root bounding box of the dynamically detected document image is the bounding box of each preset salient region of the document image, and the root bounding box is selected from these regions.Further, obtaining the relative spatial relationships between all bounding boxes includes: taking the root box as the current parent box; identifying the alignment symbol of the current parent box in the template, and the alignment relationship between the current parent box and its child boxes (HK 30135834 A Specification 3); based on the alignment relationship, finding all child boxes that satisfy the alignment relationship; when only one child box is found, taking that child box as the child box of the current parent box; when multiple child boxes are found, selecting one of the child boxes as the child box of the current parent box according to a preset rule; calculating the relative coordinates between the child box and the parent box, and returning to the step of identifying the alignment symbol of the current parent box in the template to continue searching for corresponding boxes, until all corresponding boxes have corresponding relative coordinates. Further, the preset rule is: the child box closest to the current parent box is taken as the child box of the current parent box, or a user-defined priority is used to select the child box of the current parent box. Furthermore, when the template includes derived box rules, obtaining each current parent box includes the following steps: Calculating the boundary of the derived box of the current parent box according to the derived box rules, recording the coordinates of the derived box, and adding a derived box identifier; When generating structured output, performing verification operations on the text within the derived box based on the derived box identifier. Furthermore, when processing multi-page documents, the document is split into multiple document images on a page-by-page basis, and preprocessing to structured output is performed on each document image separately. The adaptive document processing system proposed in this invention uses the adaptive document processing method described above to process documents. The adaptive document processing system includes: an image preprocessing module for preprocessing document images; an optical character recognition engine for extracting text from each part of the document image and the bounding box coordinates of the document image; a template parser, including a template construction module for user-defined templates and a result preview module for document images processed based on the user-defined module; and a structured output generator for generating structured output and associating the text of each part with its corresponding boxes. Furthermore, it also includes an AI enhancement module, which learns and optimizes the output of the optical character recognition engine and performs verification operations in conjunction with a large language model. Furthermore, the adaptive document processing system adopts a distributed processing system and outputs to downstream systems via API. This invention, by starting with the root box and searching for the relative spatial relationships between boxes, enables accurate recognition even for non-standard document images. Moreover, this invention allows users to edit templates and recognize documents based on custom templates. Furthermore, when processing multi-page documents, it can dynamically match templates and perform recognition separately, then aggregate and parse the results according to document type or page order. The following is a detailed description of the invention in conjunction with embodiments and accompanying drawings, wherein: Figure 1 is a basic flowchart of an embodiment of the invention.Figure 2 is a schematic diagram of template code according to an embodiment of the present invention. Figure 3 is a schematic diagram of structured data code according to an embodiment of the present invention. Detailed Description To make the technical problems, technical solutions, and beneficial effects to be solved by the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only for explaining the present invention and are not intended to limit the present invention. Therefore, a feature pointed out in this specification is used to describe one feature of one embodiment of the present invention, and does not imply that every embodiment of the present invention must have the described feature. Furthermore, it should be noted that this specification describes many features. Although some features may be combined together to illustrate possible system designs, these features may also be used in other combinations not explicitly stated. Therefore, unless otherwise stated, the described combinations are not intended to be limiting. As shown in Figure 1, in a basic embodiment, the adaptive document processing method of the present invention includes the following steps. The process involves receiving and preprocessing a document image; extracting the text and its bounding boxes' absolute spatial coordinates from each part of the document image, and dynamically detecting the root bounding box; the absolute spatial coordinates of each text part and its bounding box can be extracted using an Optical Character Recognition (OCR) engine, a mature and existing text recognition engine. Besides OCR, other existing text recognition technologies, such as handwriting recognition, can also be used as needed to obtain this information. Starting from the root bounding box, the relative spatial relationships between all boxes are obtained using the corresponding template and the absolute spatial coordinates of the bounding boxes of each text part; a structured output is generated, and each text part is associated with its corresponding bounding box. This invention uses a single frame as the root frame and analyzes the text in each part of the document image according to HK 30135834 A specification 6. It generates a hierarchical structure from all content of the document image and records the relative spatial relationships of the frames in the hierarchical structure. These relative spatial relationships can be between the current frame and the root frame, or between the current frame and its parent frame. This allows for accurate identification of the corresponding document image even when processing non-standard document images, improving processing efficiency. In one specific embodiment, the preprocessing of this invention includes at least one of the following: image tilt correction, noise reduction, contrast optimization, and binarization. Image tilt correction can use Hough transform to detect image tilt angle and perform rotation correction to minimize document image deviation. Noise reduction uses a medium filter to remove salt-and-pepper noise or an adaptive threshold to eliminate shadows. Contrast optimization uses histogram equalization to enhance text readability. Binarization converts the image to black and white mode, reducing the complexity of subsequent processing.In addition to the preprocessing described above, those skilled in the art can perform other image preprocessing operations as needed. In one specific embodiment, the absolute spatial coordinates include the coordinates of the upper left corner and the lower right corner of the bounding box. In other embodiments, the absolute spatial coordinates may also include the coordinates of the other two vertices of the text bounding box, or the coordinates of more than two vertices. Absolute spatial coordinates can be used to locate the root box. The root box of this invention is dynamically detected. The root box is determined by dynamically detecting the bounding box located within a preset salient area at the edge of the document image. The preset salient area is usually located at the edge of the document image, usually at the upper edge of the document image, such as the bounding box of the "invoice number" in the upper left corner, which is used as the root box. The root box is the starting point for document image layout analysis. HK 30135834 A Specification 7 Absolute spatial coordinates can also be used to verify alignment relationships. Templates usually provide alignment symbols, such as the symbol "_" indicating bottom alignment. Then, the child box corresponding to the parent box can be found by verifying whether the y-values ​​of the corresponding coordinates of the parent box and the child box conform to the rules. Absolute spatial coordinates can also be used to calculate the relative coordinates of a child frame and its corresponding parent frame. When generating structured output, the absolute coordinates of a child frame can be converted into the proportion of its corresponding parent frame, for example, the width of the child frame is 80% of the parent frame. In one embodiment, obtaining the relative relationships between the bounding boxes of all text includes the following steps: First, starting from the root frame, the root frame is taken as the current parent frame; the alignment symbol of the current parent frame in the template is identified, as well as the alignment relationship between the current parent frame and its child frames; based on the alignment relationship between the parent and child frames, all child frames that satisfy the alignment relationship are found; when only one child frame is found, that child frame is taken as the child frame of the current parent frame; when multiple child frames are found, one of the child frames is selected as the child frame of the current parent frame according to a preset rule; the relative coordinates of the child frame and the parent frame are calculated, and the child frame is taken as the current parent frame. The process returns to the step of identifying the alignment symbol of the current parent frame in the template to continue searching for corresponding frames until all corresponding frames have corresponding relative coordinates. As shown in Figure 2, this figure illustrates an invoice template of the present invention. In the template, the header is aligned to the top right and located within the page, with a width of 40%-80% of the page width. The table_row is located below the header, with a width consistent with the parent frame's width and a height of 10% of the parent frame's height. HK 30135834 A Specification 8 The alignment symbols used in this invention can be used to replace fixed coordinates with relative coordinates during structured output, thereby adapting to layout changes. The alignment relationships of this invention include, but are not limited to, child frames located below the parent frame, child frames located above the parent frame, child frames located to the right of the parent frame, child frames located to the left of the parent frame, etc.Alignment symbols can be represented by various symbols, including but not limited to: the alignment symbol _ indicating that the child box is below the parent box, the alignment symbol ^ indicating that the child box is above the parent box, the alignment symbol <| indicating that the child box is to the left of the parent box, the alignment symbol |> indicating that the child box is to the right of the parent box, etc. Those skilled in the art can use other alignment symbols to represent the corresponding alignment relationships as needed. The following is a specific example to illustrate the layered box discovery process of the present invention. First, the root box is identified by dynamically detecting a preset salient area of ​​the text image, such as the box containing "invoice number" in the upper left corner as the root box. If not found, the box located at the edge of the page is used as the root box, such as the box at the top center of the text image. Then, the parent-child relationship in the template is traversed. For example, the current parent box is "header", and the child box of the current parent box is "invoice_number". The OCR result is matched according to the alignment symbols of the current parent box and child box. For example, if the child box is aligned with the right side of the parent box, and "invoice_number is to the right of header", then the bounding box of invoice_number is a child box of the bounding box of header. OCR results refer to the results recognized by the OCR engine. The OCR engine recognizes the text of each part and the bounding boxes of each part of the text. If the match fails, for example, if no child box aligned with the right side of the parent box is found, the match is re-matched according to the tolerance range, for example, according to the coordinate range of ±5%. HK 30135834 A Specification 9 After all parent boxes and child boxes are found, a nested structure is generated, for example, a hierarchical JSON file is output. For example, header→invoice_number→table_row is a nested structure. In the above embodiment, when multiple child boxes are found, the child box of the current parent box is determined according to the preset rules. The preset rules can take the child box closest to the current parent box as the child box of the current parent box, or select the child box of the current parent box according to the user-defined priority. As shown in Figure 3, this figure shows a specific embodiment of the structured data of the present invention. It can be seen from the structured data that this embodiment has structured the invoice number, date and specific invoice content. In another embodiment, the adaptive document processing method of the present invention includes the following steps. The document image is received and preprocessed; the absolute spatial coordinates of the text and text bounding boxes of each part of the document image are extracted, and the root bounding box of the document image is dynamically detected; starting from the root bounding box, the relative spatial relationships between all boxes are obtained using the corresponding template; the derived bounding boxes of the corresponding boxes are generated using the definition of derived bounding boxes in the corresponding template. Structured output is generated, and the text of each part is associated with its corresponding bounding box. The preprocessing in this embodiment is similar to the preprocessing method in previous embodiments.The difference between this embodiment and the previous embodiment is that this embodiment generates derived boxes. The purpose of derived boxes is to correspond associated text content with a box, so that during structured output, based on the identifier of the derived box, the associated text content can be automatically validated or subjected to other specific operations. When the template includes derived box rules, obtaining each current parent box includes the following steps: According to the derived box rules defined in the corresponding template, calculate the boundary of the derived box of the current parent box, record the coordinates of the corresponding vertex of the derived box relative to the corresponding vertex of the parent box, and add a derived box identifier; the coordinates of the derived box are calculated proportionally based on the absolute coordinates of the parent box. For example, the absolute coordinates of the parent box are (x1=100, y1=200, x2=300, y2=400). If the derived box rule is the right 20% area, then the coordinates of the derived box are (x1=260, y1=200, x2=300, y2=400). When generating structured output, validation operations are performed on the text within the derived boxes based on the derived box identifiers. The derived boxes allow for automatic validation of certain text. For example, if the text in the parent box is an amount, the validation operation of the derived box is to verify whether it conforms to the text format required for an amount. This is merely an example; those skilled in the art can customize the derived boxes. In a further embodiment, when processing multi-page documents, the present invention splits the document into multiple document images on a page-by-page basis, performs preprocessing to structured output on each document image, and after obtaining the structured output, performs contextual analysis and merges the structured output of the multi-page document, or performs key-based aggregation of the structured output of the multi-page document according to type. For example, if the first page is "Invoice Header" and the second page is "Details Sheet," the page order is preserved during merging in the output JASON document. As another example, if the structured output of the multi-page document contains invoices and contracts, the corresponding invoices and contracts can be matched and merged in order, i.e., key-based aggregation. This invention also protects an adaptive document processing system that processes documents using the adaptive document processing methods described above. The adaptive document processing system includes an image preprocessing module, an optical character recognition engine, a template parser, and a structured output generator. The image preprocessing module preprocesses the document image; the optical character recognition engine extracts the text from each part of the document image and the bounding box coordinates of the document image; the template parser includes a template construction module for user-defined templates and a result preview module for document images processed based on user-defined templates; and the structured output generator generates structured output and associates the text of each part with its corresponding bounding box.In a further embodiment, the adaptive document processing system, in addition to the modules mentioned above, also includes an AI enhancement module. This AI enhancement module learns from and optimizes the output of the optical character recognition engine, and can also perform verification operations in conjunction with a large language model. The template building module of this invention can employ a GUI template builder. Users can define corresponding alignment symbols in the GUI template builder, and the system can use the AI ​​enhancement module to assist in template layout recommendations. For example, after the AI ​​enhancement module learns from the output of the optical character recognition engine, the GUI template builder uses the learning results of the AI ​​enhancement module as the user's template recommendation. Further optimization is possible, such as highlighting the root box of the corresponding text image in the GUI template builder. Furthermore, the GUI template builder can provide a drag-and-drop editing interface. Based on the user's drag-and-drop placement, it automatically generates the corresponding JSON code and visualizes the connection (HK 30135834 A specification 12). Moreover, the GUI template builder of this invention allows the system to display the parsing results in real time when the user modifies the template, such as highlighting the matched text, supporting instant debugging. When deploying the adaptive document processing system of this invention, a distributed processing system can be used, and the data can be output to downstream systems via API. Downstream systems include, but are not limited to, ERP systems, databases, AI platforms, reporting tools, etc. For example, after a paper invoice is photographed and processed using the adaptive document processing method of this invention, the corresponding structured output data can be imported into the financial module of an ERP system, imported into a database, or input into an AI platform or reporting tool. Besides invoices, this invention can also process contracts, registration forms, waybills, and other document images. Each document image can be configured with a dedicated template, and all documents can be processed and parsed in parallel using a cloud distributed architecture. The above description is merely a preferred embodiment of this invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention. HK 30135834 A Claim 1 1. An adaptive document processing method, characterized in that it includes: receiving a document image and performing preprocessing; extracting the text of each part of the document image and the absolute spatial coordinates of the bounding boxes of each part of the text, and dynamically detecting the root box of the document image; starting from the root box, using the corresponding template and the absolute spatial coordinates of the bounding boxes of each part of the text, to obtain the relative spatial relationship between all bounding boxes; generating structured output, and associating the text of each part with the corresponding bounding box.2. The adaptive document processing method as described in claim 1, wherein the preprocessing includes at least one of the following: skew correction, noise reduction, contrast optimization, and binarization of the document image. 3. The adaptive document processing method as described in claim 1, wherein the absolute spatial coordinates include the coordinates of the upper left corner and the lower right corner of the bounding box. 4. The adaptive document processing method as described in claim 1, wherein the root bounding box of the dynamically detected document image is selected from the bounding boxes of preset salient regions located at the edges of the document image. 5. The adaptive document processing method as described in claim 1, characterized in that obtaining the relative spatial relationship between all bounding boxes includes: taking the root box as the current parent box; identifying the alignment symbol of the current parent box in the template, and the alignment relationship between the current parent box and its child boxes; based on the alignment relationship, finding all child boxes that satisfy the alignment relationship; HK 30135834 A Claim 2 When the number of found child boxes is one, taking that child box as the child box of the current parent box; when the number of found child boxes is multiple, selecting one of the child boxes as the child box of the current parent box according to a preset rule; calculating the relative coordinates between the child box and the parent box, and taking that child box as the current parent box and returning to the step of identifying the alignment symbol of the current parent box in the template to continue searching for corresponding boxes, until all corresponding boxes have corresponding relative coordinates. 6. The adaptive document processing method as described in claim 5, characterized in that the preset rule is: the child box closest to the current parent box is taken as the child box of the current parent box, or the child box of the current parent box is selected based on a user-defined priority. 7. The adaptive document processing method as described in claim 5, characterized in that, when the template includes derived box rules, obtaining each current parent box includes the following steps: calculating the boundary of the derived box of the current parent box according to the derived box rules, recording the coordinates of the corresponding vertex of the derived box relative to the corresponding vertex of the parent box, and adding a derived box identifier; when generating structured output, performing a verification operation on the text within the derived box based on the derived box identifier. 8. The adaptive document processing method as described in any one of claims 1 to 7, characterized in that, when processing multi-page documents, the document is split into multiple document images on a page-by-page basis, and preprocessing to structured output is performed on each document image respectively, and after obtaining the structured output, the structured output of the multi-page document is analyzed in context and merged, or the structured output of the multi-page document is aggregated by type.9. An adaptive document processing system, characterized in that it processes documents using the adaptive document processing method as described in any one of claims 1 to 8, the adaptive document processing system comprising: an image preprocessing module for preprocessing document images; an optical character recognition engine for extracting text from each part of the document image and the bounding box coordinates of each part of the text; a template parser, including a template building module for user-defined templates and a result preview module for document images processed based on the user-defined module; and a structured output generator for generating structured output and associating each part of the text with its corresponding bounding box. 10. The adaptive document processing system as described in claim 9, further comprising: an AI enhancement module for learning and optimizing the output results of the optical character recognition engine and performing verification operations in conjunction with a large language model. 11. The adaptive document processing system as described in claim 9, wherein the adaptive document processing system employs a distributed processing system and outputs to downstream systems via an API. HK 30135834 A Instruction Manual Appendix 1 Figure 1 HK 30135834 A Instruction Manual Appendix 2 Figure 2 Figure 3 HK 30135834 A.

Images