Document structured information coordinate mapping method and device, equipment and storage medium

Abstract

The invention discloses a document structured information coordinate mapping method and device, equipment and a storage medium, and particularly relates to the technical field of computers.The document structured information coordinate mapping method comprises the steps that a document image is obtained, and layout analysis is conducted on the document image through the OCR technology; performing structured extraction on the input text information; performing matching positioning on the structured output result by using a multi-level matching strategy in combination with a semantic understanding technology, calculating an accurate character offset in the input text information, and mapping the accurate character offset to an image coordinate system; and drawing the accurate structured data into a form, and integrating the form with the document image to generate a visual document structured information interface. According to the method, the problem that in the prior art, coordinate mapping is lacked when document structured information extraction is carried out through OCR and LLM is solved, the information retrieval speed is increased, the data processing accuracy is ensured, the accuracy and efficiency during complex document processing are improved, and the working efficiency of a user is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device, equipment and storage medium for mapping document structured information coordinates. Background Art

[0002] In the field of document analysis, technologies using OCR (Optical Character Recognition) and LLM (Large Language Model) are developing rapidly and making significant progress. Modern OCR technology can not only efficiently recognize and extract text, but also continuously improve the ability to parse complex document structures (such as tables, charts, and images). Combined with LLM, these technologies further enhance the ability to understand and process document content, making it more efficient to extract key information from unstructured or semi-structured documents.

[0003] At present, although OCR technology and LLM models have made significant progress in document analysis and information extraction, there are still many challenges in mapping text and other information to specific coordinate locations. The latest research points out that the existing OCR systems and LLM models are not yet able to accurately provide coordinate information for each text fragment or data element when processing complex document structures, which limits their accuracy and efficiency in practical applications. For example, although GPT-4 already has the ability to extract structured information from images and PDF documents and can understand the semantic content of documents, the information it generates lacks precise coordinate annotations, which makes it difficult for users to quickly verify and confirm the accuracy of data in complex document processing.

[0004] In actual business, if complex documents can achieve coordinate mapping, it will help business personnel quickly confirm the correctness of the output results, and can judge the accuracy of the data by the relative coordinate position. Therefore, if the coordinate mapping of information is realized during the document analysis process, the efficiency and accuracy of data verification will be greatly improved. Business personnel can check and verify the correctness of data more intuitively, and quickly locate and correct errors during data processing. Summary of the invention

[0005] The present invention provides a method and device for document structured information coordinate mapping, equipment and storage medium, for solving the problem of lack of coordinate mapping when extracting document structured information using OCR and LLM in the prior art.

[0006] In order to achieve the above object, the present invention provides the following technical solutions: A method for mapping document structured information coordinates comprises the following steps: Obtain document images, use OCR technology to analyze the document images, identify the content and structural elements of the document, determine the reading order and character-level coordinate information, and obtain input text information; Use prompt generation technology to extract structured input text information and obtain structured output results; Use a multi-level matching strategy combined with semantic understanding technology to match and locate the structured output results, calculate the precise character offset in the input text information and map it to the image coordinate system to obtain precise structured data; Drawing the precise structured data into a form and integrating it with the document image to generate a visual document structured information interface; The multi-level matching strategy includes: Perform full-text accurate matching on the structured output results and input text information. When full-text accurate matching fails, obtain the known character offset value and perform line-level matching. Extract the line text content from the target line based on the known character offset value, and search for the target text segment at the known character offset value for local accurate matching. If line-level matching fails, paragraph matching is performed, and semantic-based retrieval is performed within the paragraph. The pre-trained language model is used to calculate the semantic similarity between the target text and each clause in the paragraph to locate the most relevant text fragment; By using word embedding technology and cosine similarity comparison, the similarity at the semantic level is captured, and approximate text matching is performed on the structured output results to eliminate OCR recognition errors.

[0007] According to one embodiment of the present invention, the method also includes optimizing the structured output results through knowledge graph and entity matching, and entity matching ensures that the extracted information is consistent with the expected entities by calculating the similarity between the entities extracted from the structured output results and the standard entities in the knowledge graph.

[0008] According to an embodiment of the present invention, a rectangular area corresponding to each structured field in the form is drawn on the document image according to the precise structured data, and the structured field is bidirectionally associated with the corresponding rectangular area.

[0009] The present invention also provides a document structured information coordinate mapping device, comprising: The OCR module is used to obtain the document image, use the OCR technology to analyze the document image, identify the content and structural elements of the document, determine the reading order and character-level coordinate information, and obtain the input text information; The LLM information extraction module uses prompt generation technology to perform structured extraction of input text information and obtain structured output results; The text matching and coordinate calculation module uses a multi-level matching strategy combined with semantic understanding technology to match and locate the structured output results, calculates the exact character offset in the input text information and maps it to the image coordinate system to obtain accurate structured data; A structure visualization module is used to draw the precise structured data into a form and integrate it with the document image to generate a visual document structured information interface; Among them, the text matching and coordinate calculation module includes: A full-text matching unit is used to perform full-text accurate matching of structured output results and input text information and obtain known character offset values; A row-level matching unit is used to extract row text content from a target row based on a known character offset value, and to search for a target text segment at the known character offset value for local precise matching; The paragraph matching unit is used to perform semantic-based retrieval within the paragraph range, using the pre-trained language model to calculate the semantic similarity between the target text and each clause in the paragraph, and locate the most relevant text fragment; Approximate text matching unit, which is used to capture semantic similarity through word embedding technology and cosine similarity comparison, and perform approximate text matching on structured output results to eliminate OCR recognition errors; The coordinate calculation unit is used to calculate the precise character offset between the structured output result and the input text information and map it to the image coordinate system to obtain precise structured data.

[0010] According to one embodiment of the present invention, the LLM information extraction module also includes a knowledge graph and an entity matching pair unit for optimizing the structured output results. The entity matching unit ensures that the extracted information is consistent with the expected entities by calculating the similarity between the entities extracted from the structured output results and the standard entities in the knowledge graph.

[0011] According to one embodiment of the present invention, the structure visualization module further includes a drawing unit for drawing a rectangular area corresponding to each structured field in the form on the document image according to the precise structured data, and bidirectionally associating the structured field with the corresponding rectangular area.

[0012] A document structured information coordinate mapping device, comprising: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement any of the above methods when executing the executable instructions.

[0013] A computer-readable storage medium stores computer instructions, wherein the computer instructions are used to enable a processor to implement any of the above methods when executed.

[0014] In the above technical solution, the technical effects and advantages provided by the present invention are: 1. The present invention can effectively improve the user experience when searching for information. Through visualization, users can intuitively see the exact location of structured information in the document, which simplifies the process of information search and location, provides intuitive guidance for subsequent document analysis and processing, and greatly improves the user's work efficiency; 2. In the process of text matching and coordinate calculation, the present invention introduces the concept of relative position weighting, which further enhances the accuracy and consistency of text positioning by quantifying the relative position relationship between structured information and its context, which not only consolidates the efficiency foundation of business processing, but also ensures the reliability and robustness of the entire processing chain; 3. The present invention can efficiently process various complex document structures, especially when faced with highly structured documents such as forms and tables. Through customized task models, the present invention can significantly improve processing performance and output accuracy. It shows extremely high applicability when faced with scenarios that require frequent reference to original documents, such as contract review, form verification and other document-intensive tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flow chart of the method for mapping the coordinates of the document structured information of the present invention; Figure 2 It is a flow chart of the multi-level matching strategy in the present invention; Figure 3 It is a schematic block diagram of the document structured information coordinate mapping device of the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] In the description of the embodiments of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limitations on the embodiments of the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0019] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0020] In the embodiments of the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "above" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0021] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0022] Combine the following Figure 1-Figure 3 Embodiments of the present invention are described.

[0023] like Figure 1-Figure 2 As shown, the present embodiment discloses a method for coordinate mapping of document structured information, which is based on OCR and LLM technology, and is intended to solve the problem of lack of coordinate mapping when extracting document structured information in the prior art, comprising: S10: acquiring a document image, using OCR technology to perform layout analysis on the document image, identifying the content and structural elements of the document, and determining the reading order and character-level coordinate information, to obtain input text information; S20: using Prompt generation technology to perform structured extraction on the input text information, to obtain a structured output result; S30: using a multi-level matching strategy combined with semantic understanding technology to match and locate the structured output result, calculate the precise character offset in the input text information and map it to the image coordinate system, to obtain precise structured data; S40: drawing the precise structured data into a form and integrating it with the document image, to generate a visual document structured information interface.

[0024] Among them, in step S10, this embodiment uses the multimodal pre-trained model LayoutLM as the OCR model, which can process text, layout and visual information at the same time, and has significant advantages in layout analysis and joint processing of multimodal information. In step S20, this embodiment uses the Prompt generation technology to perform structured extraction of input text information, and uses pre-designed prompt words to guide LLM (large language model) to extract the required information from the text and generate structured JSON output results. At the same time, this embodiment also further improves the accuracy and efficiency of the Prompt generation process through entity matching (Entity Matching) and knowledge graph (Knowledge Graph) during step S20. Entity matching can determine the degree of matching by calculating the similarity between the entities extracted from the text and the standard entities in the knowledge base. The knowledge graph can verify and improve the deficiencies of LLM in deterministic relevance knowledge through relationship constraints, and can also be used to generate specific problem instructions for Prompt instructions, thereby achieving higher extraction accuracy. The similarity can be calculated by using full-text matching technology or embedding calculation in a vector database. In this embodiment, the cosine similarity calculation method is used, and the calculation formula is as follows: .

[0025] In step S30 of the present embodiment, the efficiency of text positioning and information extraction is significantly improved through hierarchical query and semantic similarity calculation. The specific implementation method includes: S31: performing full-text accurate matching of the structured output results and the input text information. When the full-text accurate matching fails, obtaining the known character offset value, and performing line-level matching, extracting the line text content from the target line based on the known character offset value, and searching for the target text segment at the known character offset value for local accurate matching; S32: If the line-level matching fails, paragraph matching is performed, and semantic-based retrieval is performed within the paragraph range. The semantic similarity between the target text and each clause in the paragraph is calculated using the pre-trained language model to locate the most relevant text segment; S33: By comparing the word embedding technology with the cosine similarity, the similarity at the semantic level is captured, and approximate text matching is performed on the structured output results to eliminate OCR recognition errors; S34: After locating the target text, it is necessary to calculate its exact character offset with the input text information and map it to the image coordinate system to obtain accurate structured data.

[0026] Furthermore, in step S40 of this embodiment, a rectangular area corresponding to each structured field in the form is drawn on the document image according to the precise structured data, and the structured field is bidirectionally associated with the corresponding rectangular area, thereby realizing a highly interactive document automation interface. When the user selects any form field, the system can calculate the position of the corresponding rectangular area and automatically scroll the canvas to make it enter the visible area, thereby eliminating the need for searching and improving navigation efficiency.

[0027] Through this highly integrated and visually interactive coordinate mapping method, the document structured information and the corresponding original information can be intuitively presented to the user, making it easier for the user to understand the precise meaning and context of each structured field in the original document. This not only significantly improves the user experience, but also lays the foundation for subsequent document analysis, editing and verification processes.

[0028] like Figure 3 As shown, the present invention also provides an embodiment of a document structured information coordinate mapping device, which specifically includes: The OCR module 10 is used to obtain a document image, perform layout analysis on the document image using OCR technology, identify the content and structural elements of the document, determine the reading order and character-level coordinate information, and obtain input text information; LLM information extraction module 20, using prompt generation technology to perform structured extraction on input text information to obtain structured output results; The text matching and coordinate calculation module 30 uses a multi-level matching strategy combined with semantic understanding technology to match and locate the structured output results, calculates the precise character offset in the input text information and maps it to the image coordinate system to obtain precise structured data; A structure visualization module 40 is used to draw the precise structured data into a form and integrate it with the document image to generate a visualized document structured information interface; Wherein, the text matching and coordinate calculation module 30 includes: A full text matching unit 31 is used to perform full text accurate matching of the structured output result and the input text information and obtain a known character offset value; A row-level matching unit 32 is used to extract row text content from a target row based on a known character offset value, and to search for a target text segment at the known character offset value for local precise matching; The paragraph matching unit 33 is used to perform semantic-based retrieval within the paragraph range, calculate the semantic similarity between the target text and each clause in the paragraph using the pre-trained language model, and locate the most relevant text segment; An approximate text matching unit 34 is used to capture the similarity at the semantic level by comparing the word embedding technology with the cosine similarity, and to perform approximate text matching on the structured output results to eliminate OCR recognition errors; The coordinate calculation unit 35 is used to calculate the precise character offset between the structured output result and the input text information and map it to the image coordinate system to obtain precise structured data.

[0029] Furthermore, the LLM information extraction module 20 also includes a knowledge graph 21 and an entity matching unit 22, which are used to optimize the structured output results. The entity matching unit 22 ensures that the extracted information is consistent with the expected entities by calculating the similarity between the entities extracted from the structured output results and the standard entities in the knowledge graph 21.

[0030] Furthermore, the structure visualization module 40 also includes a drawing unit 41, which is used to draw a rectangular area corresponding to each structured field in the form on the document image according to the precise structured data, and bidirectionally associate the structured field with the corresponding rectangular area.

[0031] In some embodiments, the method disclosed in the present invention may be applied to some devices, which need to include: a processor and a memory for storing processor executable instructions, wherein the processor is configured to implement the above method when executing the executable instructions.

[0032] In addition, the method disclosed in the present invention may also be implemented as a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the above method when executed by a processor.

[0033] In summary, the present invention discloses a method and device for document structured information coordinate mapping, equipment and storage medium, which solves the problem of lack of coordinate mapping when extracting document structured information using OCR and LLM in the prior art, and overcomes the limitations of traditional NLP (Natural Language Processing) methods in complex text understanding, and achieves more accurate extraction through context-aware preset prompt words. The efficiency and accuracy of text positioning and information extraction are significantly improved by combining semantic understanding with character-level matching through a multi-level matching strategy. Accurate coordinate calculation lays the foundation for visualization and interactive functions, and supports efficient human-computer interaction. The problem of lack of intuitive visualization in traditional document processing systems is solved, and a more intuitive interface is provided through the combination of images and data, thereby improving user information acquisition and operation efficiency. The combination of these key technical points significantly improves the efficiency and accuracy of document automation processing, especially in the fields of finance, insurance, law, etc., which have extremely high requirements for document processing accuracy. The present invention provides strong technical support, improves work efficiency and reduces human errors.

[0034] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for mapping document structured information coordinates, characterized in that: The following steps are involved: Obtain document images, use OCR technology to analyze the document images, identify the content and structural elements of the document, determine the reading order and character-level coordinate information, and obtain input text information; Use prompt generation technology to extract structured input text information and obtain structured output results; Use a multi-level matching strategy combined with semantic understanding technology to match and locate the structured output results, calculate the precise character offset in the input text information and map it to the image coordinate system to obtain precise structured data; Drawing the precise structured data into a form and integrating it with the document image to generate a visual document structured information interface; The multi-level matching strategy includes: Perform full-text accurate matching on the structured output results and input text information. When full-text accurate matching fails, obtain the known character offset value and perform line-level matching. Extract the line text content from the target line based on the known character offset value, and search for the target text segment at the known character offset value for local accurate matching. If line-level matching fails, paragraph matching is performed, and semantic-based retrieval is performed within the paragraph. The pre-trained language model is used to calculate the semantic similarity between the target text and each clause in the paragraph to locate the most relevant text fragment; By using word embedding technology and cosine similarity comparison, the similarity at the semantic level is captured, and approximate text matching is performed on the structured output results to eliminate OCR recognition errors.

2. The method according to claim 1, characterized in that The method also includes optimizing the structured output results through knowledge graphs and entity matching. Entity matching ensures that the extracted information is consistent with the expected entities by calculating the similarity between the entities extracted from the structured output results and the standard entities in the knowledge graph.

3. The method according to claim 1, characterized in that A rectangular area corresponding to each structured field in the form is drawn on the document image according to the precise structured data, and the structured field is bidirectionally associated with the corresponding rectangular area.

4. A document structured information coordinate mapping device, characterized in that: include: The OCR module is used to obtain the document image, use the OCR technology to analyze the document image, identify the content and structural elements of the document, determine the reading order and character-level coordinate information, and obtain the input text information; The LLM information extraction module uses prompt generation technology to perform structured extraction of input text information and obtain structured output results; The text matching and coordinate calculation module uses a multi-level matching strategy combined with semantic understanding technology to match and locate the structured output results, calculates the exact character offset in the input text information and maps it to the image coordinate system to obtain accurate structured data; A structure visualization module is used to draw the precise structured data into a form and integrate it with the document image to generate a visual document structured information interface; Among them, the text matching and coordinate calculation module includes: A full-text matching unit is used to perform full-text accurate matching of structured output results and input text information and obtain known character offset values; A row-level matching unit is used to extract row text content from a target row based on a known character offset value, and to search for a target text segment at the known character offset value for local precise matching; The paragraph matching unit is used to perform semantic-based retrieval within the paragraph range, using the pre-trained language model to calculate the semantic similarity between the target text and each clause in the paragraph, and locate the most relevant text fragment; Approximate text matching unit, which is used to capture semantic similarity through word embedding technology and cosine similarity comparison, and perform approximate text matching on structured output results to eliminate OCR recognition errors; The coordinate calculation unit is used to calculate the precise character offset between the structured output result and the input text information and map it to the image coordinate system to obtain precise structured data.

5. The device according to claim 4, characterized in that The LLM information extraction module also includes a knowledge graph and an entity matching pair unit, which is used to optimize the structured output results. The entity matching unit ensures that the extracted information is consistent with the expected entities by calculating the similarity between the entities extracted from the structured output results and the standard entities in the knowledge graph.

6. The device according to claim 4, characterized in that The structure visualization module also includes a drawing unit for drawing a rectangular area corresponding to each structured field in the form on the document image according to the precise structured data, and bidirectionally associating the structured field with the corresponding rectangular area.

7. A document structured information coordinate mapping device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method according to any one of claims 1 to 3 when executing the executable instructions.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method according to any one of claims 1 to 3 when executed.

Images