A document catalog extraction method and system based on a generative language model
Patent Information
- Application Number
- CN202410255048.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-06
- Publication Date
- 2026-10-09
- Estimated Expiration
- 2044-03-06
AI Technical Summary
其一，文档级文本长度较长，目录结构往往分散于其中，造成了长距稀疏的信息分布，这对目录关联的追踪造成挑战
[0028] 1) This invention adopts a generative document directory extraction method, which uses a global document structure representation to transform document directory extraction into a hierarchical structure generation task that involves one traversal, effectively ensuring the generalization ability of the method.
Smart Images
Figure CN118170863B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a document directory extraction method and system based on a generative language model, belonging to the field of natural language processing technology. Background Technology
[0002] Document table of contents extraction is a crucial task in document understanding. Given a list of line-level text fragments, the goal of document table of contents extraction is to reconstruct it into a hierarchical directory tree consisting of heading nodes and paragraph nodes, such as... Figure 1 As shown. In this hierarchical directory tree, a node can consist of one or more consecutive line-level text segments. The depth of a heading node in the hierarchy represents its level in the logical structure, while paragraph nodes can only exist as leaf nodes. The parent node of any node represents the next higher-level heading to which it belongs, and the child nodes of the same parent node are arranged from left to right according to the reading order. Therefore, a structured document can be browsed by traversing the hierarchical directory tree in preorder.
[0003] Document directory extraction benefits many downstream tasks, such as summarizing and information retrieval, and can improve the performance of large language models in document question answering. However, document directory extraction also faces significant challenges. These challenges stem from long-range dependencies, semantic fragmentation, and structural diversity. First, document-level text is often quite long, with the directory structure scattered throughout, resulting in a sparse distribution of long-range information, which challenges the tracking of directory associations. Second, documents are often stored in PDF files, and reading them using OCR technology or PDF parsing tools yields short and incomplete line-level text, making it difficult to model complete semantics. Third, documents come from a wide variety of sources, and different categories of documents have different logical structures, making it difficult to define a unified directory extraction method.
[0004] Existing document directory extraction methods typically rely on local information within the document, using local text pairs for state transitions while ignoring global long-range dependencies and failing to model complete text semantics. Furthermore, some methods exploit shallow visual or formatting features of the document, making it difficult for them to generalize across different document categories. Summary of the Invention
[0005] The purpose of this invention is to propose a document directory extraction method and system based on a generative language model. It defines a unified extraction paradigm for documents with different structures, can provide global dependency relationships by utilizing the results of historical decisions, models complete local textual semantic information through multiple document fragments, and achieves a universal effect across different document categories.
[0006] The technical solution adopted by the present invention to achieve the above objectives is as follows:
[0007] A document directory extraction method based on a generative language model includes the following steps:
[0008] Construct a global context stack to store the hierarchical structure of generated local document fragments in the form of a document structure directory tree;
[0009] Construct a future input window to store the current local document fragment and its subsequent local document fragments at each traversal stage;
[0010] When extracting the document directory, the document is first parsed to obtain a list of its line-level document fragments;
[0011] During the current traversal phase, the current local document fragment traversed from the list of line-level document fragments and several subsequent local document fragments are stored in the future output window;
[0012] The hierarchical information stored in the global context stack is converted into text according to the global document structure to generate a text representation of the global context stack;
[0013] The text representation of the global context stack is concatenated with the text representations of the current local document fragment and its subsequent local document fragments in the future input window, and then fed into the generative language model to predict the hierarchical structure of the generated local document fragments. The hierarchical structure of the generated local document fragments is then stored in the global context stack in the form of a document structure directory tree.
[0014] After completing all traversal stages according to the above steps, the entire document's directory can be extracted.
[0015] Furthermore, the hierarchical structure of a local document fragment includes paragraphs, n-level headings, and / or concatenation with the preceding text.
[0016] Furthermore, the step of storing the hierarchical structure of local document fragments in the form of a document structure directory tree includes: inserting local document fragments predicted as n-level headings as heading nodes in the tree, with their parent node being the nearest n-1 level heading node; inserting local document fragments predicted as paragraphs as paragraph nodes in the tree, with their parent node being the nearest arbitrary level heading node; and inserting local document fragments predicted to be concatenated with the preceding text into the same node where the previous local document fragment is located.
[0017] Furthermore, the global context stack stores all nodes on the rightmost path of the document hierarchy tree sequentially from bottom to top, including three cases:
[0018] The current local document fragment is inserted as an n-level heading in the hierarchical directory tree, and its position is the rightmost child node of the nth node in the rightmost path of the current document hierarchical directory tree.
[0019] The current local document fragment is inserted as a paragraph in the document hierarchy tree, and its position is the rightmost child node of the last heading node in the rightmost path of the current document hierarchy tree.
[0020] The current local document fragment is concatenated with the local document fragment of the previous stage, and its position is the end node on the rightmost path of the current stage.
[0021] Furthermore, global document structure representation refers to using hierarchical marking symbols to mark the text of each node in the document structure directory tree to represent the different hierarchical structures of local document fragments; then, the text of each node with added hierarchical marking symbols is concatenated together in order from the bottom of the stack to the top of the stack to generate a text representation of the global context stack.
[0022] Furthermore, using hierarchical markers for marking includes: using an asterisk to represent a paragraph in a document, using n consecutive plus signs to represent an n-level heading in a document, and using an equal sign to indicate concatenation with the preceding text.
[0023] Furthermore, the text of each node, which has been marked with hierarchical symbols, is concatenated together using line breaks.
[0024] Furthermore, the amount of subsequent text in the current local document fragment can be controlled by setting the value of the future input window.
[0025] Furthermore, the generative language model is a pre-trained model, such as, but not limited to, GPT or Llama.
[0026] A document directory extraction system based on a generative language model includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the steps of the method described above.
[0027] Compared with the prior art, the beneficial effects of the present invention are:
[0028] 1) This invention adopts a generative document directory extraction method, which uses a global document structure representation to transform document directory extraction into a hierarchical structure generation task that involves one traversal, effectively ensuring the generalization ability of the method.
[0029] 2) This invention employs a global context stack and stores the generated document hierarchy in the form of a document structure directory tree. It uses past decisions to influence current predictions, thereby leveraging the context learning capabilities of generative language models to effectively solve the global dependency problem in document directory extraction.
[0030] 3) By using a future input window that integrates subsequent text, subsequent local document fragments are combined with the current local document fragments, and shorter line-level texts are assembled together, ensuring more complete local text semantics and improving the efficiency of the document directory extraction method in the inference stage. Attached Figure Description
[0031] Figure 1 A flowchart illustrating the document directory extraction task.
[0032] Figure 2 This is a flowchart illustrating the document directory extraction method based on a generative language model.
[0033] Figure 3 This represents the input and output format of a document directory extraction method based on a generative language model. Detailed Implementation
[0034] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below.
[0035] This invention proposes a document directory extraction method based on a generative language model, such as... Figure 2 As shown, this method employs a hierarchical structure generation approach that iterates through line-level document fragments once. Given a list of line-level document fragments parsed from the actual document, it iterates through the local document fragments in the list, progressively generating the hierarchical structure (or document structure) of each local document fragment within the global directory. Specifically, the possible hierarchical structures of local document fragments generally include two types: 1) n-level headings; 2) paragraphs. In addition, since a complete heading or paragraph may be split into several lines, a third possibility in the document hierarchy needs to be considered: 3) (as mentioned above) concatenation of line-level fragments. Through the traversal generation of these three hierarchical structures, this method can completely predict the global structure of the document and achieve directory extraction.
[0036] Document directory extraction requires multiple traversal stages to predict the hierarchical structure of local document fragments. To adapt to different document categories and achieve generalization across various document types, this invention employs a pre-trained generative language model (e.g., GPT, Llama) to predict the hierarchical structure at each traversal stage. To more efficiently and concisely represent the possible hierarchical structures of local document fragments, this invention proposes a Markdown-like global document structure representation. This representation uses concise and unified notations to represent various hierarchical structures, and all document categories must adhere to this representation. Figure 3The diagram illustrates a specific embodiment of this full document structure representation. Specifically, this representation uses an asterisk "*" to represent paragraphs in the document and n consecutive plus signs "+" to represent n-level headings. Furthermore, it uses an equal sign "=" to indicate concatenation with preceding text. During the traversal of the list of line-level document fragments, the generative language model receives the currently input local document fragment, predicts the above three types of hierarchical symbols, and thus gradually completes the construction of the global structure.
[0037] To address the problem of global long-range dependencies and to inspire current predictions through past decisions, this invention specifically proposes a global context stack to store the document structure generated during traversal. Simultaneously, to model more complete local textual semantic information and alleviate the shortsightedness problem in document directory extraction, this invention further proposes a future input window to store the current local document fragment and its subsequent local document fragments. At each traversal stage, the local document fragments in the global context stack and the future input window are concatenated together in text form and used as input to the generative language model. The specific contents of the global context stack and the future input window are described below:
[0038] 1. Global Context Stack
[0039] In each traversal phase, the hierarchical structure predicted in the previous traversal phase can be organized into a document hierarchy tree. Local document fragments predicted as n-level headings are inserted into the tree as heading nodes, their parent node being the nearest (n-1)-level heading node; local document fragments predicted as paragraphs are inserted into the tree as paragraph nodes, their parent node being the nearest arbitrary-level heading node; local document fragments predicted as (concatenated with the preceding text) are inserted into the same node as the previous local document fragment, such as... Figure 2 The left side is shown. In each traversal phase, nodes in the global context stack are transformed into a part of the generative language model input in the form of text, influencing the current prediction through previous decisions, thus leveraging the context learning capability of the generative language model.
[0040] The global context stack stores the key parts of the generated document structure. These key parts are all the nodes on the rightmost path of the document hierarchy tree stored sequentially from bottom to top in the global context stack, with the root node always at the bottom of the stack. These key parts are closely related to the current local document fragment, effectively solving the problem of long-distance global dependencies in document directory extraction. These nodes are key because, in practice, documents organized as document structure tree forms are often browsed in preorder traversal order. Therefore, the local document fragment processed in the previous stage is the last node on the rightmost path of the document structure tree. Furthermore, consider three possible hierarchical structures for the current local document fragment: 1) The current text fragment is inserted into the document structure tree as an n-level heading (considering the root node as level 0). Since this text fragment will become the last node of the new rightmost path in the next stage, its position is clearly the rightmost child node of the nth node on the rightmost path of the current directory tree; 2) The current text fragment is inserted into the hierarchical directory tree as a paragraph. Since this text fragment will become the end node of the new rightmost path in the next stage, its position is obviously the rightmost child node of the last heading node in the rightmost path of the current directory tree; 3) The current text fragment is concatenated with the text fragment from the previous stage. Since the position of the local document fragment in the previous stage is the end node on the current rightmost path, the text fragment will obviously be concatenated to that node. In summary, the rightmost path in the document structure directory tree stores all levels of nodes related to the possible hierarchical structure of the current text fragment. Storing these nodes in the global context stack is used to resolve the long-distance dependency of document directory extraction on global information.
[0041] To pass the global context stack as text input to the generative language model, this invention represents the global context stack using the global document structure representation of a Markdown-like markup language described above. Specifically, for n-level heading nodes in the global context stack, n consecutive plus signs "+" are added before the heading text; for paragraph nodes in the global context stack, an asterisk "*" is added before the paragraph text. The text of each node after adding the hierarchical markers is concatenated together in order from the bottom to the top of the stack (using newline characters) to form the text representation of the global context stack.
[0042] 2. Future Input Window
[0043] In addition to representing the global context stack of the generated document structure, this invention also supplements the local document fragment input at each prediction step with a future input window, such as... Figure 2The right side of the diagram is shown. The original input window represents the number of local document fragments received by the generative language model in each traversal stage. Under normal circumstances, the value of the input window is 1 (e.g., ...). Figure 2 Local document fragment x i The future input window is a method of expanding the original input window, with the aim of including partial document fragments (such as...) from several future stages. Figure 2 Local document fragment x i+1 and x i+2 ) and the current local document fragment (such as Figure 2 Local document fragment x i This, combined with other methods, serves as a supplement to the input of the generative language model, thereby modeling more complete local textual semantic information. Based on this, the present invention can simultaneously generate hierarchical structures of multiple local document fragments in a single traversal stage, reducing the number of traversals required for document directory extraction and thus improving the operational efficiency of the invention during the inference process.
[0044] Specifically, assuming the future input window has a value of w, and w > 1, then in each traversal stage, the text of the global context stack, the current local document fragment, and at most w-1 subsequent local document fragments are concatenated together using newline characters and used as input to the generative language model. This makes the local semantics more complete and alleviates the shortsightedness problem caused by the generative language model's lack of contextual understanding. With the future output window expanded, the generative language model can simultaneously output the hierarchical structure of at most w local document fragments in one stage, inserting this series of local document fragments into the document structure directory tree simultaneously. This reduces the number of traversals required for document directory extraction, effectively improving the efficiency of the inference process.
[0045] The following is a specific application example:
[0046] Scene:
[0047] List of line-level document snippets: ["Special Bond Credit Rating Report", "Credit Quality Analysis", "Economic Strength", "Main Drivers of Economic Growth", "Driven by Consumption and Investment", "Credit Quality Analysis of This Bond Issue", "Key Terms", "The bond issuance period is 20 years, and the bond interest is paid semi-annually."]
[0048] Implementation:
[0049] Assuming the future input window value is 3, for the above list of line-level document fragments, the document directory extraction implementation of this invention requires traversing 3 stages and calling the generative language model 3 times. The stages are as follows:
[0050] 1) The global context stack is empty, and the current local document fragment is "Special Bond Credit Rating Report\nCredit Quality Analysis\nEconomic Strength". The generative language model receives the global context stack (which is empty at this stage) and the current local document fragment, and generates the hierarchical structure result as: "+\n++\n+++".
[0051] 2) The global context stack is "+Special Bond Credit Rating Report\n++Credit Quality Analysis\n+++Economic Strength", and the current local document fragment is "The main drivers of economic growth\ncome from consumption and investment.\nCredit quality analysis of this bond issue". The generative language model receives the global context stack and the current local document fragment, and generates the hierarchical structure result as: "*\n=\n++".
[0052] 3) The global context stack is “+Special Bond Credit Rating Report\n++Credit Quality Analysis of This Bond Issue”, and the current local document fragment is “Main Terms\nThe bond issuance period is 20 years, and the bond interest is paid semi-annually.” The generative language model receives the global context stack and the current local document fragment, and the generated hierarchical structure result is: “+++\n*\n=”.
[0053] Ultimately, the document's directory structure can be represented as: "+Special Bond Credit Rating Report\n++Credit Quality Analysis\n+++Economic Strength\n*The main drivers of economic growth are consumption and investment.\n++Credit Quality Analysis of This Bond Issue\n+++Main Terms\n*This bond issue has a term of 20 years, and bond interest is paid semi-annually."
[0054] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Those skilled in the art can modify or make equivalent substitutions to the technical solutions of the present invention without departing from the principles and scope of the present invention. The scope of protection of the present invention should be determined by the claims.
Claims
1. A document directory extraction method based on a generative language model, characterized in that, Includes the following steps: Construct a global context stack to store the hierarchical structure of generated local document fragments in the form of a document structure directory tree; Construct a future input window to store the current local document fragment and its subsequent local document fragments at each traversal stage; When extracting the document directory, the document is first parsed to obtain a list of its line-level document fragments; During the current traversal phase, the current local document fragment traversed from the list of line-level document fragments and several subsequent local document fragments are stored in the future input window; The hierarchical information stored in the global context stack is converted into text according to the global document structure representation to generate the text representation of the global context stack. The global document structure representation refers to the use of hierarchical marking symbols to mark the text of each node in the document structure directory tree to represent the different hierarchical structures of local document fragments. Then, the text of each node with added hierarchical marking symbols is concatenated together in order from the bottom of the stack to the top of the stack to generate the text representation of the global context stack. The text representation of the global context stack is concatenated with the text representations of the current local document fragment and its subsequent local document fragments in the future input window, and then fed into the generative language model to predict the hierarchical structure of the generated local document fragments. The hierarchical structure of the generated local document fragments is then stored in the global context stack in the form of a document structure directory tree. After completing all traversal stages according to the above steps, the entire document's directory can be extracted.
2. The method as described in claim 1, characterized in that, The hierarchical structure types of local document fragments include paragraph type, n-level heading type, and concatenation with the preceding text type.
3. The method as described in claim 2, characterized in that, The steps for storing the hierarchical structure of local document fragments in the form of a document structure directory tree include: inserting a local document fragment predicted as an n-level heading into the tree as a heading node, with its parent node being the nearest n-1 level heading node; inserting a local document fragment predicted as a paragraph into the tree as a paragraph node, with its parent node being the nearest arbitrary level heading node; and inserting a local document fragment predicted as concatenated with the preceding text into the same node where the previous local document fragment is located.
4. The method as described in claim 3, characterized in that, The global context stack stores all nodes on the rightmost path of the document hierarchy tree in a bottom-up, sequential manner, including three cases: The current local document fragment is inserted as an n-level heading in the hierarchical directory tree, and its position is the rightmost child node of the nth node in the rightmost path of the current document hierarchical directory tree. The current local document fragment is inserted as a paragraph in the document hierarchy tree, and its position is the rightmost child node of the last heading node in the rightmost path of the current document hierarchy tree. The current local document fragment is concatenated with the local document fragment of the previous stage, and its position is the end node on the rightmost path of the current stage.
5. The method as described in claim 1, characterized in that, Using hierarchical markers for labeling includes: using an asterisk to represent a paragraph in a document, using n consecutive plus signs to represent an n-level heading in a document, and using an equal sign to indicate concatenation with the preceding text.
6. The method as described in claim 1, characterized in that, The text of each node, which has been marked with hierarchical symbols, is concatenated together using newline characters.
7. The method as described in claim 1, characterized in that, The amount of subsequent text in the current local document fragment can be controlled by setting the value of the future input window.
8. The method as described in claim 1, characterized in that, Generative language models are pre-trained models, using GPT or Llama.
9. A document directory extraction system based on a generative language model, characterized in that, It includes a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the steps of the method according to any one of claims 1-8.