Document structured analysis method and system based on basic discourse units

Abstract

The invention discloses a document structured analysis method and system based on basic discourse units, and relates to the technical field of artificial intelligence. The method comprises the steps that a to-be-processed document is received through an input layer; segmenting the to-be-processed document into a plurality of basic discourse units through a segmentation layer; distributing a unique integer index for each basic discourse unit according to the basic discourse units through an index distribution layer to obtain the basic discourse units associated with the unique integer indexes; a semantic tree corresponding to a to-be-processed document is obtained based on a basic discourse unit associated with a unique integer index through a solver, each node of the semantic tree comprises a closed interval formed by the integer indexes, and the closed interval is [a starting index-an ending index]; through the analysis layer, content accurate tracing is carried out on the to-be-processed document based on the closed interval, the structured analysis result of the to-be-processed document is obtained, and the problem that in the prior art, document structured compression and content accurate tracing cannot be achieved is solved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and system for document structured analysis based on basic discourse units. Background Technology

[0002] In recent years, large language models such as GPT-4 and DeepSeek-V3 have demonstrated outstanding performance in tasks such as text summarization and content generation, promoting the industrial application of natural language processing technology. However, existing technologies still have many bottlenecks in long document structured analysis scenarios, making it difficult to meet the requirements for high-precision processing.

[0003] Current mainstream document analysis solutions either rely on large language models to directly generate summaries or treat text as a one-dimensional linear sequence and perform token pruning and compression. While the former can output structured content, the generated results lack source traceability, cannot accurately correspond to the original text, and are prone to content illusion; the latter achieves text simplification by removing low-information words, but it destroys the syntactic integrity of sentences and loses long-distance semantic dependencies. At the same time, when faced with extremely long texts, models generally suffer from the "lost in the middle" phenomenon, with a significant decrease in attention to key information in the middle of the document, resulting in information omissions in the analysis results.

[0004] Furthermore, traditional rule-based document structure extraction methods can only identify explicit format features such as titles and paragraphs, making it difficult to uncover the deep semantic and logical connections within a document.

[0005] Therefore, there is an urgent need for a document structure analysis method that can both achieve structured document compression and accurate content traceability. Summary of the Invention

[0006] In view of this, the present invention proposes a document structured analysis method and system based on basic discourse units, which can achieve both structured document compression and accurate content tracing.

[0007] To achieve the above objectives, the present invention provides the following technical solution: A document structure analysis method based on basic discourse units includes: Obtain the document to be processed and input the document to be processed into a pre-trained model, the model including an input layer, a segmentation layer, an index allocation layer, a solver, an analysis layer and an output layer; The document to be processed is received through the input layer; The document to be processed is divided into multiple basic text units by the segmentation layer; The index allocation layer assigns a unique integer index to each basic text unit according to the natural order of the basic text units in the document to be processed, thereby obtaining the basic text units with associated unique integer indexes. The solver obtains the semantic tree corresponding to the document to be processed based on the basic discourse units with associated unique integer indices. Each node of the semantic tree includes a closed interval composed of integer indices, and the closed interval is [start index - end index]. The analysis layer performs precise content tracing on the document to be processed based on the closed interval, and obtains the structured analysis results of the document to be processed. The structured analysis results are output through the output layer.

[0008] Based on the above technical solution, the present invention can be further improved as follows: Optionally, assigning a unique integer index to each basic text unit includes: The effective index range of the integer index is determined to be 0~N, where N is the total number of basic text units in the document to be processed minus one.

[0009] Optionally, after obtaining the semantic tree corresponding to the document to be processed based on the basic discourse units with associated unique integer indices through the solver, the method further includes: Real-time monitoring of the semantic tree output by the model; After the model outputs a special marker indicating the start of the recording interval, the position of the next word after the special marker is defined as the starting index slot; After the model outputs the delimiter, the position of the next word after the delimiter is defined as the terminating index slot; Numeric mask constraints are applied to the start index slot and the end index slot, allowing the model to output integer indices within the valid index range only at the start index slot and the end index slot.

[0010] Optionally, applying numeric mask constraints to the start and end index slots includes: The logarithmic probability value corresponding to non-numeric lexical units is forced to be negative infinity.

[0011] Optionally, before inputting the document to be processed into the pre-trained model step, the method further includes: Receive historical documents and divide the historical documents into multiple historical basic discourse units; According to the natural order of the historical basic text units in the historical documents, a unique integer index is assigned to each historical basic text unit to obtain the historical basic text units with associated unique integer indexes. Training samples are constructed based on historical basic discourse units with associated unique integer indices, and the training samples are processed to obtain the final training samples. The model is pre-trained based on the final training samples.

[0012] Optionally, processing the training samples to obtain the final training samples includes: The training samples are subjected to layout extraction and semantic segmentation to obtain preprocessed data; The preprocessed data is input into the solver to generate semantic tree samples; The semantic tree samples are input into the critics for iterative optimization until a high score threshold or the maximum number of iterations is reached, in order to obtain the final training samples.

[0013] A document structured analysis system based on basic discourse units includes: The acquisition module is used to acquire documents to be processed. The model includes an input layer, a segmentation layer, an index allocation layer, a solver, an analysis layer, and an output layer; The input layer is used to receive the document to be processed; The segmentation layer is used to segment the document to be processed into multiple basic text units; The index allocation layer is used to assign a unique integer index to each basic text unit according to the natural order of the basic text units in the document to be processed, thereby obtaining the basic text units with associated unique integer indexes. The solver is used to obtain the semantic tree corresponding to the document to be processed based on the basic discourse units with associated unique integer indices. Each node of the semantic tree includes a closed interval composed of integer indices, and the closed interval is [start index - end index]. The analysis layer is used to perform precise content tracing of the document to be processed based on the closed interval, and obtain the structured analysis results of the document to be processed. The output layer is used to output the structured analysis results.

[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the steps of the method described herein.

[0015] A non-transitory computer-readable storage medium having a computer program stored thereon, the computer program implementing the steps of the method when executed by a processor.

[0016] The present invention has the following advantages: This invention presents a document structured analysis method based on basic discourse units. By segmenting documents into basic discourse units and assigning unique integer indices, it outputs a semantic tree containing indexed closed intervals. These indexed closed intervals rigidly bind semantic tree nodes to the original text, allowing for precise location of corresponding content via [starting index - ending index]. This completely solves the problems of lack of traceability and difficulty in verifying errors in traditional solutions. The structured output strictly relies on the original discourse units, without fictitious information, eliminating content illusions from the root and improving the fidelity and credibility of the analysis results. By constructing a hierarchical structure with complete discourse units, it achieves efficient compression of long documents while avoiding syntactic damage and semantic loss caused by token pruning, alleviating the "lost in the middle" phenomenon, and ensuring that no key information in long documents is omitted. Attached Figure Description

[0017] For illustrative and not limiting purposes, the present invention will now be described in conjunction with embodiments and accompanying drawings, wherein: Figure 1 This is a flowchart illustrating the document structure analysis method based on basic discourse units in an embodiment of the present invention. Figure 2 This is a schematic diagram of the main components of the document structure analysis system based on basic discourse units in an embodiment of the present invention; Figure 3 This is a schematic diagram of the physical structure of the electronic device provided by the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0019] It should be noted that the terms "first," "second," etc., in the specification and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be used interchangeably where appropriate for the embodiments of the invention described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0020] It should be noted that, where there is no conflict, the embodiments and features of the present invention can be combined with each other. The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0021] Figure 1 This is a flowchart illustrating the document structure analysis method based on basic discourse units in an embodiment of the present invention, as shown below. Figure 1 As shown, the document structure analysis method based on basic discourse units provided in this embodiment of the invention includes the following steps S101 to S107.

[0022] S101, Obtain the document to be processed and input it into the pre-trained model.

[0023] The model includes an input layer, a partitioning layer, an index allocation layer, a solver, an analysis layer, and an output layer.

[0024] S102, receive the document to be processed through the input layer.

[0025] S103 divides the document to be processed into multiple basic discourse units through a segmentation layer.

[0026] Based on the syntactic boundaries of the document to be processed (such as punctuation marks like periods, question marks, and exclamation marks) or specific conjunction logic, the continuous text stream is divided into the smallest atomic semantic units, namely basic discourse units (EDUs).

[0027] For example: Suppose the original text is: "Artificial intelligence is developing rapidly. Due to its powerful computing capabilities, it has transformed many industries. However, security issues cannot be ignored." The model divides it into three EDUs: EDU 1: "Artificial intelligence is developing rapidly." EDU 2: "It has transformed many industries due to its powerful computing capabilities." EDU 3: "However, security issues cannot be ignored." Each EDU is assigned a unique integer index (Index ID) starting from 0 or 1, according to the natural order in which the EDUs appear in the original document.

[0028] S104. By using the index allocation layer to assign a unique integer index to each basic text unit according to the natural order of the basic text units in the document to be processed, the basic text units with associated unique integer indexes are obtained.

[0029] The effective index range of the integer index is determined to be 0~N, where N is the total number of basic text units in the document to be processed minus one.

[0030] The EDUs mentioned in step S103 are assigned coordinates: ID=0, ID=1, ID=2. At this point, the entire document is no longer viewed as a character stream by the model, but as an ordered sequence of indices. This sequence of integers constitutes the "coordinate system".

[0031] S105, the solver obtains the semantic tree corresponding to the document to be processed based on the basic discourse units of the associated unique integer index. Each node of the semantic tree includes a closed interval composed of integer indices, and the closed interval is [start index - end index].

[0032] The model is trained to output a hierarchical semantic tree, rather than free text. Each node in the tree contains "explicit pointers" for lossless source tracing. The semantic tree output by the model is monitored in real time. Explicit pointers are not memory addresses, but rather markers in the model's output that directly reference specific integer indices in the coordinate system described above. These indices allow the output to physically and precisely retrieve specific sentences from the original text, rather than relying on semantic similarity matching.

[0033] In this invention, pointers exist in the form of "closed intervals". The specific format is [start ID - end ID]. This interval represents all continuous content from the EDU corresponding to the "start index" to the EDU corresponding to the "end index" (including both ends).

[0034] For example, [0-2] represents covering all three sentences from EDU 0 to EDU 2 in the example above. In a standard implementation, a single semantic node corresponds to one and only one closed interval. This forces the model to group documents into consecutive logical blocks.

[0035] If a topic involves scattered paragraphs (such as sentences 0-2 and 10-12), the model will generate two independent child nodes, corresponding to the intervals [0-2] and [10-12] respectively, thus maintaining the logical rigor of the tree structure.

[0036] After the model outputs a special marker indicating the start of the recording interval, the position of the next word after the special marker is defined as the starting index slot; After the model outputs the delimiter, the position of the next word after the delimiter is defined as the terminating index slot; Numeric mask constraints are applied to the start index slot and the end index slot, allowing the model to output integer indices within the valid index range only at the start index slot and the end index slot.

[0037] The logarithmic probability value corresponding to non-numeric lexical units is forced to be negative infinity.

[0038] The log probability value corresponding to the tokens that are outside the effective index range is forced to be negative infinity.

[0039] To prevent the model from generating non-existent coordinates (i.e., illusions), specific numeric mask constraints are introduced during the inference phase. In the process of generating the output sequence in the autoregressive Large Language Model (LLM), after the model outputs a special marker indicating the "start of the recording interval" (such as "["), the next token position is the "start index slot"; after outputting a separator (such as "-"), the next token position is the "end index slot".

[0040] Just before entering these specific slots, the system intercepts the model's Logits (probability distribution) output. The system constructs a numeric mask containing only the valid index range of the current document (e.g., 0 to N). The log probability values ​​for non-numeric terms are forced to negative infinity. The log probability values ​​for terms outside the valid index range are also forced to negative infinity. This ensures that the model can only output coordinates that actually exist in the document at these slots, fundamentally eliminating the possibility of fictitious references.

[0041] S106, the analysis layer performs precise source tracing of the content of the document to be processed based on closed intervals, and obtains the structured analysis results of the document to be processed.

[0042] S107 outputs structured analysis results through the output layer.

[0043] The document structure analysis method based on basic discourse units also includes: Receive historical documents and divide the historical documents into multiple historical basic discourse units; According to the natural order of the historical basic text units in the historical documents, a unique integer index is assigned to each historical basic text unit to obtain the historical basic text units with associated unique integer indexes. Training samples are constructed based on historical basic discourse units with associated unique integer indices, and the training samples are processed to obtain the final training samples. The model is pre-trained based on the final training samples.

[0044] The process of processing the training samples to obtain the final training samples includes: The training samples are subjected to layout extraction and semantic segmentation to obtain preprocessed data; Since document structure contains both explicit visual features and implicit semantic logic, data construction is divided into two parts: The first part is feature extraction: extracting hard formatting features from the document such as Markdown heading symbols (#), indentation levels, font size variations, list symbols, etc. This mainly constructs the document's "skeleton," ensuring the generated tree structure conforms to the document's physical layout. The second part is semantic segmentation: for long paragraphs or plain text blocks, segmentation is performed based on "topic shift," "argumentative function shift (e.g., from phenomenon to cause)," or "narrative coherence." To fill the document, large blocks of text that cannot be distinguished by layout are refined into smaller semantic particles. The coarse-grained framework obtained in the first part is used as a hard constraint, and the fine-grained nodes obtained from "semantic segmentation" are filled within the framework, together forming high-quality training samples.

[0045] The preprocessed data is input into the solver to generate semantic tree samples; the solver is responsible for generating "hierarchical semantic tree samples". That is, it takes a piece of original text as input and outputs a JSON or Markdown structure containing hierarchical relationships, abstract titles, and an attempted EDU index range ([Start-End]).

[0046] The semantic tree samples are input into the critics for iterative optimization until a high score threshold or the maximum number of iterations is reached, in order to obtain the final training samples.

[0047] Critics' criteria: Coverage integrity: Check whether the intervals divided by the Solver have missed important sentences from the original text.

[0048] Abstract accuracy: Determines whether the generated title faithfully summarizes all EDU content corresponding to the range ID.

[0049] Boundary rationality: Check whether the interval boundary breaks the close logical relationship between sentences (e.g., separating "because..." from the preceding text).

[0050] If the Critic detects an error, it will generate a prompt, requiring the Solver to regenerate the structure.

[0051] Termination conditions: The loop stops when any of the following conditions are met: ① The score given by Critic exceeds the preset high score threshold (e.g., 95 points); ② The preset maximum number of iteration rounds (e.g., 3 rounds) is reached. At this time, the sample is discarded or the current highest score sample is retained, and the final training sample is obtained based on the current highest score sample.

[0052] Figure 2 This is a schematic diagram illustrating the main components of the document structuring analysis system based on basic discourse units according to an embodiment of the present invention. Figure 2As shown, the document structure analysis system 1 based on basic discourse units provided in this embodiment of the invention includes an acquisition module 10 and a model 40.

[0053] Module 10 is used to acquire documents to be processed. Model 20, the model includes an input layer, a segmentation layer, an index allocation layer, a solver, an analysis layer, and an output layer; The input layer is used to receive the document to be processed; The segmentation layer is used to segment the document to be processed into multiple basic text units; The index allocation layer is used to assign a unique integer index to each basic text unit according to the natural order of the basic text units in the document to be processed, thereby obtaining the basic text units with associated unique integer indexes. The solver is used to obtain the semantic tree corresponding to the document to be processed based on the basic discourse units with associated unique integer indices. Each node of the semantic tree includes a closed interval composed of integer indices, and the closed interval is [starting index - ending index]. The analysis layer is used to perform precise content tracing of the document to be processed based on the closed interval, and obtain the structured analysis results of the document to be processed. The output layer is used to output the structured analysis results.

[0054] Figure 3 This is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, such as... Figure 3 As shown, the electronic device 30 includes: a processor 301, a memory 302, and a bus 303; The processor 301 and the memory 302 communicate with each other via the bus 303. The processor 301 is used to call program instructions in the memory 302 to execute the methods provided in the above-described method embodiments, and to execute the methods provided in the embodiments of the present invention.

[0055] This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions, which cause a computer to execute the method provided in this embodiment of the invention.

[0056] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various storage media capable of storing program code, such as ROM, RAM, magnetic disk, or optical disk.

[0057] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A document structure analysis method based on basic discourse units, characterized in that, include: Obtain the document to be processed and input the document to be processed into a pre-trained model, the model including an input layer, a segmentation layer, an index allocation layer, a solver, an analysis layer and an output layer; The document to be processed is received through the input layer; The document to be processed is divided into multiple basic text units by the segmentation layer; The index allocation layer assigns a unique integer index to each basic text unit according to the natural order of the basic text units in the document to be processed, thereby obtaining the basic text units with associated unique integer indexes. The solver obtains the semantic tree corresponding to the document to be processed based on the basic discourse units with associated unique integer indices. Each node of the semantic tree includes a closed interval composed of integer indices, and the closed interval is [start index - end index]. The analysis layer performs precise content tracing on the document to be processed based on the closed interval, and obtains the structured analysis results of the document to be processed. The structured analysis results are output through the output layer.

2. The document structure analysis method based on basic discourse units according to claim 1, characterized in that, Assigning a unique integer index to each basic text unit includes: The effective index range of the integer index is determined to be 0~N, where N is the total number of basic text units in the document to be processed minus one.

3. The document structure analysis method based on basic discourse units according to claim 2, characterized in that, After obtaining the semantic tree corresponding to the document to be processed based on the basic discourse units with associated unique integer indices through the solver, the method further includes: Real-time monitoring of the semantic tree output by the model; After the model outputs a special marker indicating the start of the recording interval, the position of the next word after the special marker is defined as the starting index slot; After the model outputs the delimiter, the position of the next word after the delimiter is defined as the terminating index slot; Numeric mask constraints are applied to the start index slot and the end index slot, allowing the model to output integer indices within the valid index range only at the start index slot and the end index slot.

4. The document structure analysis method based on basic discourse units according to claim 3, characterized in that, The step of applying a numeric mask constraint to the starting index slot and the ending index slot includes: The logarithmic probability value corresponding to non-numeric lexical units is forced to be negative infinity.

5. The document structure analysis method based on basic discourse units according to claim 3, characterized in that, The step of applying a numeric mask constraint to the starting index slot and the ending index slot includes: The log probability value corresponding to the tokens that are outside the effective index range is forced to be negative infinity.

6. The document structure analysis method based on basic discourse units according to claim 1, characterized in that, Before inputting the document to be processed into the pre-trained model step, the method further includes: Receive historical documents and divide the historical documents into multiple historical basic discourse units; According to the natural order of the historical basic text units in the historical document, a unique integer index is assigned to each historical basic text unit to obtain the historical basic text units with associated unique integer indexes. Training samples are constructed based on historical basic discourse units with associated unique integer indices, and the training samples are processed to obtain the final training samples. The model is pre-trained based on the final training samples.

7. The document structure analysis method based on basic discourse units according to claim 6, characterized in that, The process of processing the training samples to obtain the final training samples includes: The training samples are subjected to layout extraction and semantic segmentation to obtain preprocessed data; The preprocessed data is input into the solver to generate semantic tree samples; The semantic tree samples are input into the critics for iterative optimization until a high score threshold or the maximum number of iterations is reached, in order to obtain the final training samples.

8. A system for document structured analysis based on basic discourse units, characterized in that, include: The acquisition module is used to acquire documents to be processed. The model includes an input layer, a segmentation layer, an index allocation layer, a solver, an analysis layer, and an output layer; The input layer is used to receive the document to be processed; The segmentation layer is used to segment the document to be processed into multiple basic text units; The index allocation layer is used to assign a unique integer index to each basic text unit according to the natural order of the basic text units in the document to be processed, thereby obtaining the basic text units with associated unique integer indexes. The solver is used to obtain the semantic tree corresponding to the document to be processed based on the basic discourse units with associated unique integer indices. Each node of the semantic tree includes a closed interval composed of integer indices, and the closed interval is [start index - end index]. The analysis layer is used to perform precise content tracing of the document to be processed based on the closed interval, and obtain the structured analysis results of the document to be processed. The output layer is used to output the structured analysis results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images