A long document understanding and summary generation method, system and medium

Abstract

The application relates to a long document understanding and summary generation method, system and medium, and relates to the technical field of natural language processing; wherein the method comprises the following steps: analyzing a long document according to a preset document structure template, separating text segments, and extracting structure data; positioning and compiling the text segments according to the structure data, and constructing structure position coding; extracting the semantics of the text segments to obtain a text semantic vector, and combining the structure position coding to obtain a segment-level representation vector; combining cross-paragraph global anchor points, aggregating the segment-level representation vectors according to a structure constraint sparse attention mechanism, forming an inter-segment connection relationship, and constructing an attention mask matrix; decoding the segment-level representation vectors according to a term style template constraint combined with the attention mask matrix to generate target summary text; the calculation scale is reduced, the coverage of structure points is improved, and the output controllability is improved through the term and style template constraint.

Description

Technical Field

[0001] This application relates to the field of natural language processing technology, and in particular to a method, system, and medium for long document understanding and summarization. Background Technology

[0002] Long documents (such as contracts, research reports, audit reports, technical specifications, medical records, etc.) typically have a clear hierarchical structure and object types (headings, body paragraphs, tables, figure captions, lists, etc.). Existing global self-attention models have a computational complexity of O(n²) for attention when processing long documents. When the document exceeds several thousand tokens, the computational cost increases dramatically, making it impractical for deployment on standard hardware. To reduce computational cost, existing work (such as Longformer and BigBird) has introduced sparse attention mechanisms: Longformer uses a local sliding window with a fixed geometric pattern of a small number of pre-set global tokens; BigBird adds random attention on top of this.

[0003] Existing patents disclose a text summarization method and system based on sparse attention acceleration, comprising: reading long text data, performing word segmentation and embedding encoding processing, extracting sequence feature vectors, and mapping them to a query matrix Q, a key matrix K, and a value matrix V; constructing a summarization generation network, including a sparse attention calculation module, a feedforward calculation module, a prediction head module, and a key-value caching module; inputting the query matrix Q, key matrix K, and value matrix V into the summarization generation network, first passing through a backbone network composed of multiple stacks of the sparse attention calculation module and the feedforward calculation module, and then being processed by the prediction head module, while the key-value caching module caches historical decoding states in real time to obtain an initial text feature vector; based on the initial text feature vector, performing an autoregressive decoding process through the summarization generation network to output the final summary result. The above invention can accelerate the decoding process and ensure the semantic integrity and coherence of the generated summary.

[0004] The existing technical solutions mentioned above have the following drawbacks: Attention connection patterns are typically determined based on pure geometric positions or random sampling, completely independent of the document's semantic structure. They cannot perceive chapter hierarchy trees, paragraph boundaries, and object types (headings, tables, captions, etc.) and their referencing relationships. This results in key cross-chapter information failing to propagate along the semantic path, and the relationships between headings and their subordinate paragraphs, tables and cited text being masked and truncated, leading to omissions of key structural points in the output summary. While structure-aware masking strategies driven by chapter hierarchy trees and object types are commonly used to propagate attention connections along the document's semantic structure rather than relying on fixed geometric windows or random sampling, they generally lack constraints on domain terminology consistency and output format, leading to terminology drift and non-standard formatting in professional scenarios. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this application is to provide a method, system, and medium for long document understanding and summarization, which reduces ineffective attention computation through a masking mechanism of local windows within the structural domain and cross-domain controlled connections; aggregates cross-paragraph information through global anchors; and improves domain consistency and output format stability through terminology and template constraints.

[0006] This was achieved using the following technical solutions: Firstly, this application provides a method for long document understanding and summarization, including: The document is parsed based on a preset document structure template or the document's layout features, and text fragments are separated to extract structural data. Based on the structural data, locate and compile text fragments to construct structural position codes; Semantic extraction of text fragments yields text semantic vectors, which are then combined with structural position encoding to obtain fragment-level representation vectors. Based on the structurally constrained sparse attention mechanism, combined with cross-segment global anchors, fragment-level representation vectors are aggregated to form inter-segment connection relationships and construct an attention mask matrix. Based on terminology style template constraints and attention mask matrix, the fragment-level representation vector is decoded to generate the target summary text.

[0007] By adopting the above technical solution, based on document structure parsing and structural position encoding, utilizing structural constraint sparse attention and cross-paragraph global anchor point aggregation of fragment semantics, and combining terminology style template constraint decoding to generate summaries, the structural consistency, key information coverage, and domain terminology accuracy of long document summaries are significantly improved.

[0008] Preferably, the long document is parsed according to a preset document structure template, text segments are separated, and structural data is extracted, including: The preset document structure template is parsed and converted into the corresponding organization rule identifier; Perform format conversion and invalid data filtering on long documents to extract valid structured text streams; Perform boundary detection and text decomposition on the effective structured text stream to extract text fragments; Text fragments are separated based on organizational rules to generate structured data that includes chapter hierarchy trees, paragraph boundaries, object types, and relationships between objects.

[0009] By adopting the above technical solution, based on the document structure template transformation organization rule identifier, the text fragments are separated by boundary detection and text decomposition algorithms, and structural data containing chapter hierarchy tree, paragraph boundaries, object types and relationship associations are automatically generated, realizing the refined structured parsing of long documents, which significantly improves the content consistency, logical coherence and information retrieval efficiency of downstream summary generation.

[0010] Preferably, the structural location encoding is constructed based on the structural data location and the compiled text fragment, including: By tracing the text fragments according to the chapter hierarchy tree and chapter identifiers, the root node is determined and the standard node name is extracted; The node standard names are associated with the delimiters to form a hierarchical path string, which is then mapped to a hierarchical path code. All text fragments in the root node are sorted according to the original document order and stored in variable-length encoding as sibling sequence number encoding; The layout of the text fragments is analyzed and inferred based on heuristic rules, and a preset page type identifier is assigned to generate a short page string as the page block code. The hierarchical path code, the same-level sequence number code, and the page block code are spliced ​​and merged to form the initial position code; If the initial position code is repeated, extract the sub-index inside the segment and correct the sequence number of the current initial position code to obtain the corrected position code; If the encoding length of the initial position code and / or the modified position code is greater than the preset length threshold, then the current position code is hashed and compressed to obtain the structure position code; If not, then the current initial position code is determined to be a structural position code.

[0011] By adopting the above technical solution, a hierarchical path encoding is formed by tracing nodes based on the chapter hierarchy tree. Combined with the same-level sequence number and page block encoding, the text fragments are identified globally and uniquely in the document by correcting duplicates through sub-indexes and hashing to compress the long encoding. This significantly improves the structural fidelity and semantic alignment accuracy of the generated summary.

[0012] Preferably, semantic extraction of text fragments yields text semantic vectors, which, combined with structural position encoding, result in fragment-level representation vectors, including: Remove unnecessary whitespace and escape special characters from the text fragment to obtain the standard text content; The standard text content is semantically transformed according to the preset semantic encoding model to generate a text semantic vector; Dimensional reduction mapping is performed on the structural position encoding based on the encoding type to obtain structural features; The text semantic vector and the structural features are deeply fused through a cross-modal attention mechanism to generate a fragment-level representation vector; Use text fragments as nodes and chapter hierarchy trees, relationships between objects, and / or term coverage as edges to generate an initial structure graph; The initial structure graph is iteratively propagated through a graph attention network to update and optimize the fragment-level representation vector. Summarization is performed based on the optimized fragment-level representation vector. Reinforcement learning is performed on the summarization generation strategy according to the preset reward function, and the fragment-level representation vector is updated in combination with the policy gradient update function ∇J(θ). Reward function R: R = α·R_cov + β·R_term + γ·R_len; Where α, β and γ are all hyperparameters; R_cov=ROUGE-L(ŷ,y*) is the term coverage score, ROUGE is the text evaluation metric, and y* is the target reference sequence; R_term=|T_ŷ∩T_ref| / |T_ref| is the term hit rate, T_ŷ is the set of specialized terms, and T_ref is the set of key terms; R_len=1-max(0,|ŷ|-L_max) / L_max is the length penalty term; ∇J(θ)=E[(Rb)·∇logπθ(ŷ|x)], where E is the expectation function, ∇logπθ(ŷ|x) is the scoring function, πθ(ŷ|x) is the policy function, x is the candidate word, θ is the network parameter, and b is the greedy decoding baseline value.

[0013] By adopting the above technical solution, fragment-level representations are generated based on the fusion of text semantic vectors and structural position encoding. Through graph attention networks and reinforcement learning policy gradient updates, combined with reward functions of ROUGE coverage, term hit rate and length penalty, the term accuracy, structural fidelity and generation quality of long document summaries are significantly improved.

[0014] Preferably, based on the structurally constrained sparse attention mechanism, combined with cross-segment global anchors, fragment-level representation vectors are aggregated to form inter-segment connectivity, and an attention mask matrix is ​​constructed, including: Based on the structural constraint sparse attention mechanism and the initial word position index, structural domain analysis is performed on the fragment-level representation vector; The structurally constrained sparse attention mechanism is a combination of the intra-domain local window attention mechanism and the global anchor attention mechanism; the local window length in the intra-domain local window attention mechanism is 2. N Up to 2 N+K There are 2 lexical units; the number of global anchors in the global anchor attention mechanism is 2. m Up to 2 m+k 1; N, K, m, and k are all positive integers; If any two fragment-level representation vectors are in the same structural domain, then calculate the relative distance between the fragments; If the relative distance between segments is less than or equal to the preset local window threshold, the connection relationship between segments is determined to be an allowed connection; No, then the connection relationship between the segments is determined to be a shielded connection; If any two fragment-level representation vectors are heterogeneous domains, then the two are judged according to the preset cross-domain linking rules; If both satisfy any of the cross-domain linking rules, the connection between the fragments is determined to be an allowed connection; otherwise, the connection between the fragments is determined to be a blocked connection. Based on the inter-segment connectivity and cross-segment global anchors, the segment-level representation vectors are aggregated to generate an attention mask matrix.

[0015] By adopting the above technical solution, based on the structural domain similarity and difference analysis and cross-segment global anchor points, the connection relationship between segments is determined by the relative distance threshold of the same structural domain and the cross-domain linking rules of different structural domains, and a sparse attention mask matrix is ​​constructed, which significantly improves the computational efficiency and structural consistency of long document summarization.

[0016] Preferably, based on terminology style template constraints and an attention mask matrix, the fragment-level representation vector is decoded to generate the target summary text, including: Parse the terminology style template constraints to extract the terminology constraints and style template constraints; The fragment-level representation vector is decoded based on term constraints, the initial lexical score is calculated, candidate lexical units are extracted, and matching is performed in conjunction with a pre-defined domain terminology list. If the current candidate word is successfully matched, the initial word score is fused and corrected according to the positive bias parameter, and the corrected word score s is calculated. b ; Based on the pointer copy mechanism and the terminology copy probability function p copy :p copy =σ(W m ×h t +W c ×c t +s b The candidate lexical units are copied and output to obtain the target lexical unit; σ is the Sigmoid activation function, W m W is the hidden state weight matrix. c Let c be the context weight matrix. t For context vectors; If the current candidate word match fails, a decoding bundle search is performed on the fragment-level representation vector to calculate the term coverage of the generated sequence; Candidate nominatives are sorted in descending order based on terminology coverage, and the candidate nominative with the highest terminology coverage is selected as the target nominative. Based on the control tag sequence constrained by the style template and combined with the attention mask matrix, the target words are combined to generate the initial summary text; The initial summary text is optimized using the length regularization loss function L_len: L_len=λ·max(0,|ŷ|-L_max)² to obtain the target summary text; λ is the penalty weight coefficient, |ŷ| is the actual length of the target word generation sequence, and L_max is the upper limit of the slot length.

[0017] By adopting the above technical solution, high term coverage terms are selected through pointer copying and cluster search mechanism under term constraints, combined with style template control tag sequence and attention mask for combined decoding, and length regularization loss is used to optimize the generated text, which significantly improves the term accuracy, style consistency and length controllability of the summary.

[0018] Secondly, this application also provides a long document understanding and summarization system, which adopts the following technical solution: A long document understanding and summarization system, used to implement long document understanding and summarization methods, includes: The structure parsing module is used to parse long documents based on a preset document structure template, separate text fragments, and extract structural data; The structural position encoding module is used to locate and compile text fragments based on structural data, and construct structural position codes. The structure fusion encoding module is used to extract text fragments semantically, obtain text semantic vectors, and combine them with structure position encoding to obtain fragment-level representation vectors; The structural constraint sparse attention module is used to aggregate fragment-level representation vectors, form inter-fragment connection relationships, and construct an attention mask matrix by combining cross-segment global anchors according to the structural constraint sparse attention mechanism. The abstract decoding module is used to decode the fragment-level representation vector and generate the target abstract text based on the terminology style template constraints and the attention mask matrix.

[0019] By adopting the above technical solution, structural position encoding is generated based on document structure template parsing, and fragment-level representation is obtained by fusing text semantics and structural information. An attention mask matrix is ​​constructed using structural constraint sparse attention and cross-paragraph global anchors. The summary is generated by combining terminology style template constraints and pointer copy mechanism to decode, thereby improving the structural consistency, terminology accuracy and generation efficiency of long document summaries.

[0020] Thirdly, this application also provides a storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the long document understanding and summarization method as described above.

[0021] In summary, this application includes at least the following beneficial effects: (1) This application reduces invalid attention computation through a masking mechanism of local windows within the structural domain and cross-domain controlled connections; aggregates cross-paragraph information through global anchors; and improves domain consistency and output format stability through terminology and template constraints. (2) This application generates fragment-level representations by fusing text semantic vectors and structural position encoding, combines graph attention networks and reinforcement learning policy gradient updates, and combines reward functions of ROUGE coverage, term hit rate and length penalty to improve the term accuracy, structural fidelity and generation quality of long document summaries. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the overall process of the long document understanding and summarization method in this application; Figure 2 This is a flowchart illustrating step S2 in this application; Figure 3 This is a flowchart illustrating step S4 in this application; Figure 4 This is a flowchart illustrating step S5 in this application; Figure 5 This is a schematic diagram of the long document understanding and summarization system in this application. Detailed Implementation

[0023] The present application will be further described in detail below with reference to the accompanying drawings.

[0024] To facilitate understanding of the technical solution of this invention, the following terms are defined in this application: A structural domain refers to the collection of fragments under a specific level node in a chapter hierarchy tree. Preferably, the level node is a section node (such as a second-level heading node), and all text fragments (paragraphs, tables, figure captions, lists, etc.) under this node belong to the same structural domain.

[0025] Cross-domain linking refers to attentional connections between segments that do not belong to the same structural domain but meet preset rules. Cross-domain linking rules include at least the following: links between a title segment and segments in its subordinate structural domains; links between a table or figure caption segment and the segment it references; and links between segments across paragraphs in different sub-structural domains under the same parent chapter node.

[0026] Reference Figure 1 This application discloses a method for long document understanding and summarization, comprising: S1: Parse long documents based on preset document structure templates or document layout features, separate text fragments, and extract structural data; S2: Locate and compile text fragments based on structural data, and construct structural position codes; S3: Semantically extract text fragments to obtain text semantic vectors, and combine them with structural position encoding to obtain fragment-level representation vectors; S4: Based on the structural constraint sparse attention mechanism, combined with cross-segment global anchors, aggregate segment-level representation vectors to form inter-segment connection relationships and construct an attention mask matrix; S5: Based on the terminology style template constraints and the attention mask matrix, decode the fragment-level representation vector to generate the target summary text.

[0027] In this embodiment, the input document is structurally parsed, a chapter hierarchy tree is constructed, and a structural domain node identifier is assigned to each text fragment. The object type is identified, and the ownership / reference relationship is recorded. A structural position encoding P_struct is constructed for the text fragment, which includes: hierarchical path encoding (at least two of path sequence embedding, path hash embedding, and tree depth), sibling sequence number encoding (integer sequence number embedding), and page block encoding (object type identifier and position identifier).

[0028] The semantic representation is fused with P_struct and input into the encoder to construct an attention mask matrix M: M(i,j)=0 when the relative distance between elements within the same structural domain does not exceed the window threshold (128 to 2048 terms); M(i,j)=0 when cross-domain linking rules (title → subordinate paragraph, table / figure caption → cited paragraph, etc.) are satisfied; otherwise, M(i,j)=-10000 (weights approach 0 after Softmax); and 16 to 256 global anchors are set to aggregate cross-paragraph information. In the decoding stage, terminology constraints (bias / forced copy / reordering) and style template constraints (paragraph structure / number of key points / sentence length) are introduced to output a summary text that meets terminology consistency and formatting requirements.

[0029] Text fragment: refers to a sentence, paragraph, or sub-fragment within a paragraph; Structural domain: refers to the set of fragments under a specific level node (preferably a section node) in a chapter hierarchy tree; Cross-domain link: refers to a connection that does not belong to the same structural domain but meets preset rules (including title → subordinate paragraph, table / caption → cited paragraph, etc.); Attention mask matrix M: a matrix used to limit attention connections between fragments, where i and j are text fragment indices, and M(i,j) represents the permissibility of connecting fragment i to fragment j; Token: refers to the smallest unit of text after being processed by the tokenizer. In this invention, the local window size, relative distance, etc., are all measured in tokens.

[0030] In one specific implementation, step S1 includes: The preset document structure template is parsed and converted into the corresponding organization rule identifier; Perform format conversion and invalid data filtering on long documents to extract valid structured text streams; Perform boundary detection and text decomposition on the effective structured text stream to extract text fragments; Text fragments are separated based on organizational rules to generate structured data that includes chapter hierarchy trees, paragraph boundaries, object types, and relationships between objects.

[0031] In another specific implementation, long documents can be parsed based on their formatting features, and at least one formatting feature from the document's title style, font size, bolding, numbering rules, indentation rules, or table of contents information can be extracted and converted into corresponding organizational rule identifiers.

[0032] In this embodiment, a document structure template based on the WIPO standard is first loaded and parsed into organizational rule identifiers such as chapter keywords and indentation levels. Then, the PDF document is converted into plain text, and invalid data such as headers, footers, and citation marks are filtered out to extract valid structured text flow. Next, regular expressions and boundary detection models are used to decompose text fragments based on the title. Finally, based on the organizational rule identifiers, the text fragments are separated to generate structured data containing chapter hierarchy trees, paragraph boundaries, object types, and citation relationships, thereby outputting JSON format parsing results for direct use in subsequent patent retrieval and infringement analysis, significantly improving the efficiency of intellectual property processing.

[0033] In one specific implementation, step S2 includes: By tracing the text fragments according to the chapter hierarchy tree and chapter identifiers, the root node is determined and the standard node name is extracted; The node standard names are associated with the delimiters to form a hierarchical path string, which is then mapped to a hierarchical path code. All text fragments in the root node are sorted according to the original document order and stored in variable-length encoding as sibling sequence number encoding; The layout of the text fragments is analyzed and inferred based on heuristic rules, and a preset page type identifier is assigned to generate a short page string as the page block code. The hierarchical path code, the same-level sequence number code, and the page block code are spliced ​​and merged to form the initial position code; If the initial position code is repeated, extract the sub-index inside the segment and correct the sequence number of the current initial position code to obtain the corrected position code; If the encoding length of the initial position code and / or the modified position code is greater than the preset length threshold, then the current position code is hashed and compressed to obtain the structure position code; If not, then the current initial position code is determined to be a structural position code.

[0034] In this embodiment, structural analysis and structural position encoding (see...) Figure 2Taking a technical specification document containing 10 first-level chapters, with an average of 4 second-level sections per chapter, as an example: Recognize the title style to construct a chapter hierarchy tree with a depth of 3 levels, with the root node numbered 0, first-level nodes 01 to 10, and second-level nodes 0101 to 1004; It identifies five object types: title, body text, table, figure caption, and list, and records the mapping relationship between tables / figure captions and cited paragraphs; (3) Construct P_struct for each segment: hierarchical path encoding includes path hash embedding (128 dimensions) and tree depth scalar (64 dimensions) concatenated; same-level sequence number encoding adopts sinusoidal position encoding (64 dimensions); page block encoding includes object type one-hot embedding plus page number scalar, totaling 64 dimensions. P_struct has a total dimension of 256 dimensions. After linear transformation and concatenation with text semantic representation (768 dimensions), it is fed into the encoder.

[0035] In one specific implementation, step S3 includes: Remove unnecessary whitespace and escape special characters from the text fragment to obtain the standard text content; The standard text content is semantically transformed according to the preset semantic encoding model to generate a text semantic vector; The structural position encoding is dimensionality-reduced and mapped according to the encoding type, and the structural features obtained after dimensionality reduction are deeply fused with the text semantic vector through a cross-modal attention mechanism to generate a fragment-level representation vector. Use text fragments as nodes and chapter hierarchy trees, relationships between objects, and / or term coverage as edges to generate an initial structure graph; The initial structure graph is iteratively propagated through a graph attention network to update and optimize the fragment-level representation vector. Summarization is performed based on the optimized fragment-level representation vectors, and reinforcement learning is performed on the summarization generation strategy according to the preset reward function. The generation model parameters are updated by combining the policy gradient update function ∇J(θ). Reward function R: R = α·R_cov + β·R_term + γ·R_len; Where α, β and γ are all hyperparameters; R_cov=ROUGE-L(ŷ,y*) is the term coverage score, ROUGE is the text evaluation metric, and y* is the target reference sequence; R_term=|T_ŷ∩T_ref| / |T_ref| is the term hit rate, T_ŷ is the set of specialized terms, and T_ref is the set of key terms; R_len=1-max(0,|ŷ|-L_max) / L_max is the length penalty term; ∇J(θ)=E[(Rb)·∇logπθ(ŷ|x)], where E is the expectation function, ∇logπθ(ŷ|x) is the scoring function, πθ(ŷ|x) is the policy function, x is the candidate word, θ is the network parameter, and b is the greedy decoding baseline value.

[0036] In this embodiment, the structural graph propagation is performed as follows: all text fragments are treated as nodes, and a heterogeneous graph is constructed with the hierarchical subordinate relationship (weight 1.0), the reference relationship (weight 0.8), and the fragment pairs with semantic similarity exceeding the threshold θ=0.75 as edges. A graph attention network (GAT) is used to perform 1 to 6 rounds of message propagation to update the fragment-level representation.

[0037] Reinforcement learning fine-tuning: After supervised pre-training, the REINFORCE algorithm is used to fine-tune the decoding strategy.

[0038] The reward function is defined as: R = α·R_cov + β·R_term + γ·R_len, where R_cov = ROUGE - L(ŷ, y*) is the coverage score; R_term = |T_ŷ∩T_ref| / |T_ref| is the term hit rate; and R_len = 1 - max(0, |ŷ| - L_max) / L_max is the length penalty term. Optimal hyperparameters: α=0.5, β=0.3, γ=0.2; Policy gradient update: ∇J(θ)=E[(Rb)·∇logπθ(ŷ|x)], where b is the greedy decoding baseline value.

[0039] In this embodiment, firstly, redundant whitespace is removed and LaTeX special characters are escaped from the text fragments parsed from the PDF to obtain standard text content. Then, a pre-trained PubMedBERT semantic encoding model is used to convert the content into a 768-dimensional text semantic vector. Simultaneously, PCA dimensionality reduction mapping is performed on the structural position encoding based on the chapter encoding type (e.g., "1. Introduction", "2. Methods"). A cross-modal attention mechanism is used to deeply fuse the text semantic vector with structural features, generating a fragment-level representation vector for each text segment. Using paragraphs as nodes, an initial structure graph is constructed based on parent-child relationships in the chapter hierarchy tree (e.g., 3.1 belongs to 3), cross-paragraph co-occurrence edges of the term "CRISPR," and reference edges from "Materials and Methods" to "Results." A three-layer graph attention network is then used to process the initial structure. The graph is iteratively propagated to update and optimize fragment-level representation vectors. Finally, policy gradient reinforcement learning is adopted, with a reward function set as R = 0.5·R_cov + 0.3·R_term + 0.2·R_len, where R_cov is the target ROUGE-L value (threshold ≥ 0.45), R_term is the hit rate of key gene editing terms (≥ 0.8), and R_len is the length penalty (upper limit ≤ 300 words). The parameters of the generating model are updated by ∇J(θ) = E[(Rb)·∇logπθ], so that the generated review abstract not only covers the core methods of the original text (such as CRISPR-Cas9 delivery efficiency) but also maintains hierarchical logic. The ROUGE-L is improved from 0.33 to 0.48, and the term hit rate reaches 0.87, which is better than traditional graph models.

[0040] In one specific implementation, step S4 includes: Based on the structural constraint sparse attention mechanism and the initial word position index, structural domain analysis is performed on the fragment-level representation vector; The structurally constrained sparse attention mechanism is a combination of the intra-domain local window attention mechanism and the global anchor attention mechanism; the local window length in the intra-domain local window attention mechanism is 2. N Up to 2 N+K There are 2 lexical units; the number of global anchors in the global anchor attention mechanism is 2. m Up to 2 m+k 1; N, K, m, and k are all positive integers; If any two fragment-level representation vectors are in the same structural domain, then calculate the relative distance between the fragments; If the relative distance between segments is less than or equal to the preset local window threshold, the connection relationship between segments is determined to be an allowed connection; No, then the connection relationship between the segments is determined to be a shielded connection; If any two fragment-level representation vectors are heterogeneous domains, then the two are judged according to the preset cross-domain linking rules; If both satisfy any of the cross-domain linking rules, the connection between the fragments is determined to be an allowed connection; otherwise, the connection between the fragments is determined to be a blocked connection. Based on the inter-segment connectivity and cross-segment global anchors, the segment-level representation vectors are aggregated to generate an attention mask matrix.

[0041] In this embodiment, structurally constrained sparse attention and mask matrix (see...) Figure 3 Taking a document with a maximum input of N=8192 words as an example: global self-attention generates approximately 67.1 million attention elements, while the proposed solution has approximately 4.96 million active elements (4.19 million local windows within the domain + 520,000 global anchors + 250,000 cross-domain links), saving approximately 92.6% of computation. Implementation details: Before Softmax, the masking position is assigned a value of -10000 (large negative numbers make the corresponding weights approach 0), and connections are allowed to be assigned a value of 0; global anchors are evenly distributed across each structural domain, with approximately 21 anchors allocated to each of the three levels of documents to balance the cross-layer information flow.

[0042] In one specific implementation, step S5 includes: Parse the terminology style template constraints to extract the terminology constraints and style template constraints; The fragment-level representation vector is decoded based on term constraints, the initial lexical score is calculated, candidate lexical units are extracted, and matching is performed in conjunction with a pre-defined domain terminology list. If the current candidate word is successfully matched, the initial word score is fused and corrected according to the positive bias parameter, and the corrected word score s is calculated. b ; Based on the pointer copy mechanism and the terminology copy probability function p copy :p copy =σ(W m ×h t +W c ×c t +s b The candidate lexical units are copied and output to obtain the target lexical unit; σ is the Sigmoid activation function, W m W is the hidden state weight matrix. c Let c be the context weight matrix. t For context vectors; If the current candidate word match fails, a decoding bundle search is performed on the fragment-level representation vector to calculate the term coverage of the generated sequence; Candidate nominatives are sorted in descending order based on terminology coverage, and the candidate nominative with the highest terminology coverage is selected as the target nominative. Based on the control tag sequence constrained by the style template and combined with the attention mask matrix, the target words are combined to generate the initial summary text; The initial summary text is optimized using the length regularization loss function L_len: L_len=λ·max(0,|ŷ|-L_max)² to obtain the target summary text; λ is the penalty weight coefficient, |ŷ| is the actual length of the target word generation sequence, and L_max is the upper limit of the slot length.

[0043] In this embodiment, terminology and style template constraints (see...) Figure 4 ).

[0044] Terminology constraint implementation: Pre-construct a domain terminology table T={t1,t2,…,t} n At each decoding step, a positive bias β (preferably β∈[0.5,2.0]) is applied to the scores of candidate tokens belonging to T, while a pointer copy mechanism is enabled: when the term tᵢ exists in the input document, the decoder copies the tokens with probability p. copy =σ(W m h t +W c c t +s b Copy the term to the output, where h t For the decoder's hidden state, c t The context vector is used; if the target term still does not appear in the decoding candidates, the candidate sequence is reordered according to the term coverage after the beam search, and the candidate with the highest coverage is selected first.

[0045] Style template constraint implementation: Taking financial report summaries as an example, the template format is defined as "[Background] (within 50 characters) + [Core Findings] (3 items, each not exceeding 40 characters) + [Conclusion] (within 30 characters)". The implementation method is to concatenate the template control tag sequence at the decoder input. <bg> 、 <find1> ~ <find3> 、 <conc>In addition, the term L_len=λ·max(0,|ŷ|-L_max)² is added to the length regularization loss function, where L_max is the upper limit of the slot length and λ=0.1; after decoding, each slot is truncated or padded to ensure that the output structure conforms to the template specification.

[0046] Reference Figure 5 A long document understanding and summarization system, applied to long document understanding and summarization methods, including: The structure parsing module is used to parse long documents based on a preset document structure template, separate text fragments, and extract structural data; The structural position encoding module is used to locate and compile text fragments based on structural data, and construct structural position codes. The structure fusion encoding module is used to extract text fragments semantically, obtain text semantic vectors, and combine them with structure position encoding to obtain fragment-level representation vectors; The structural constraint sparse attention module is used to aggregate fragment-level representation vectors, form inter-fragment connection relationships, and construct an attention mask matrix by combining cross-segment global anchors according to the structural constraint sparse attention mechanism. The abstract decoding module is used to decode the fragment-level representation vector and generate the target abstract text based on the terminology style template constraints and the attention mask matrix.

[0047] In this embodiment, the structure parsing module constructs a chapter hierarchy tree using at least one of the following: title style information, font size or bold information, numbering rules, indentation rules or directory information, and generates a node identifier for the structural domain to which each text fragment belongs.

[0048] Hierarchical path encoding includes at least two of the following: path sequence embedding, path hash embedding, and tree depth, and the sibling sequence encoding is the sequence encoding generated based on the integer sequence number under the same parent node.

[0049] Page block coding includes object type identifier and location identifier. The location identifier includes at least one of page number, block coordinates or column identifier.

[0050] Sparse attention employs a combination of local window attention and global anchor attention within the structural domain, where the local window size ranges from 128 to 2048 lexical units and the number of global anchors ranges from 16 to 256.

[0051] Cross-domain linking rules should include at least the following: links between heading segments and their subordinate structural domain segments; links between table or figure caption segments and their referenced paragraph segments; and cross-paragraph segment links between different sub-structural domains under the same parent chapter node.

[0052] Terminology constraints include at least one of the following: applying a bias to the decoding candidate scores based on the terminology list, forcing copying, or reordering candidates, to ensure domain terminology consistency in the summary output.

[0053] Text fragments include sentences, paragraphs, and sub-fragments within paragraphs, and paragraph boundaries and heading boundaries are used as the priority dividing points when segmenting.

[0054] The construction of the attention mask matrix M includes: allowing connection when the structural domain node identifiers of two fragments are the same and their relative distance does not exceed a preset window threshold; allowing connection when cross-domain linking rules are met; otherwise, blocking connection.

[0055] When fragment i and fragment j simultaneously satisfy multiple allowed connection conditions, the mask assignment of M(i,j) is determined in the following order: the link between the title fragment and its subordinate structural domain fragments takes priority over the link between the table or figure caption fragment and its referenced paragraph fragment, then takes priority over the cross-paragraph fragment link between different sub-structural domains under the same upper-level chapter node, and finally takes priority over the connection of local windows within the domain.

[0056] Style template constraints include constraints on paragraph structure, number of key points, or sentence length in the summary output. These constraints are implemented through template slot filling, regular expression constraints, or decoding reordering.

[0057] Construct a structural graph and perform graph propagation: Treat text fragments as nodes and hierarchical relationships, referencing relationships, or semantic similarity relationships as edges. Update the fragment-level representation through a graph neural network, where the number of graph propagation rounds is 1 to 6.

[0058] The abstract decoding employs a pointer copy mechanism, enabling the decoder to copy terms from the input document fragments to reduce term drift.

[0059] The summary generation strategy is optimized based on reinforcement learning: a reward function is constructed that includes coverage, term consistency and length constraints, and a policy optimization algorithm is used to fine-tune the decoding strategy.

[0060] The structure parsing constructs a chapter hierarchy tree using at least one of the following: title style information, font size or bold information, numbering rules, indentation rules or table of contents information, and generates a node identifier for the structural domain to which each text fragment belongs.

[0061] Hierarchical path encoding includes at least two of the following: path sequence embedding, path hash embedding, and tree depth, and the sibling sequence encoding is the sequence encoding generated based on the integer sequence number under the same parent node.

[0062] Page block coding includes object type identifier and location identifier. The location identifier includes at least one of page number, block coordinates or column identifier.

[0063] The structurally constrained sparse attention mechanism employs a combination of local window attention and global anchor attention. The local window size ranges from 128 to 2048 tokens, and the number of global anchors ranges from 16 to 256. When connections are allowed, M(i,j) = 0; when blocks, M(i,j) is set to a pre-defined large negative value, ensuring that the attention weight at the blocked position approaches zero after Softmax computation.

[0064] A storage medium storing at least one instruction, at least one program, code set, or instruction set, wherein the at least one instruction, at least one program, code set, or instruction set is loaded and executed by a processor to implement the long document understanding and summarization method as described above.

[0065] The embodiments described in this specific implementation are preferred embodiments of this application and are not intended to limit the scope of protection of this application. Therefore, all equivalent changes made in accordance with the structure, shape and principle of this application should be covered within the scope of protection of this application.< / conc> < / find3> < / find1> < / bg>

Claims

1. A method for long document comprehension and summarization, characterized in that, include: The document is parsed based on a preset document structure template or the document's layout features, and text fragments are separated to extract structural data. Based on the structural data, locate and compile the text fragment to construct a structural position encoding; Semantic extraction is performed on the text fragments to obtain text semantic vectors, and combined with the structural position encoding to obtain fragment-level representation vectors; Based on the structurally constrained sparse attention mechanism, combined with cross-segment global anchors, fragment-level representation vectors are aggregated to form inter-segment connection relationships and construct an attention mask matrix. Based on the terminology style template constraints and the attention mask matrix, the fragment-level representation vector is decoded to generate the target summary text.

2. The long document understanding and summarization method according to claim 1, characterized in that, The process of parsing long documents according to a preset document structure template, separating text fragments, and extracting structural data includes: The preset document structure template is parsed and converted into the corresponding organization rule identifier; Perform format conversion and invalid data filtering on long documents to extract valid structured text streams; Boundary detection and text decomposition are performed on the effective structured text stream to extract text fragments; The text fragments are separated according to the organization rule identifier to generate structured data containing chapter hierarchy tree, paragraph boundaries, object types, and relationships between objects.

3. The long document understanding and summarization method according to claim 1, characterized in that, Based on the structural data, the text fragment is located and compiled to construct a structural position encoding, including: By tracing the text fragments according to the chapter hierarchy tree and chapter identifiers, the root node is determined and the standard node name is extracted; The node standard names are associated with the delimiters to form a hierarchical path string, which is then mapped to a hierarchical path code. All text fragments in the root node are sorted according to the original document order and stored in variable-length encoding as sibling sequence number encoding; The layout of the text fragments is analyzed and inferred based on heuristic rules, and a preset layout type identifier is assigned to generate a short layout string as the layout block code. The hierarchical path code, the sibling sequence number code, and the page block code are spliced ​​and merged to form the initial position code; If the initial position code is repeated, the sub-index inside the segment is extracted to correct the sequence number of the current initial position code, thus obtaining the corrected position code; If the encoding length of the initial position code and / or the modified position code is greater than a preset length threshold, then the current position code is hash-compressed to obtain the structure position code; If not, then the current initial position code is determined to be a structural position code.

4. The long document understanding and summarization method according to claim 1, characterized in that, The semantic extraction of the text fragment yields a text semantic vector, which, combined with the structural position encoding, results in a fragment-level representation vector, including: Remove unnecessary whitespace and escape special characters from the text fragment to obtain the standard text content; The standard text content is semantically transformed according to a preset semantic encoding model to generate a text semantic vector; Dimensional reduction mapping is performed on the structural position encoding based on the encoding type to obtain structural features; The text semantic vector and the structural features are deeply fused through a cross-modal attention mechanism to generate a fragment-level representation vector.

5. The long document understanding and summarization method according to claim 1, characterized in that, The structure-constrained sparse attention mechanism, combined with cross-segment global anchors, aggregates segment-level representation vectors to form inter-segment connectivity and constructs an attention mask matrix, including: Based on the structural constraint sparse attention mechanism and the initial word position index, structural domain analysis is performed on the fragment-level representation vector; If any two fragment-level representation vectors are in the same structural domain, then calculate the relative distance between the fragments; If the relative distance between the segments is less than or equal to a preset local window threshold, the connection relationship between the segments is determined to be an allowed connection; No, then the connection relationship between the segments is determined to be a shielded connection; If any two fragment-level representation vectors are heterogeneous domains, then the two are judged according to the preset cross-domain linking rules; If both satisfy any of the cross-domain linking rules, the connection between the fragments is determined to be an allowed connection; otherwise, the connection between the fragments is determined to be a blocked connection. Based on the inter-segment connectivity and cross-segment global anchors, the segment-level representation vectors are aggregated to generate an attention mask matrix.

6. The long document understanding and summarization method according to claim 1, characterized in that, The process of decoding fragment-level representation vectors and generating target summary text based on terminology style template constraints and the attention mask matrix includes: Parse the terminology style template constraints to extract the terminology constraints and style template constraints; The fragment-level representation vector is decoded based on term constraints, the initial lexical score is calculated, candidate lexical units are extracted, and matching is performed in conjunction with a pre-defined domain terminology list. If the current candidate word is successfully matched, the initial word score is fused and corrected according to the positive bias parameter, and the corrected word score s is calculated. b ; Based on the pointer copy mechanism and the terminology copy probability function p copy :p copy =σ(W m ×h t +W c ×c t +s b The candidate lexical units are copied and output to obtain the target lexical unit; σ is the Sigmoid activation function, W m W is the hidden state weight matrix. c Let c be the context weight matrix. t For context vectors; If the current candidate word match fails, a decoding bundle search is performed on the fragment-level representation vector to calculate the term coverage of the generated sequence; Candidate lexical units are sorted in descending order according to the terminology coverage rate, and the candidate lexical unit with the highest terminology coverage rate is selected as the target lexical unit. Based on the control tag sequence constrained by the style template and combined with the attention mask matrix, the target words are combined to generate the initial summary text; The initial summary text is optimized using the length regularization loss function L_len: L_len=λ·max(0,|ŷ|-L_max)² to obtain the target summary text; λ is the penalty weight coefficient, |ŷ| is the actual length of the target word generation sequence, and L_max is the upper limit of the slot length.

7. The long document understanding and summarization method according to claim 4, characterized in that, The method also includes optimizations based on structure graph propagation and reinforcement learning, specifically including the following steps: Use text fragments as nodes and chapter hierarchy trees, relationships between objects, and / or term coverage as edges to generate an initial structure graph; The initial structure graph is iteratively propagated through a graph attention network to update and optimize the fragment-level representation vector. Summarization is performed based on the optimized fragment-level representation vector. Reinforcement learning is performed on the summarization generation strategy according to the preset reward function, and the fragment-level representation vector is updated in combination with the policy gradient update function ∇J(θ). The reward function R is: R = α·R_cov + β·R_term + γ·R_len; Where α, β and γ are all hyperparameters; R_cov=ROUGE-L(ŷ,y*) is the term coverage score, ROUGE is the text evaluation metric, and y* is the target reference sequence; R_term=|T_ŷ∩T_ref| / |T_ref| is the term hit rate, T_ŷ is the set of specialized terms, and T_ref is the set of key terms; R_len=1-max(0,|ŷ|-L_max) / L_max is the length penalty term; ∇J(θ)=E[(Rb)·∇logπθ(ŷ|x)], where E is the expectation function, ∇logπθ(ŷ|x) is the scoring function, πθ(ŷ|x) is the policy function, x is the candidate word, θ is the network parameter, and b is the greedy decoding baseline value.

8. The long document understanding and summarization method according to claim 5, characterized in that, The structural constraint sparse attention mechanism is a combination of the domain-local window attention mechanism and the global anchor point attention mechanism; the local window length in the domain-local window attention mechanism is 2. N Up to 2 N+K The number of global anchors in the global anchor attention mechanism is 2. m Up to 2 m+k There are 1, N, K, m, and k, all of which are positive integers.

9. A long document comprehension and summarization system, used to implement the method as described in any one of claims 1-8, characterized in that, include: The structure parsing module is used to parse long documents based on a preset document structure template, separate text fragments, and extract structural data; The structural position encoding module is used to locate and compile the text fragment based on the structural data, and construct the structural position encoding; The structure fusion encoding module is used to semantically extract the text fragments to obtain text semantic vectors, and combine them with the structure position encoding to obtain fragment-level representation vectors; The structural constraint sparse attention module is used to aggregate fragment-level representation vectors, form inter-fragment connection relationships, and construct an attention mask matrix by combining cross-segment global anchors according to the structural constraint sparse attention mechanism. The summary decoding module is used to decode the fragment-level representation vector and generate the target summary text based on the terminology style template constraints and the attention mask matrix.

10. A storage medium storing at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by a processor to implement the long document understanding and summarization method as described in any one of claims 1 to 8.

Images