Big model-based long text official document key information extraction agent method

Abstract

The application discloses a long-text official document key information extraction agent method based on a large model, relates to the technical field of artificial intelligence, and comprises the following steps: collecting original official document long-text data; performing structural analysis and hierarchical coding on the original official document text data; performing dynamic semantic segment division based on a topic model guide; constructing a long-text official document key information extraction model based on a bidirectional semantic encoder; performing model training and trainable parameter updating; performing long-text official document key information extraction; and constructing a long-text official document key information extraction agent based on the large model. The application adopts a multi-dimensional structural coding method which fuses official document hierarchies, formats and positions, converts domain prior knowledge into computable vectors, adopts a dynamic planning text segmentation algorithm based on topic consistency and semantic density scoring, guarantees the integrity of long-text semantic segments, and introduces a structure-guided cross-segment attention mechanism in a Transformer encoder, so that precise modeling of long-distance semantic dependence is realized through structural similarity constraints.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for extracting key information from long-text official documents based on a large model. Background Technology

[0002] In the process of government digitalization and enterprise informatization, institutions at all levels generate a massive amount of long official documents every day. These documents are generally characterized by their length, rigorous structure, standardized format, and high information density.

[0003] Traditional information extraction methods mainly rely on rule templates or general sequence labeling models. Rule templates require a large number of manually written rules and have poor generalization ability, while general sequence labeling models struggle to effectively handle structural information and long-range semantic dependencies in long texts. In recent years, although large language models have demonstrated powerful text understanding capabilities, they still have the following shortcomings in processing long texts with strong professionalism and fixed formats, such as official documents: they face computational efficiency and technical limitations when directly processing extremely long texts; they lack sensitivity to the unique hierarchical structure of official documents (such as chapters, sections, articles, clauses, and items); they are prone to producing "illusionary" outputs that do not match the actual content of the document; and they cannot provide stable and reliable structured outputs to meet the needs of downstream automated processes. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to overcome the shortcomings of the prior art and provide an intelligent agent method for extracting key information from long text documents based on a large model.

[0005] The technical solution adopted to solve the above technical problems is: a smart agent method for extracting key information from long-text official documents based on a large model, which includes the following steps:

[0006] S1. Collect raw long text data of official documents, preprocess and label the raw long text data of official documents, divide the labeled dataset into training set, validation set and test set to form benchmark data for training and evaluating the long text document key information extraction model.

[0007] S2 performs structured parsing on the original official document text data, extracts its inherent hierarchical structure and format features, calculates the corresponding multi-dimensional structure code for each character, and generates the structure code sequence of the entire document.

[0008] S3, based on topic model-guided dynamic semantic segmentation, divides long texts into multiple semantically complete segments to ensure that each segment carries relatively independent semantic information;

[0009] S4. Construct a long text document key information extraction model based on bidirectional semantic encoder: This model takes semantic fragments and their corresponding substructure encoding sequences as input, and achieves accurate end-to-end extraction of key information of various documents through structure-enhanced semantic encoding, cross-fragment bidirectional semantic propagation and multi-granularity pointer network prediction.

[0010] S5, Model Training and Trainable Parameter Update;

[0011] S6, Extraction of Key Information from Long Text Documents: Extracts key information from related documents throughout the entire process to obtain accurate structured data;

[0012] S7. Construct an intelligent agent for extracting key information from long-text official documents based on a large model.

[0013] Furthermore, the types of long text data in S1 include notices, announcements, reports, opinions, letters, and minutes.

[0014] Furthermore, the data annotation types in S1 include issuing authority, addressee, subject of clause, specific matter, time limit, and date of issuance.

[0015] Furthermore, the method for structured parsing of the original official document text data in S2 includes the following steps:

[0016] S201, based on the parsing rules of regular expressions and finite state automata, scans the original official document text, identifies all information tags that represent the document structure and hierarchy, constructs a finite state automaton according to the official document structure syntax, and uses the tags identified by the regular expression as input to trigger state transitions, which are used to determine the type and nesting relationship of the tags.

[0017] S202, Calculate character-level multidimensional structure encoding: For each character in the original long text data of the official document, according to all the identified information tags representing the document structure and hierarchy, the document part type, nesting level depth and visual format features are fused, and a multidimensional structure encoding vector is assigned to it through the hierarchical encoding function.

[0018] S203, Generate a structured encoding sequence: Traverse all characters in the original document text, apply the structured encoding function to each character to obtain the structured encoding vector corresponding to each character, and arrange the structured encodings of all characters in the original text order to form a complete structured encoding sequence.

[0019] Furthermore, the method for dynamic semantic segmentation based on topic model guidance in S3 includes the following steps:

[0020] S301, preprocess the long text of the official document and calculate the salience weight of the topic;

[0021] S302, calculates the semantic segment scoring function, which comprehensively evaluates the quality of any candidate semantic segment composed of a continuous word sequence from three dimensions: topic consistency, semantic tightness and length fit.

[0022] S303, Optimal Segment Partitioning Based on Dynamic Programming: Using period, semicolon, question mark, exclamation mark, newline punctuation mark, and structural marker information as candidate points, the dynamic programming algorithm is used to find the globally optimal segment partitioning scheme that maximizes the total score of all segments.

[0023] S304, Generate fragment-level output: Align and cut the original word sequence, part-of-speech sequence, and structural tag information according to the optimal partition boundary to generate structured data corresponding to each semantic fragment.

[0024] Furthermore, the method for preprocessing long official document texts in S301 is as follows: removing garbled characters, unifying full-width and half-width characters, and segmenting the text into sentences;

[0025] The method for calculating topic saliency weights in S301 is as follows: using the jieba word segmentation tool to segment words to obtain word sequences and their part-of-speech sequences, using a latent Dirichlet distribution topic model pre-trained on a large-scale official document corpus to infer the topic distribution of the current document, and calculating the topic saliency weight of each word to measure the importance of the word and its contribution to the core topic of the document.

[0026] Furthermore, the method for constructing a long-text document key information extraction model based on a bidirectional semantic encoder in S4 includes the following steps:

[0027] S401, structural enhancement of fragment-level semantic coding;

[0028] S402, Cross-segment bidirectional semantic propagation coding: Inputting the initial enhanced representation sequence of each semantic segment into... The bidirectional Transformer encoder of the layer introduces a structure-guided cross-segment attention window into the encoder's self-attention mechanism, and uses the similarity of the structural encoding vectors at corresponding positions to apply soft constraints to the cross-segment attention.

[0029] S403, Multi-granularity pointer network information extraction: A multi-granularity pointer network is used to jointly predict the start and end positions of each predefined key information type based on the deep representation of the encoder output, while simultaneously considering the granularity level to which the information belongs. In the prediction process, structural compliance features based on official document norms are incorporated to constrain the rationality of the output.

[0030] Furthermore, the method for structural enhancement of fragment-level semantic encoding in S401 is as follows:

[0031] S40101 maps discrete word symbols and part-of-speech tags into dense vector representations through a trainable embedding lookup table;

[0032] S40102 transforms the structure encoding vector to the same dimension as the word vector using a trainable linear projection matrix. It then sums the word vector, part-of-speech vector, and the projected structure vector, and incorporates a standard sine / cosine absolute position encoding vector.

[0033] S40103 generates an initial enhanced semantic representation for each word after layer normalization.

[0034] Furthermore, the method for model training and trainable parameter update in S5 includes the following steps:

[0035] S501, calculate the loss function to guide model training: adopt a loss function that combines boundary prediction loss and structural consistency regularization. Optimize boundary prediction by calculating the negative log-likelihood loss of each training sample. At the same time, introduce a structural consistency regularization term to ensure that the structural encoding of the predicted start and end positions is similar to the structural encoding of the true positions. This enables the model to accurately predict information boundaries while maintaining consistency with prior knowledge of the document structure, thereby improving the rationality and robustness of the extraction results.

[0036] S502, Iterative Model Training: The constructed key information extraction model is iteratively trained to update all its trainable parameters. The training process is carried out on the prepared training set, and an optimization algorithm is used to minimize the overall loss function. Once the preset maximum number of training cycles is reached, the model is determined to have learned sufficiently and reached a convergent state. Iterative training is then stopped. After training is completed, the model parameter set with the best validation set performance is saved as the final model that can be used for actual document key information extraction tasks.

[0037] The beneficial effects of the present invention are as follows: (1) The present invention adopts a multi-dimensional structure coding method that integrates document hierarchy, format and location, and transforms domain prior knowledge into computable vectors, breaking through the limitation of the traditional sequence labeling model that regards text as a flat sequence.

[0038] (2) The present invention adopts a dynamic programming text segmentation algorithm based on topic consistency and semantic tightness scoring, which replaces the fixed-length sliding window cutting and ensures the integrity of long text semantic segments.

[0039] (3) This invention introduces a structure-guided cross-segment attention mechanism in the Transformer encoder, and achieves accurate modeling of long-distance semantic dependencies through structural similarity constraints.

[0040] (4) The present invention constructs an intelligent agent architecture of “dedicated extraction model + general large model”, anchors the large model generation process with structured extraction results, and realizes the synergy of accurate extraction and deep understanding reasoning. Attached Figure Description

[0041] Figure 1 This is a complete flowchart of an embodiment of the intelligent agent method for extracting key information from long-text official documents based on a large model according to the present invention.

[0042] Figure 2 This is a flowchart illustrating the method of structured parsing and hierarchical coding of raw official document text data.

[0043] Figure 3 This is a flowchart of a topic model-guided dynamic semantic segmentation method.

[0044] Figure 4 This is a flowchart illustrating the method for constructing a key information extraction model for long-text official documents based on a bidirectional semantic encoder.

[0045] Figure 5 This is a flowchart illustrating the process of constructing an intelligent agent for extracting key information from long-text official documents based on a large model. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0047] like Figure 1 As shown in this embodiment, the intelligent agent method for extracting key information from long-text official documents based on a large model includes the following steps:

[0048] S1. Collect raw long text data of official documents, preprocess and label the raw long text data of official documents, and divide the labeled dataset into training set, validation set and test set to form benchmark data for training and evaluating subsequent key information extraction models.

[0049] To construct the training dataset required for this system, a large amount of raw long-text official documents were collected from multiple channels. Data sources primarily included portals of governments at all levels, publicly available policy document databases of various functional departments, publicly available document archives, and some compliant internal document databases of enterprises. This ensured the diversity and representativeness of the data in terms of document type, issuing authority level, text format, and content theme. The collected long-text official documents included various statutory document types such as notices, announcements, reports, opinions, letters, and minutes, with text lengths ranging from several thousand to tens of thousands of words, to meet the needs of processing "long texts."

[0050] After obtaining the original text, preliminary cleaning and preprocessing are required to remove irrelevant web page tags and formatting marks, as well as perform error correction and layout restoration after optical character recognition of the scanned document, to ensure the accuracy and integrity of the text content.

[0051] The core step in constructing the training dataset is to perform refined data annotation on these official documents. The annotation strategy combines sequence annotation and fragment-level annotation, performed by professionals with a background in official document knowledge using an annotation platform. The types of data annotation include: issuing authority, addressee, subject of clauses, specific matters, time limits, and date of issuance. For entities such as "issuing authority," "addressee," and "date of issuance," which typically have fixed formats and positions, a character-level BIOES (Begin, Inside, Outside, End, Single) sequence annotation scheme is used. For complex semantic units such as "subject of clauses," "specific matters," and "time limits," which may span multiple words, phrases, or even clauses, fragment-level boundary annotation is used. This involves marking the start and end character positions of the information in the text, while simultaneously recording its semantic granularity (word, phrase, or clause).

[0052] S2 performs structured parsing on the original official document text data, extracting its inherent hierarchical structure and format features. It calculates the corresponding multi-dimensional structural code for each character, generating a structural coding sequence for the entire document. These features are then encoded into a structural coding sequence strictly aligned with the text sequence, providing prior structural knowledge for the neural network model. Figure 2 As shown.

[0053] S201, based on the parsing rules of regular expressions and finite state automata, scans the original official document text, identifies all information tags that represent the document structure and hierarchy, and constructs a finite state automaton according to the document structure syntax.

[0054] Information markings include explicit numbering, such as Chapter 1, Section 1.2, Article 3, and clauses and items, as well as document parts with specific formats, such as titles, issuing authorities, addressees, attachments, and dates of issuance.

[0055] In this embodiment, regular expressions are written for different levels of numbering patterns, for example:

[0056] Chapter entry number: r'Chapter [012345678900000\d]+, r'(\d+)\.(\d+)Section', r'Article [012345678900000\d]+;

[0057] Date: r'\d{4}year\d{1,2}month\d{1,2}day';

[0058] Specific location: r'Issuing authority: (.+)', r'Main recipient authority: (.+)'.

[0059] A finite state automaton is constructed based on the grammar of official document structure (e.g., "chapter" must have "section", "article" can have "clause"). The state represents the current parsing level (e.g., "in chapter title", "in article text"). The markers identified by the regular expression are used as input to trigger state transitions, thereby determining the type and nesting relationship of the markers. For example, when the state is "waiting for chapter title", if "Chapter 1" is identified, the state is transitioned to "in chapter title", and the current depth is recorded as 1.

[0060] S202, calculates character-level multidimensional structure encoding;

[0061] For each character in the original official document text, based on the identified structural marker information, a multi-dimensional structural encoding vector is assigned to it through a hierarchical encoding function, integrating document part type, nesting depth, and visual format features, represented as:

[0062] ,

[0063] In the above formula, This represents a structure encoding function, whose input is a center character and its local context window, and whose output is the structure encoding vector of that center character. It is defined as follows: ; Indicates the first The structural encoding vector corresponding to each character has a dimension of . This indicates the document structure context in which the character resides; This indicates the first [item] in the original official document text. The character is the first character in the original text. A basic character unit, in Chinese, is usually a Chinese character, number, English letter, or punctuation mark; Indicates a character index that is distinct from 'i'; Indicates a character index; Represented by characters Expand to the center and to the left and right A local context window for each character, used to help determine the boundaries and precise type of structural tags; This represents the radius of the context window; an example value is 10.

[0064] The weight coefficients representing document part type features are preset hyperparameters used to adjust the importance of these features in the final encoded vector. This represents a document part type embedding vector with dimension . It is a representation that maps discrete "document part types" to a continuous, dense, low-dimensional vector space. For example, it maps issuing authority, addressee, main text, chapter title, section title, article, clause, item, attachment, and date of issuance to a low-dimensional vector space. The dimension of the document part type embedding vector is a preset hyperparameter that determines the dimension of the continuous vector space to which discrete types are mapped. The weight coefficients representing the hierarchical structure features are preset hyperparameters used to adjust the importance of hierarchical features in the final encoded vector; This indicates the maximum level depth, used to limit the depth of document levels that the model can handle. An example value is 5, corresponding to the five common levels of structure: chapter, section, article, clause, and item. This represents a hierarchical index, traversing from 1 to... All preset levels; The hierarchy attenuation factor is a scalar hyperparameter between 0 and 1, used to control the degree of attenuation of the influence of different levels on the coding, so that higher-level structures have a greater influence on the coding. An example value is 0.8. Character In the The nesting depth in a hierarchical structure, if the character is located in a certain level Under the first-level heading, This refers to the nesting level of the heading, specifically counting from 1. If the character is not in any nesting level... Within the scope of level headings ; Indicates the first The unit indicator vector of the level is A one-hot encoded vector in dimension 1 only has a certain degree of one-hot encoding in dimension 2. The value for the top dimension is 1, and the values ​​for the other dimensions are 0; The weight coefficients representing format features are preset hyperparameters used to adjust the importance of format features in the final encoded vector; The format feature embedding vector has a dimension of . Used to encode characters Visual formatting information, such as font, font size, bold, italic, color, and indentation, is obtained through a predefined format dictionary mapping.

[0065] In this embodiment, the document part type embedding vector is obtained through a trainable embedding layer. First, all document part types are mapped to discrete integer indices, and then a vector of shape is initialized. A trainable embedding matrix is ​​provided; by querying this matrix, any type of index can be converted to its corresponding value. A dense vector, which is the document part type embedding vector. During model training, this embedding matrix is ​​a trainable parameter matrix.

[0066] Format Feature Embedding Vector It is obtained by combining a predefined format dictionary mapping and an embedding layer. For example, assuming that the format features include font, font size, and whether it is bold, the dictionary is first built, that is:

[0067] ,

[0068] ,

[0069] ,

[0070] For a character, if its format is Then convert it into an index tuple. As format attributes, each format attribute is then individually embedded using an embedding layer to obtain an embedding vector. These embedding vectors are then concatenated and finally projected onto a fixed linear layer. Dimension, that is, to obtain the format feature embedding vector. .

[0071] S203, Generate the structure encoding sequence.

[0072] Iterate through all characters in the original document text, apply the structure encoding function to each character to obtain the structure encoding vector corresponding to each character, and arrange the structure encodings of all characters in the original text order to form a complete structure encoding sequence, represented as follows:

[0073]

[0074] In the above formula, Represents a structure encoding sequence with dimension . , It represents the total length of the official document, that is, the total number of characters in the document, and represents the two-dimensional structural information of the entire official document. It maintains the original order in the horizontal dimension of the text sequence; and encodes the structured prior knowledge such as the document part to which each character belongs, the nesting level depth, and the visual format in the vertical dimension of the vector. This represents the structure encoding vector corresponding to the first character; This represents the structure encoding vector corresponding to the second character; Indicates the first The structure encoding vector corresponding to each character.

[0075] Official documents have strict format specifications and hierarchical structures, usually including chapters, sections, articles, clauses, items, as well as specific titles and numbers. Conventional entity recognition methods usually treat text as a flat sequence, which easily ignores structural information, making it difficult for the model to use the unique organizational logic of official documents to constrain the context in which entities appear.

[0076] S3, based on topic model-guided dynamic semantic segmentation, comprehensively considers the text's topic consistency, semantic density, and length constraints to divide long texts into multiple semantically complete segments, ensuring that each segment carries relatively independent semantic information, such as... Figure 3 As shown.

[0077] Official documents are often very long, and directly inputting them into a neural network can lead to high computational complexity and long-distance dependency problems. Conventional processing methods often simply cut the text into segments by sliding a window of fixed length, which can easily destroy the semantic integrity of sentences, paragraphs, or clauses and sever the complete semantic units in which entities are located.

[0078] S301, preprocess the long text of the official document and calculate the salience weight of the topic.

[0079] The jieba word segmentation tool is used to segment words, obtaining word sequences and their part-of-speech sequences. A latent Dirichlet distribution topic model pre-trained on a large-scale official document corpus is used to infer the topic distribution of the current document. The topic saliency weight of each word is calculated, represented as:

[0080] ,

[0081] In the above formula, Indicates the first The salience weight of a word within the current document topic distribution is used to measure the importance of the word and its contribution to the core topic of the document. The larger the value, the greater the contribution of the word to the core topic of the document. Indicates a topic index. This indicates that the document belongs to the first... The probability of each topic comes from the document-topic distribution inferred by the latent Dirichlet distribution model, which is used to reflect the topic tendency of the document. For example, an official document may have a probability of 0.7 on the topic of "personnel appointment and removal" and a probability of 0.3 on the topic of "budget". Indicates the first The first topic The probability of a word appearing comes from the topic-word distribution obtained by training the latent Dirichlet distribution model. It is used to reflect the word distribution of a topic and to represent which words are frequently used in each topic. For example, under the topic of "personnel appointment and removal", words such as "appointment", "removal", "comrade", and "position" will have a higher probability.

[0082] When using the jieba word segmentation tool for word segmentation, in the field of official documents, a custom dictionary can be loaded to improve the segmentation accuracy of professional terms and fixed phrases. For example, commonly used words in official documents such as "addressee", "date of issuance", "your unit", and "not approved" can be added to the user dictionary to avoid incorrect segmentation.

[0083] It should be noted that in the Latent Dirichlet Distribution Topic Model, a "topic" refers to a latent semantic category represented by a set of highly probable words. For example, in official document corpora, topics such as "personnel appointments and removals," "budget," "project construction," and "notifications" may automatically emerge. Each topic corresponds to a probability distribution across all words.

[0084] S302, Define the semantic segment scoring function

[0085] The quality of any candidate semantic segment composed of a continuous word sequence is calculated using a scoring function. This function comprehensively evaluates the segment based on three dimensions: topic consistency, semantic tightness, and length fit, and is expressed as follows:

[0086] ,

[0087] In the above formula, Indicates candidate text fragments The overall score indicates that the segment segmentation is of better quality, and is used to guide the optimal segmentation. Indicates the first word formed by a continuous sequence of words. The candidate text fragments are those generated by the dynamic programming algorithm during the search for the optimal partition. For example, from the first... The word to the 1 Candidate text fragments consisting of 10 words ; Represents a semantic fragment index. The first word in the sequence of words Each word is segmented into a sequence of words using word segmentation tools such as jieba. , That is, any word in the sequence; The first word in the sequence of words The word, and Word pairs are used to calculate semantic associations. It is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Indicates the first The salience weight of each word in the current document topic distribution;

[0088] This represents the set of high-frequency keywords under the dominant topic of the current document. It is used to filter words that significantly contribute to the topic and consists of several words with the highest probability under the dominant topic inferred by the topic model. The set size is 20 in the example. It comes from the Latent Dirichlet Distribution Topic Model. Specifically, first, one or several topics with the highest probability are found as the dominant topic based on the document-topic distribution. Then, the top 20 words with the highest probability are selected from the topic-word distribution corresponding to these dominant topics to form the set. For example, if the main theme of a certain official document is "project construction", then... It may contain 20 words such as {"project", "construction", "tendering", "construction", "acceptance", "funds", etc.};

[0089] This represents the adjustment coefficient for the semantic tightness item; it's a hyperparameter used to control the weight of this item in the overall score. An example value is 0.5. Indicator Harmony The point mutual information between words is used to measure the strength of the semantic association between two words; the larger the value, the stronger the semantic association between the two words. This indicates traversing candidate text fragments. All ordered or unordered word pairs in the text. This represents the coefficient of the length penalty term, a hyperparameter with an example value of 0.01. It is used to adjust the penalty intensity when the segment length deviates from the target length. Indicates candidate text fragments The character length, expressed in characters. This represents the target segment length, a preset hyperparameter used to control the expected value of the segment size; an example value is 512.

[0090] S303, Optimal Segment Partitioning Based on Dynamic Programming: Using punctuation marks such as periods, semicolons, question marks, exclamation marks, and newlines, as well as structural marker information, as candidate points, the dynamic programming algorithm is used to find the globally optimal segment partitioning scheme that maximizes the total score of all segments.

[0091] Define the state representation in dynamic programming from the previous The maximum sum of segment scores that can be obtained from a text sequence of n words is expressed by the state transition equation as follows:

[0092] ,

[0093] In the above formula, Indicates the previous The maximum total score that can be obtained by dividing a word into several segments represents the sum of the scores obtained by dividing the previous segment into several segments. The globally optimal score that can be obtained by dividing a word into several segments. Indicates the previous The maximum total score that can be obtained by dividing a word into several segments. Indicates the first The set of all possible starting positions of a segment at each word. This means taking the maximum value of the expression corresponding to all possible starting positions of the segment. Indicates candidate text fragments Overall score Indicates from the first The word to the 1 A candidate text fragment consisting of 10 words is a text block composed of word sequences. For example, if the word sequence is ["about", "issue", "...", "method", "of", "notification"], then... It refers to "the measures for issuing..."

[0094] The standard solution process for dynamic programming algorithms involves recursively calculating the final state value and backtracking to find the optimal partition boundary, outputting the final semantic segment sequence. "Recursion" refers to starting from the smallest subproblem (such as...) Starting with the state transition equation, larger subproblems are calculated step by step. ), until the optimal value of the original problem is obtained. "Backtracking" refers to recording the decision points for each optimal subproblem during the recursive process, and then... Then, by following these decision points in reverse from the endpoint, the start and end positions of all segments can be found, thus obtaining the globally optimal partitioning scheme.

[0095] S304, generates fragment-level output

[0096] Based on the optimal partitioning boundary, the original word sequence, part-of-speech sequence, and structural tag information are aligned and segmented to generate structured data corresponding to each semantic segment.

[0097] For example, for the first Candidate text fragments It records the word sequence, part-of-speech sequence, and corresponding substructure encoding sequence it contains; for example, the first... Candidate text fragments The substructure encoding sequence corresponding to the i-th character to the j-th character in the original text is the substructure encoding sequence. Zhong Cong arrive subsequence.

[0098] In this embodiment, the dynamic programming algorithm is divided into two stages: the first stage is recursion, from... Start calculating to , The total number of words is recorded, and each state is also recorded as originating from a predecessor state, i.e., which word index makes the transition occur. The maximum; the second phase is backtracking, from the final state. Starting from the recorded predecessor state, trace backward to the initial state. All the split points on this path constitute the globally optimal partitioning scheme.

[0099] S4. Construct a key information extraction model for long-text official documents based on a bidirectional semantic encoder: This model takes semantic fragments and their corresponding substructure encoding sequences as input, and achieves accurate end-to-end extraction of key information from various types of official documents through structure-enhanced semantic encoding, cross-fragment bidirectional semantic propagation, and multi-granularity pointer network prediction. Figure 4 As shown.

[0100] S401, perform structural enhancement on fragment-level semantic encoding: For each input semantic fragment, fuse its word sequence, corresponding part-of-speech tag sequence, and strictly aligned substructure encoding sequence from the structural encoding sequence.

[0101] S40101 maps discrete word symbols and part-of-speech tags into dense vector representations through a trainable embedding lookup table;

[0102] S40102 transforms the structure encoding vector to the same dimension as the word vector using a trainable linear projection matrix. It then sums the word vector, part-of-speech vector, and the projected structure vector, and incorporates a standard sine / cosine absolute position encoding vector.

[0103] S40103 generates an initial enhanced semantic representation for each word after layer normalization.

[0104] Represented as: ,

[0105] In the above formula, Indicates the first In the semantic fragment, the first The initial fusion representation vector of each word has a dimension of . It integrates lexical, part-of-speech, structural, and positional information as the input to layer 0 of the Transformer encoder. The index representing the word position within a semantic segment, at the 1st... In each semantic fragment, The range of values ​​is Number of words in the segment , Indicates the first The number of words contained in a semantic segment The representation layer normalization operation is used to stabilize the training process of deep neural networks. The word embedding lookup table is of shape . The parameter matrix is ​​used to map discrete word symbols to... A dense vector of dimension 1 is a trainable parameter; Indicates the first The first semantic segment in the After obtaining the word sequence of the entire document through jieba word segmentation, the word sequence is cut into fragments according to the optimal partitioning boundary. Then it is the first The first segment in the segment One word, This represents a part-of-speech embedding lookup table, which is of shape [shape missing]. The parameter matrix is ​​used to map discrete part-of-speech tags to... A dense vector of dimension; Indicates the first In the semantic fragment, the first Jieba segmentation can simultaneously obtain the part-of-speech tags (POS) of each word. During segmentation, the POS tag sequence is also simultaneously cut. It is the first The first segment Part-of-speech tags for each word The structure encoding projection matrix has the following shape: , used to 3D structure encoding vector Linear transformation to the same latent space dimension as word vectors , are trainable parameters. The dimension of the structure encoding vector; Indicates the relationship with the first In the semantic fragment, the first The word-strictly aligned structured encoding vector, based on the character range corresponding to the segment, is derived from... Extracting the corresponding subsequence from the text requires averaging and aggregating the structural encoding vectors of all characters corresponding to the word, since the structural encoding is character-level while this is word-level. ;

[0106] The absolute position encoding vector is generated using sine and cosine functions, and has a dimension of [missing information]. , used for the first in the sequence Inject sequence information at each location. The Each element is defined as

[0107] ,in, Representing 10000 This power-law encoding method generates a unique and smooth vector representation for each position, enabling the model to capture the order information in the sequence and supporting expansion to sequence lengths not seen during training. This represents the dimension of the hidden layer of the model. It is a preset hyperparameter that determines the uniform size of all representation vectors within the model. An example value is 768.

[0108] S402, cross-fragment bidirectional semantic propagation coding;

[0109] The initial augmented representation sequence input for each semantic segment contains A layered bidirectional Transformer encoder, designed to model semantic dependencies across different segments in long texts, introduces a structure-guided cross-segment attention window into the encoder's self-attention mechanism. For each layer of the encoder, the query vector at a certain position within the current segment can not only focus on all positions within the current segment but also on positions within several adjacent segments in the original document order. The similarity of the structural encoding vectors at corresponding positions is used to apply a soft constraint to this cross-segment attention. The attention weights are calculated as follows:

[0110] ,

[0111] In the above formula, Indicates the first In the layer encoder, the first The first segment The position is the first The first segment The attention weight for the i-th position; a larger value indicates a stronger correlation, i.e., the i-th position... The first segment The word at the position (query point) and the The first segment The stronger the semantic association between words (keypoints) at each position; Represents the natural exponential function; Indicates the index of a neighboring segment; Indicates the first The set of fragment indices that can be followed for each fragment is defined as the set of fragments whose index distance does not exceed a preset radius distance. This is used to control computational complexity. An example value for the preset radius distance is 2. Indicates the first The number of words contained in a segment, i.e., the segment length;

[0112] The unnormalized attention score represents the correlation strength between the query point and the key point; a larger value indicates a stronger correlation. The calculation method is expressed as follows: ;

[0113] Indicates the first The first in the layer The first segment The query vector at each position is represented by the previous layer. After linear transformation To obtain, that is , express transpose; Indicates the first The query projection matrix of the layer encoder has the following shape: ; Indicates the first The first layer encoder output The first segment The hidden state vectors of each word; Indicates the first The first in the layer The first segment The key vector at each position is represented by the previous layer. After linear transformation To obtain, that is ; Indicates the first The key projection matrix of the layer encoder has the following shape: ; Indicates the first The first layer encoder output The first segment The hidden state vectors of each word; The dimension representing the key vector and query vector is typically set to [value]. Divide by the number of attention heads; for example, the value is 64. The weight coefficient representing the structural similarity bias is a hyperparameter used to adjust the degree of influence of structural similarity on attention scores; an example value is 0.3. express and Cosine similarity; This represents the cross-segment attention penalty coefficient, a hyperparameter used to impose a moderate penalty on positions that are not currently being attended to, in order to balance the allocation of attention resources within and between segments. An example value is 0.1. It is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise; Indicates the first The segment and the first The absolute distance of the index of each segment.

[0114] go through After stacking the layer encoders, a deep semantic representation of each word within each segment is obtained, which deeply fuses local context, long-distance dependencies, and structural priors. This is defined as follows: Indicates the first The output of the last layer encoder The first segment The deep semantic representation vector of each word is deeply integrated with local context, long-distance dependency and structural prior information.

[0115] S403, multi-granularity pointer network information extraction;

[0116] A multi-granularity pointer network is employed to jointly predict the start and end positions of each predefined key information type based on the deep representation of the encoder output, while simultaneously considering the granularity level of the information. During the prediction process, structural compliance features based on official document standards are incorporated to constrain the rationality of the output. For each information type... The probability of its starting position is calculated as follows:

[0117] ,

[0118] In the above formula, Indicates at the granularity level Below, with the first The semantic fragment of the first Each word as a type The probability of the starting position of key information; Represents the natural exponential function; Represents the granularity level, taken from a predefined discrete set. ;

[0119] The unnormalized score representing the starting position is calculated as follows: ; The projection vector, representing the score at the starting position, is a trainable parameter with dimension O(n). ; Represents the hyperbolic tangent activation function; The linear transformation matrix representing the initial position score is a trainable parameter with the following shape. ; Representation type A specific query vector is generated by fusing the global context vector and the type embedding vector, with a dimension of [missing information]. ; Indicates granularity level The corresponding learnable embedding vector has a dimension of ; This represents the bias term used in calculating the score at the starting position, and it is a trainable parameter. This represents a vector concatenation operation; The weight coefficient representing the structural conformity feature is a hyperparameter used to adjust the strength of the influence of prior rules on the prediction; an example value is 0.5. Indicates the first The output of the last layer encoder The first segment Deep semantic representation vector of each word

[0120] This indicates structural conformity characteristics used to assess location. In particle size Below as type The rationality of the information starting point is considered; a higher value indicates better compliance with official document standards. The calculation method is expressed as follows: ; Represents a set of document structure rules One of the specific rules in it; Indicates key information types The relevant set of document structure rules is a set of predefined structure rules based on knowledge of the document domain. For example, for the "issuing authority" type, the rules may include "must appear in the area of ​​document part type 'issuing authority'". Representation rules The confidence weights are preset hyperparameters used to adjust the importance of different rules in the structural compliance calculation. For example, for strong constraint rules, such as "issuing authority" must appear in the corresponding position, It can be set to 1.0, and for weak constraint rules it can be set to 0.5; This represents the matching function, which checks the structure encoding vector. Does it comply with the rules? Specified structural pattern If the condition is met, return 1; otherwise, return 0. Representation rules The defined structural pattern is part of the rules and is predefined by domain experts. For example, the rule "the issuing authority must appear in the 'issuing authority' section" corresponds to... This means "the document part type is 'issuing authority'"; Representation rules In particle size The applicable confidence level is a preset value between 0 and 1.

[0121] Beam position probability The calculation method is symmetrical to the starting position, but uses a separate set of trainable parameters, and finally achieves the joint probability through search. The type can be extracted from the largest start-end pair. Key information text fragments.

[0122] In order to extract the type A key information instance requires finding a set of starting positions. and end position This makes the joint probability of these two positions... In practical implementation, the Viterbi algorithm can be used for approximate search to avoid enumerating all possibilities. For example, for each segment and each granularity, several candidate starting positions with the highest probability are retained, and then the ending position with the highest probability is found for each candidate starting position. Finally, the combination with the highest joint probability is selected as the prediction result.

[0123] In this embodiment, the extracted key information text fragment is a triple: information type, text content, and granularity level. For example, for the "issuing authority" type, the extracted text fragment might be: issuing authority, "State Taxation Administration" phrase.

[0124] To accurately extract key information from long official documents, such as issuing authority, addressee, subject of clauses, specific matters, time limits, and date of issuance, conventional sequence labeling models usually treat the text as a flat sequence. This can easily overlook the long-distance semantic dependencies within the text and the constraints imposed on information boundaries by the unique hierarchical structure of official documents, leading to boundary errors or contextual ambiguities in the extraction results.

[0125] S5, Model Training and Trainable Parameter Update;

[0126] S501 calculates the loss function to guide model training;

[0127] To train the key information extraction model, a loss function that combines boundary prediction loss and structural consistency regularization is adopted. Boundary prediction is optimized by calculating the negative log-likelihood loss for each training sample. Simultaneously, a structural consistency regularization term is introduced to ensure that the structural encoding of the predicted start and end positions is similar to the structural encoding of the actual positions. This allows the model to accurately predict information boundaries while maintaining consistency with prior knowledge of the document structure, improving the rationality and robustness of the extraction results. This is expressed as:

[0128] ,

[0129] In the above formula, This represents the overall loss function, used to guide model training. Indicates the number of samples in the training batch; Indicates the first Key information types for each sample; Indicates at the granularity level Below, with the first The semantic fragment of the first Each word as a type The probability of the starting position of key information is represented by the probability at the first... In each sample, for type With the first The semantic fragment of the first The word is used as the starting position, and the granularity level is The probability of; Indicates the first The semantic fragment index of the actual starting position in each sample; Indicates the first The word position index within a segment where the actual starting position is located in each sample; Indicates the first The granularity level corresponding to the true starting position in each sample;

[0130] Indicates at the granularity level Below, with the first The semantic fragment of the first Each word as a type The probability of the starting position of key information; Indicates the first The semantic segment index where the actual end position is located in each sample; Indicates the first The word position index within a segment where the actual end position is located in each sample; Indicates the first The granularity level corresponding to the actual end position in each sample;

[0131] The weight coefficient representing the structural consistency regularization term is a hyperparameter used to balance the boundary prediction loss and the structural consistency loss. An example value is 0.5. This represents the cosine similarity function, used to calculate the similarity between two vectors, with a range of [-1, 1]. Indicates the first The structure encoding vector corresponding to the true starting position in each sample is obtained from the structure encoding sequence; Indicates the first The structure encoding vector corresponding to the true end position in each sample; Indicates the first The structural encoding vector corresponding to the starting position predicted by the model in each sample is calculated by the model based on the predicted position. Indicates the first The structure encoding vector corresponding to the end position predicted by the model in each sample.

[0132] S502, iterative model training;

[0133] The constructed key information extraction model is iteratively trained to update all its trainable parameters. The training process is carried out on a prepared training set, and optimization algorithms such as AdamW are used to minimize the overall loss function. .

[0134] S50201: At the beginning of each training cycle, the document samples and their labeled data in the training set are processed by the structured parsing and hierarchical encoding in S2 and the semantic fragment segmentation in S3 to form batch data that the model can directly input. Each batch contains multiple samples, and each sample contains word sequences, part-of-speech sequences, structured encoding sequences, and real key information boundaries and type labels.

[0135] S50202, the forward propagation process of the model follows S4, performing segment-level semantic encoding with structural enhancement and bidirectional semantic propagation encoding across segments in sequence, and finally calculating the probability distribution of the start and end positions of various key information through a multi-granularity pointer network;

[0136] S50203, based on the loss function defined in S501, calculates the sum of the boundary prediction negative log-likelihood loss and the structural consistency regularization loss for the current batch of data, and calculates the gradient of the loss with respect to all trainable parameters of the model through the backpropagation algorithm.

[0137] The optimizer uses these gradients to update parameters, gradually bringing the boundaries of the key information predicted by the model closer to the boundaries of the true labels, while making the structural encoding of the predicted location more similar to the structural encoding of the true location.

[0138] S50204: During training, model performance is periodically evaluated on an independent validation set, monitoring metrics such as precision, recall, and F1 score for key information extraction. Training continues until the core performance metrics on the validation set no longer improve over multiple consecutive training epochs, or until the preset maximum number of training epochs is reached. At this point, the model is considered to have learned sufficiently and reached convergence, and iterative training is stopped.

[0139] S50205: After training, save the set of model parameters with the best performance on the validation set as the final model that can be used for actual document key information extraction tasks.

[0140] S6, Extraction of key information from long-text official documents;

[0141] S601. For a new long document to be processed, the rule-based and automata-based structured parsing method defined in S2 is used to parse the original text, identify all structural markers, and calculate the corresponding multi-dimensional structure encoding vector for each character to generate the structure encoding sequence of the entire document.

[0142] S602 applies the topic-enhanced dynamic semantic segmentation method in S3, using a pre-trained topic model and dynamic programming algorithm to segment long texts into multiple semantically complete segments, and records the word sequence, part-of-speech sequence, and aligned substructure encoding sequence extracted from the global structure encoding sequence for each segment.

[0143] S603: Load the optimal model parameters obtained during the training phase, initialize the key information extraction model, and input each semantic segment and its associated information (word sequence, part-of-speech sequence, substructure encoding sequence) obtained after preprocessing into the key information extraction model with loaded parameters. The model performs a forward propagation process. After structural enhancement encoding and cross-segment bidirectional semantic propagation encoding, it uses a multi-granularity pointer network to calculate the probability distribution of all possible positions (across segments) in the entire document for each predefined key information type (such as issuing authority, addressee, etc.) as the start and end points.

[0144] S604. For each type of key information, the Viterbi beam search algorithm is used to find the position that maximizes the joint product of the start probability and the end probability. Combined with the predicted granularity level, one or more key information instances of that type can be extracted. The extracted results are organized into a structured output, such as a list or dictionary recording the type of each information instance, the corresponding original text content, the segment index it belongs to, and the start and end positions.

[0145] S7. Constructing an intelligent agent for extracting key information from long-text official documents based on a large model: Building upon the extraction of key information from long-text official documents, to further enhance the system's intelligence and application value, an intelligent agent system with a large language model as the core coordinator is constructed. This agent uses the structured key information extracted in S6 as precise and condensed contextual knowledge, combining it with the AIGC large model, which possesses powerful semantic understanding, reasoning, and generation capabilities, to form a collaborative workflow. For example... Figure 5 As shown.

[0146] The intelligent agent system is deployed as a backend service, and its core workflow consists of three parts: task parsing, tool invocation, and interaction with a large model.

[0147] S701, when a user submits a natural language request involving document understanding or processing (e.g., “Summarize the core requirements of this notice” or “Find all clauses related to ‘safe production’”), the agent first calls the built-in large model to perform intent parsing and task decomposition of the user request;

[0148] S702, for the subtask that requires extracting specific information from official documents, the intelligent agent automatically schedules and executes S2 to S6 of this embodiment, that is, extracts key information from the associated official documents throughout the entire process to obtain accurate structured data.

[0149] In S703, the agent uses these extracted results as reliable and verified factual evidence, combining them with the user's original question and portions of the official document (such as relevant excerpts) to construct high-quality prompts, which are then submitted to the large model for in-depth analysis and content generation. Guided by this context rich in structured information, the large model can generate highly accurate, document-compliant, and logically rigorous responses, such as generating document summaries, extracting specific clause lists, answering specific questions about the document content, and even automatically generating execution points or timelines based on the extracted "specific matters" and "time limits."

[0150] In addition, the intelligent agent can also use the logical judgment ability of the large model to verify and correct the initial extraction results of S6, such as judging whether the extracted "specific matters" are semantically complete, or identifying potential contradictions.

[0151] The intelligent agent not only achieves a leap from "extracting information" to "understanding and applying information", but also reliably anchors the capabilities of the large model on the structured facts of the official document, avoiding the illusions or biases that may arise. Ultimately, it builds an intelligent agent system that can proactively understand user intentions, accurately process the content of the official document, and provide deep intelligent services.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention.

Claims

1. A method for extracting key information from long-text official documents based on a large model, characterized by: Includes the following steps: S1. Collect raw long text data of official documents, preprocess and label the raw long text data of official documents, divide the labeled dataset into training set, validation set and test set to form benchmark data for training and evaluating the long text document key information extraction model. S2 performs structured parsing on the original official document text data, extracts its inherent hierarchical structure and format features, calculates the corresponding multi-dimensional structure code for each character, and generates the structure code sequence of the entire document. S3, based on topic model-guided dynamic semantic segmentation, divides long texts into multiple semantically complete segments to ensure that each segment carries relatively independent semantic information; S4. Construct a long text document key information extraction model based on bidirectional semantic encoder: This model takes semantic fragments and their corresponding substructure encoding sequences as input, and achieves accurate end-to-end extraction of key information of various documents through structure-enhanced semantic encoding, cross-fragment bidirectional semantic propagation and multi-granularity pointer network prediction. The method for constructing a long-text document key information extraction model based on a bidirectional semantic encoder in S4 includes the following steps: S401, structural enhancement of fragment-level semantic coding; S402, Cross-segment bidirectional semantic propagation coding: Inputting the initial enhanced representation sequence of each semantic segment into... The bidirectional Transformer encoder of the layer introduces a structure-guided cross-segment attention window into the encoder's self-attention mechanism, and uses the similarity of the structural encoding vectors at corresponding positions to apply soft constraints to the cross-segment attention. S403, Multi-granularity pointer network information extraction: Using a multi-granularity pointer network, based on the deep semantic representation of the encoder output, the start and end positions of each predefined key information type are predicted, and the granularity level of the information is considered simultaneously. In the prediction process, structural compliance features based on official document norms are incorporated to constrain the rationality of the output. S5, Model Training and Trainable Parameter Update; S6, Extraction of Key Information from Long Text Documents: Extracts key information from related documents throughout the entire process to obtain accurate structured data; S7. Construct an intelligent agent for extracting key information from long-text official documents based on a large model.

2. The intelligent agent method for extracting key information from long-text official documents based on a large model as described in claim 1, characterized in that: The types of long text data in S1 include notices, announcements, reports, opinions, letters, and minutes.

3. The intelligent agent method for extracting key information from long-text official documents based on a large model as described in claim 1, characterized in that: The data annotation types in S1 include issuing authority, addressee, subject of clause, specific matter, time limit, and date of issuance.

4. The intelligent agent method for extracting key information from long-text official documents based on a large model as described in claim 1, characterized in that, The method for structured parsing of the original official document text data in S2 includes the following steps: S201, based on the parsing rules of regular expressions and finite state automata, scans the original official document text, identifies all information tags that represent the document structure and hierarchy, constructs a finite state automaton according to the official document structure syntax, and uses the tags identified by the regular expression as input to trigger state transitions, which are used to determine the type and nesting relationship of the tags. S202, Calculate character-level multidimensional structure encoding: For each character in the original long text data of the official document, according to all the identified information tags representing the document structure and hierarchy, the document part type, nesting level depth and visual format features are fused, and a multidimensional structure encoding vector is assigned to it through the hierarchical encoding function. S203, Generate a structured encoding sequence: Traverse all characters in the original document text, apply the structured encoding function to each character to obtain the structured encoding vector corresponding to each character, and arrange the structured encodings of all characters in the original text order to form a complete structured encoding sequence.

5. The intelligent agent method for extracting key information from long-text official documents based on a large model according to claim 1, characterized in that, The method for dynamic semantic fragment segmentation based on topic model guidance in S3 includes the following steps: S301, preprocess the long text of the official document and calculate the salience weight of the topic; S302, calculates the semantic segment scoring function, which comprehensively evaluates the quality of any candidate semantic segment composed of a continuous word sequence from three dimensions: topic consistency, semantic tightness and length fit. S303, Optimal Segment Partitioning Based on Dynamic Programming: Using period, semicolon, question mark, exclamation mark, newline punctuation mark, and structural marker information as candidate points, the dynamic programming algorithm is used to find the globally optimal segment partitioning scheme that maximizes the total score of all segments. S304, Generate fragment-level output: Align and cut the original word sequence, part-of-speech sequence, and structural tag information according to the optimal partition boundary to generate structured data corresponding to each semantic fragment.

6. The intelligent agent method for extracting key information from long-text official documents based on a large model according to claim 5, characterized in that, The method for preprocessing long official document texts in S301 is as follows: removing garbled characters, unifying full-width and half-width characters, and segmenting the text into sentences; The method for calculating topic saliency weights in S301 is as follows: using the jieba word segmentation tool to segment words to obtain word sequences and their part-of-speech sequences, using a latent Dirichlet distribution topic model pre-trained on a large-scale official document corpus to infer the topic distribution of the current document, and calculating the topic saliency weight of each word to measure the importance of the word and its contribution to the core topic of the document.

7. The intelligent agent method for extracting key information from long-text official documents based on a large model according to claim 1, characterized in that, The method for structural enhancement of fragment-level semantic coding in S401 is as follows: S40101 maps discrete word symbols and part-of-speech tags into dense vector representations through a trainable embedding lookup table; S40102 transforms the structure encoding vector to the same dimension as the word vector using a trainable linear projection matrix. It then sums the word vector, part-of-speech vector, and the projected structure vector, and incorporates a standard sine / cosine absolute position encoding vector. S40103 generates an initial enhanced semantic representation for each word after layer normalization.

8. The intelligent agent method for extracting key information from long-text official documents based on a large model according to claim 1, characterized in that, The method for model training and trainable parameter update in S5 includes the following steps: S501, calculate the loss function to guide model training: adopt a loss function that combines boundary prediction loss and structural consistency regularization. Optimize boundary prediction by calculating the negative log-likelihood loss of each training sample. At the same time, introduce a structural consistency regularization term to ensure that the structural encoding of the predicted start and end positions is similar to the structural encoding of the true positions. This enables the model to accurately predict information boundaries while maintaining consistency with prior knowledge of the document structure, thereby improving the rationality and robustness of the extraction results. S502, Iterative Model Training: The constructed key information extraction model is iteratively trained to update all its trainable parameters. The training process is carried out on the prepared training set, and an optimization algorithm is used to minimize the overall loss function. Once the preset maximum number of training cycles is reached, the model is determined to have learned sufficiently and reached a convergent state. Iterative training is then stopped. After training is completed, the model parameter set with the best validation set performance is saved as the final model that can be used for actual document key information extraction tasks.

Images