An AI-based multi-document automatic parsing method, system and platform

Abstract

This invention provides an AI-based method, system, and platform for automatic multi-document parsing. The method includes: acquiring a candidate set of ghost references and performing cyclical sequential joint inference using a conditional random field to obtain a ghost reference set; acquiring four-dimensional feature vectors based on the ghost reference set and the original document, constructing a Bayesian network using these vectors, and obtaining the inference result through inference via the Bayesian network; creating a document heterogeneous graph based on the ghost reference set and the original document, generating aggregated text node vectors through attention aggregation based on multiple edge types, and obtaining community results using a community adversarial strategy; constructing a document dynamic graph based on the community results and temporal data, obtaining hidden states by processing the temporal and spatial dimensions of the document dynamic graph, and outputting the parsing result. This method uses ghost references as a unified analytical clue and central feature to construct a multi-dimensional joint evaluation intelligent parsing framework to improve the depth of document parsing.

Description

Technical Field

[0001] This invention relates to the field of document parsing technology, and more specifically, to an AI-based method, system, and platform for automatic parsing of multiple documents. Background Technology

[0002] With the increasing number of digital documents, how to automatically and accurately evaluate the quality and credibility of individual documents from a large and heterogeneous collection of documents, and to gain insight into the knowledge connections and evolution of documents, has become a major issue in the fields of knowledge management and information evaluation.

[0003] Traditional methods rely mainly on manual review or simple automated rules, which are inefficient, subjective, and difficult to scale. At the same time, some existing automated evaluation techniques often use isolated analytical dimensions, such as link-based methods. For example, the improved PageRank algorithm focuses on evaluating the authority of a document on the network, but ignores its inherent content integrity and logical consistency.

[0004] Of particular note is the lack of systematic exploration and utilization of the common yet information-rich problem of invalid or missing references within documents (collectively referred to as "ghost references") in existing technologies. Ghost references are not only an important negative indicator of document quality, but their distribution patterns may also imply deep knowledge connections between documents that are not explicitly linked. Summary of the Invention

[0005] In view of the aforementioned problems, and in conjunction with the first aspect of the present invention, embodiments of the present invention provide an AI-based automatic multi-document parsing method, comprising:

[0006] The ghost reference candidate set is obtained by preprocessing the original document, a six-dimensional feature vector is extracted based on the ghost reference candidate set, and a cyclical serialization joint inference is performed in combination with a conditional random field to obtain the ghost reference set.

[0007] A four-dimensional feature vector is obtained based on the ghost reference set and the original document. A Bayesian network is constructed by combining the four-dimensional feature vector. The inference result is obtained after inference through the Bayesian network.

[0008] A document heterogeneous graph with multiple types of nodes and edges is created based on the ghost reference set and the original document. Attention aggregation is performed based on the multiple types of edges to generate aggregated text node vectors. Community adversarial strategies are combined to obtain community results.

[0009] A document dynamic graph is constructed based on community results and time-series data. Hidden states are obtained after processing the time and spatial dimensions of the document dynamic graph, and maturity results are obtained by combining it with a maturity classifier.

[0010] Dempster synthesis is performed based on inference results, community results, and maturity results to output analytical results.

[0011] Furthermore, embodiments of the present invention also provide an AI-based multi-document automatic parsing system, comprising:

[0012] The AI-based multi-document automatic parsing system includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the AI-based multi-document automatic parsing method of the embodiment.

[0013] Furthermore, embodiments of the present invention also provide an AI-based multi-document automatic parsing platform, comprising:

[0014] The initial extraction sub-platform is used to extract a candidate set of ghost references from the original document and combine it with a conditional random field to perform cyclical serialization joint inference to obtain the ghost reference set;

[0015] The first parsing sub-platform is used to combine ghost reference sets, original documents and Bayesian networks to perform inference parsing in order to obtain inference results;

[0016] The second parsing sub-platform is used to perform community parsing by combining ghost reference sets, original documents, and community adversarial strategies to obtain community results.

[0017] The third analysis sub-platform is used to combine community results and time-series data to perform maturity analysis in order to obtain maturity results.

[0018] The results output platform is used to output reasoning results, community results, and maturity results as analytical results.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] This invention constructs a multi-dimensional joint evaluation intelligent parsing framework using ghost references as a unified analytical clue and central feature. Starting from the micro-defect of ghost references, it gradually rises to document-level features, community-level associations, and time-level evolution, thereby outputting the final parsing result and improving the depth of document parsing. Attached Figure Description

[0021] Figure 1 The diagram illustrates the execution flow of an AI-based automatic multi-document parsing method that can implement the ideas of this application, according to some embodiments of this application.

[0022] Figure 2 The diagram illustrates an AI-based automatic multi-document parsing system that can implement the ideas of this application, according to some embodiments of this application.

[0023] Figure 3 The illustration shows a schematic diagram of an AI-based multi-document automatic parsing platform that can implement the ideas of this application, according to some embodiments of this application. Detailed Implementation

[0024] The present invention will now be described in detail with reference to the accompanying drawings. Figure 1 The diagram illustrates the execution flow of an AI-based automatic multi-document parsing method that implements the ideas of this application, according to some embodiments of this application. The following is a detailed description of this AI-based automatic multi-document parsing method.

[0025] Specifically, an AI-based method for automatic parsing of multiple documents includes:

[0026] Step 1: Obtain a ghost reference candidate set by preprocessing the original document, extract a six-dimensional feature vector based on the ghost reference candidate set, and perform cyclical serialization joint inference in combination with conditional random field to obtain the ghost reference set.

[0027] In this embodiment, step 1 further includes:

[0028] Step 1-1: Perform preprocessing on the original document, including at least document extraction, anchor point identification, and directional extraction, to generate an anchor point candidate library and a directional expression set, and match them to construct a ghost reference candidate set.

[0029] In this embodiment, step 1-1 further includes:

[0030] Step 1-1-1: After determining the format of the original document, extract the content according to the format to obtain the text structure data.

[0031] After the platform obtains the original document, it automatically determines the document's format based on the file extension or header information. This typically includes PDF, Word, HTML, or plain text files. The corresponding parsing library is then called to process the original document. For PDF files, libraries such as PyMuPD or pdfplumber are used to extract text, font, and position information, and to attempt to identify the table of contents and page layout. For Word files, libraries such as python-docx are used to parse the document structure based on its inherent styles and paragraph levels. For HTML files, libraries such as BeautifulSoup are used to parse HTML tags to obtain content and structure. For plain text files, they can be read directly, and initial segmentation is performed using simple heuristic rules such as specific labels and blank lines.

[0032] After extracting the basic text content from the original document, the document analyzes the text's formatting features (e.g., font, size, indentation), numbering patterns (e.g., 1.1), and semantics. The document content is then segmented and categorized into different structural units, including but not limited to document titles, chapters, sections, paragraphs, tables, footnotes, endnotes, reference entries, appendices, etc. A globally unique ID is generated for each identified structural unit. The text content in all structural units is then concatenated or arranged in sequence to generate a complete and continuous plain text version, i.e., text structure data. The physical location of each structural unit in the original document is recorded, including the starting page number and the bounding box coordinates on that page.

[0033] In some possible implementations, the input may be a scanned PDF, in which case an OCR engine needs to be invoked to process it. The OCR engine can convert the pixel information in the image into calculable and searchable text, thereby performing layout analysis on the images on each page, distinguishing text blocks, images, tables, etc., and performing character recognition on the text blocks. The text blocks recognized by the OCR engine are remapped to structural units according to their position, font, and content, and assigned a unique ID.

[0034] Step 1-1-2: Obtain standardized text fragments based on text structure data, perform semantic encoding on the standardized text fragments, and then build an index to generate an anchor point candidate library.

[0035] Iterate through each structural unit in the text structure data, extract the title text of the structural unit, such as the chapter title "2.3 Research Methods". If there is no explicit title, generate descriptive text based on its type and position, such as "the first table on page 5 of document d1". Then, extract the content summary text from the main content of the structural unit. Specifically, extract the first or last paragraph of the unit content, and use a text summarization algorithm to extract 1-2 key sentences from the middle part. Concatenate the first or last paragraph with the key sentences to form the content summary text of the unit. At the same time, extract the paragraph immediately before and after the chapter in which the unit is located, and use them as context text.

[0036] Understandably, preparing three types of text materials with different focuses—title text, content summary text, and context text—can provide a complete anchor point for Uncle Cat from three dimensions: identity recognition, content summary, and environment, thereby improving the success rate of finding the corresponding target through semantic matching.

[0037] The Sentence-BERT model, based on the Siamese BERT network, is used to encode texts of varying lengths into dense vectors rich in semantic information. The title text, content summary text, and context text of each structural unit are input into the Sentence-BERT model to obtain title vectors, content summary vectors, and context vectors. The title vector is mainly used to match the target name explicitly mentioned in the citation, the content summary vector is mainly used to match citations that describe the target content but do not specify the exact name, and the context vector is mainly used to match citations that are context-related.

[0038] After associating each structural unit with three types of vectors, a unique ID, and the document ID, an index is created. FAISS is used to store all vectors and build the index. Typically, separate indexes are created for the three types of vectors, or a composite index containing all vectors with type identifiers is created to generate an anchor candidate library. This transforms the text structure data into machine-understandable semantic vectors and provides retrieval capabilities for subsequent matching.

[0039] Step 1-1-3: After performing regular expression localization on the text structure data, dependency parsing is performed on the localization results based on natural language processing, and a set of directional expressions is generated through citation classification.

[0040] Write a series of regular expression rules to scan text to match common quoting patterns, including but not limited to explicit trigger words and implicit indicator structures.

[0041] For explicit trigger words, such as “see,” “refer to,” “see details,” “as shown in the figure below,” “as shown in formula (1),” etc., for implicit indicator structures such as “as described above,” “as will be explained below,” or phrases containing nouns such as “chapter,” “figure,” “table,” “appendix,” “formula,” etc., and preceded by prepositions such as “as,” “in,” etc.

[0042] Using a pre-compiled set of regular expression rules, the entire document's plain text is scanned, and all text fragments that match the pattern, along with their start and end positions in the text, are recorded and used as candidate sentences.

[0043] Natural language processing tools are invoked to perform dependency parsing on candidate sentences, thereby obtaining a grammatical dependency tree between words in the sentence. The referencing trigger words of verbs or prepositions in the dependency tree are then identified, and their grammatical objects or modifiers are found. The text of these objects or modifiers is the target name to be extracted. For example, in the sentence "See Appendix B for details," by analyzing the verb-object relationship between "see" and "Appendix B," "Appendix B" can be accurately extracted. The extracted target name text, the complete sentence containing it, and the sentence's position information in the document are bound and stored as a preliminary representation object. Simultaneously, the context information of the representation object is expanded. Taking the sentence containing the representation as the center, one complete sentence is taken forward and one sentence backward. If it is at the beginning or end of a paragraph, the sentence is taken up to the paragraph boundary.

[0044] Understandably, regular expressions can only find the approximate location, but cannot accurately define the boundaries of the referenced target, such as whether it is "Appendix B" or "Explanation of Appendix B". Dependency parsing can understand the logical relationships between sentence components, thereby extracting the referenced target entity text.

[0045] Then, the trigger words or sentence structures of each expression are analyzed. A rule-based classification method (analyzing verb roots) or a lightweight text classifier is used to determine the expression's intent based on its word choice and sentence structure, labeling it with a tag indicating the intended reference. This tagging includes definitions / references, explanations / examples, supplements, reasoning / basis, and comparisons. Definitions / references refer to a standard or definition, such as "according to RFC791" or "defined as," while explanations / examples refer to charts or examples, such as "as..." Figure 1 "As shown" and "for example" are used to indicate appendices or supplementary materials, such as "more data in the appendix". "Reasoning / basis" refers to the conclusions or data in the preceding text as the basis, such as "based on the findings in Section 2". "Comparison" refers to comparison with other parts, such as "different from scheme A", thereby generating directional statements and integrating all directional statements into a set of directional statements.

[0046] Step 1-1-4 involves progressively matching the set of directional expressions using the anchor point candidate library to construct a ghost reference candidate set. The progressive matching includes first-level, second-level, and third-level matching, with each level of matching progressing sequentially.

[0047] The first-level query is performed, using the target name of each directional expression in the directional expression set as a query key. A string matching query is performed in the anchor candidate library. If the match is successful, it means that there is indeed a structural unit in the document whose title or ID is exactly the same as this target name, indicating that the expression is a normal reference (not a ghost). If the match fails, the query continues to the second-level query.

[0048] Entering the second-level query, the edit distance (Lewinstein distance) between the target name in the directional expression set and the title text of all anchors in the anchor candidate library is calculated. This distance measures the minimum single-character edit required to convert one string into another. All anchors with an edit distance less than or equal to a preset threshold are identified. If there is one and only one anchor, it usually means that there are minor spelling errors, spaces, or punctuation differences in the target name, such as "Chapter 2.1" and "Chapter + space + 2.1", which is determined to be a normal reference. If there are multiple anchors or none, it indicates that the target name may be ambiguous or the error exceeds the tolerance range. At this time, the process proceeds to the third-level matching.

[0049] Entering the third-level matching, using the same encoding method as in step 1-1-2, the context of the directional expression, i.e., the preceding, middle, and following text, is encoded into a query vector. Then, the query vector and the context vector of the anchor point are subjected to an approximate nearest neighbor search within the index partition of the candidate anchor point library to obtain the top K anchor points most similar to the query vector and their corresponding cosine similarity scores. The highest cosine similarity score is taken. If it is greater than a preset threshold, such as 0.85, it indicates that the context is highly semantically related to the context of a certain anchor point. Even if the literal name does not match, it is very likely to point to that anchor point, and it is determined to be a normal index. If it is less than the preset threshold, it indicates that even at the semantic level, a highly related existing anchor point cannot be found, and it is determined to be a ghost reference candidate. All ghost reference candidates are summarized to generate a ghost reference candidate set.

[0050] Steps 1-2 involve extracting a six-dimensional feature vector, including matching features, text features, and reference features, through the ghost reference candidate set and matching construction process, combined with the anchor point candidate library.

[0051] Among them, the matching features include similarity features and candidate quantity features. The highest cosine similarity score generated during the three-level matching mentioned above is used as the similarity feature, which can quantify the intensity of not being able to find the target and record it as the similarity feature. The candidate quantity feature is directly the number of ghost reference candidates, which can reflect the ambiguity or vagueness of the target reference.

[0052] The text features include normative features and proportion features. For normative features, rules are used to determine whether the target name conforms to common document internal citation norms. For example, it is checked whether it contains numerical codes, whether it starts with a specific keyword, whether it uses uppercase letters or specific delimiters, etc. A normative score between 0 and 1 is generated by scoring according to the rules or training a binary classifier, which is used as the normative feature. The higher the normative feature, the more likely the target name is to be arbitrarily written or non-existent. For proportion features, the proportion of numeric characters in the target name string is calculated as the proportion feature. A normative citation usually has a reasonable proportion of numeric characters, while a high or low proportion may indicate a problem.

[0053] The citation features include context features and citation intent features. For context features, the three sentences of text before, during, and after the ghost citation candidate are segmented and input into the Word2Vec model to output word vectors. The average vector of the word vectors is calculated and used as an estimate of the semantic center of the context. Then, the variance of the cosine distance of each word vector to the center is calculated and used as the context feature. The higher the value, the more dispersed the semantics of the context words and the less focused the topic. This indicates that the argument logic of the statement itself is relatively chaotic and there is a possibility of citation errors. For citation intent features, the labels of citation intents generated in step 1-1-3 are one-hot encoded and used as citation intent features. For example, if there are 5 categories of citation intents [definition, explanation, supplement, reasoning, comparison], they can be encoded as [0,1,0,0,0]. Different citation intents correspond to different ghost citation risk probabilities. For example, citing the definition incorrectly may be more serious than citing the supplementary material incorrectly.

[0054] Steps 1-3 involve performing serialization joint inference based on six-dimensional feature vectors and combined with conditional random fields, and outputting the inference results. The conditional random field includes at least a graphical model based on linear chain conditional random fields, node potential functions, and edge potential functions.

[0055] The ghost reference candidate set is defined as an observation sequence. Each ghost reference candidate corresponds to a hidden label that needs to be predicted. Then, a linear chain random field is used as a graph model. In this chain, each ghost reference candidate is a node and its state is the hidden label. Undirected edges are established between adjacent candidate nodes to form a chain, which is used to capture the mutual influence of adjacent reference judgment results in the sequence.

[0056] Understandably, references in a document are not isolated. For example, in a continuous argument, if the previous reference is identified as a ghost reference, then the next reference pointing to an ambiguous target is also more likely to be a ghost reference. Linear chain random fields can explicitly model such dependencies between adjacent labels.

[0057] Define a node potential function that evaluates the local probability that a candidate node will be assigned a hidden label given a six-dimensional feature vector. Its expression is:

[0058] ;

[0059] in, This is represented as the i-th hidden label. Let i be the i-th given feature. Represented as a weight parameter vector, Represented as a vector of characteristic functions, its combination and For example, a feature function can be given as when and The function outputs 1 if the similarity feature is less than 0.3, otherwise it outputs 0.

[0060] Define a potential function that evaluates neighboring hidden label pairs. The probability of co-occurrence is used to model the transition patterns between hidden labels, and its expression is:

[0061] ;

[0062] in, To learn the parameter vector, This can be represented as a vector of transition feature functions, for example, a feature function can be given when... and When there are two consecutive ghost reference candidates, the output is 1.

[0063] Based on the graphical model, node potential function, and edge potential function of linear chain conditional random fields, a conditional random field model is constructed. The random field model is trained using manually labeled real label sequences and six-dimensional feature vectors as training data. Maximum likelihood estimation is used as the training benchmark. The goal is to find a set of model parameters that maximizes the sum of conditional probabilities of the real label sequences across all training data. This allows the model to determine the importance of each feature in the six-dimensional feature vector to ghost references and to clarify the transition patterns between labels.

[0064] After training, the trained Conditional Random Field model can be used to infer ghost reference candidates. Using a forward-backward algorithm, the hidden label is calculated for each i. The posterior probability of 1 is the inference result.

[0065] Steps 1-4 involve evaluating the uncertainty of the inference results to obtain re-inference data. The conditional random field then performs iterative inference based on the re-inference data until a termination condition is met, at which point a ghost reference set is output.

[0066] For the inference result p, determine its confidence level. Simultaneously calculate its information entropy. The higher the entropy value, the greater the uncertainty of the model. A confidence threshold is preset, and ghost reference candidates below this threshold are regarded as samples of model uncertainty. The top k ghost reference candidates with the highest entropy values ​​are selected from the samples and manually labeled to obtain true hidden labels. These are used as the augmented training set, i.e., re-inference data. The model is retrained and inferred using the re-inference data until the maximum number of iterations is reached or all iterations are higher than the confidence threshold. Then the iteration stops, and in the last round of model inference, the final ghost reference set with high confidence is output, thus providing more reliable basic data for subsequent document parsing.

[0067] Step 2: Obtain four-dimensional feature vectors based on the ghost reference set and the original document, construct a Bayesian network by combining the four-dimensional feature vectors, and obtain the inference result after inference through the Bayesian network.

[0068] In this embodiment, step 2 further includes:

[0069] Step 2-1: Calculate the reference density feature, type distribution feature, semantic dispersion feature, and temporal concentration feature based on the ghost reference set and the original document, and construct a four-dimensional feature vector.

[0070] For each original document, iterate through the ghost references, count the total number of ghost references in the original document, obtain the total number of words in each original document, and calculate the citation density feature, which represents the number of ghost references per thousand words.

[0071] Ghost citations are categorized according to predefined target types, including but not limited to chapters, figures, appendices, references, URLs, and free text. The number of ghost citations belonging to each type is counted, and the Shannon entropy of these type distributions is calculated using the following formula: ,in Represented as Shannon entropy, This represents the total number of references for the ghost. The number of ghost references belonging to type e is represented by the number of ghost references. As a characteristic of type distribution A lower value indicates that the problem is concentrated in a certain type of target, which may point to a specific defect. The citations are missing and irregular, scattered across various targets, indicating that the overall organization of the document is chaotic or poorly written.

[0072] Using the Word2Vec model, each ghostly referenced target name is encoded as a word vector. All ghostly reference word vectors are used as input, and the K-Means clustering algorithm is applied for clustering. The number of clusters can be set according to the scale. After clustering, the number of target names in each cluster is counted, and then the distribution entropy of the cluster size, i.e., the semantic dispersion feature, is calculated. Where N represents the number of clusters, n represents the number of clusters, and c represents the number of target names. A higher value indicates that the semantic probability of ghost references is dispersed, and the overall document has missing links in multiple unrelated knowledge points.

[0073] Query the historical versions of the original document, iterate through all its ghost references, record the document version number where each ghost reference first appears, find the document version number with the most ghost references, and record the ghost reference count for that version. Calculate the time concentration, i.e., the time concentration characteristic: , where V represents the total number of versions.

[0074] Step 2-2: Define the network structure of parent nodes, child nodes and corresponding dependencies using four-dimensional feature vectors, and perform conditional probability distribution modeling based on Gaussian mixture model to generate Bayesian network.

[0075] The reference density feature, type distribution feature, semantic dispersion feature, and temporal concentration feature are respectively taken as parent nodes, i.e., observed variables, and the reference entropy, the continuous variable to be inferred, is taken as child nodes, i.e., child nodes. This represents the overall information uncertainty and structural fragility of the document caused by ghost references after combining all features. At the same time, when all parent nodes point to child nodes, it can be indicated that the probability distribution of reference entropy is jointly affected by the four feature values.

[0076] Define the probability distribution of child nodes under parent nodes. Since child nodes are continuous variables and their relationship with multiple parent nodes may be a complex nonlinear relationship, a Gaussian mixture model is used to model this conditional probability distribution. The Gaussian mixture model can approximate any complex smooth distribution by weighted sum of multiple Gaussian distributions.

[0077] This conditional distribution can be represented as:

[0078] ;

[0079] in, Represented as a four-dimensional feature vector, where H represents the values ​​of the parent node, H is the reference entropy (i.e., the child node), and M represents the number of Gaussian distributions. Let it be represented as the probability density function of the m-th Gaussian distribution. Let be the mean of the m-th Gaussian distribution. Let the variance be the variance of the k-th Gaussian distribution. It represents the mixed weights of the m-th Gaussian distribution, ranging from 0 to 1.

[0080] Steps 2-3 involve using a Bayesian network to perform probabilistic inference on the original document and calculate the mean and confidence interval labels, with the mean and confidence interval labels as the output inference results.

[0081] Using the labeled training set, the network is trained with maximum likelihood estimation to learn all the parameters of the conditional distribution mentioned above. After training, for the document to be parsed, the four-dimensional feature vector of the document is input into the Bayesian network as observation evidence, and an approximate inference algorithm is used to infer the posterior distribution of the child nodes.

[0082] In some possible implementations, variational inference or Markov chain Monte Carlo can be used. Variational inference is a deterministic approximation method that approximates the true posterior using a simple distribution and optimizes it to be as close as possible to the true posterior. It is fast and suitable for large-scale inference. Markov chain Monte Carlo is an approximation method based on random sampling that estimates the posterior by generating a chain of samples from the posterior distribution. It is highly accurate but usually computationally more expensive. The choice of method depends on the specific circumstances in the implementation.

[0083] After obtaining the posterior distribution, the mean of the posterior distribution is calculated. The mean can be used as a point estimate of the document's citation entropy. It integrates all feature information and is a single numerical summary of the uncertainty of document information. At the same time, based on the posterior distribution, the highest posterior density interval or percentile interval of its 95% is calculated to obtain the confidence interval label, which quantifies the uncertainty of the estimate. The wider the interval, the more uncertain the model's inference of citation entropy based on existing features may be. The mean and confidence interval label are used as the inference result output.

[0084] Step 3: Create a document heterogeneous graph with multiple types of nodes and edges based on the ghost reference set and the original document. Generate aggregated text node vectors by attention aggregation based on the multiple types of edges, and obtain community results by combining community adversarial strategy.

[0085] In this embodiment, step 3 further includes:

[0086] Step 3-1: Create document nodes, ghost nodes, and content nodes based on the ghost reference set and the original document, and create multi-type edges to obtain a document heterogeneous graph, wherein the multi-type edges are represented as an exhaustive combination of node types.

[0087] Create three different types of nodes: document nodes, ghost nodes, and content nodes.

[0088] The document node is created for each document in the original document, and it is also the target node. The ghost node is created by traversing all ghost references and extracting all unique target names, such as Appendix B. Figure 3 2. Create a node for it, which represents the general, real or unmet references in the document. It can be used to discover implicit connections between documents. The content node is to perform full-text word segmentation on all original documents and calculate the TF-IDF value of each word. Select the top X words with the highest TF-IDF values ​​as the content words representing the main body of the entire document set. Create a node for each content word to carry the semantic main information of the document.

[0089] Creating three different types of edges—document-ghost edges, document-content edges, and ghost-content edges—can be understood as an exhaustive combination of each pair of node types described earlier.

[0090] The document-ghost edge establishes an undirected edge between the corresponding document node and the ghost node in the graph. The edge weight is the number of times the ghost reference appears in the current document. This edge directly encodes the fact that a document has a specific problem. The weight gives strength, making the frequent ghost references have a greater impact on the network structure. The document-content edge compares the TF-IDF value mentioned above with a preset threshold. If the TF-IDF value of the content word is greater than the preset threshold, an undirected edge is established between the corresponding document node and the content node. The edge weight is the TF-IDF value. This edge ensures that documents with similar topics will share the connected content word nodes, thus getting closer to each other in the graph. The ghost-content edge calls the word vector of the ghost reference and encodes the word vector of the content word. It calculates the cosine similarity of the corresponding vectors of each pair of ghost nodes and content nodes. If the similarity exceeds a high threshold, such as greater than or equal to 0.7, the content node and the ghost node are considered related and an undirected edge is established. The edge weight is set to cosine similarity. This edge allows documents that share similar ghost targets to be brought closer at the topic level through the content nodes connected by the ghost targets.

[0091] Create heterogeneous graphs using document nodes, ghost nodes, content nodes, and multiple edge types.

[0092] Step 3-2: For each type of edge in the document heterogeneous graph, find the neighboring nodes of the document node and calculate the attention weight between the document node and its neighboring nodes. After weighting, obtain the aggregated text node vector.

[0093] For each type of multi-element edge in the document heterogeneous graph, step S-3-2 is performed separately. Here, we will use the document-ghost edge as an example.

[0094] For a given document node, find the set of all its neighbor nodes connected by document-ghost edges. Transform the document node and each of its neighbors using a linear transformation with a weight matrix related to the edge type, thus mapping them to a unified semantic space. Obtain the projected vectors h1 and h2. Based on h1 and h2, calculate the importance coefficients between the document node v and each of its neighbor nodes u using an attention mechanism. This can be done using a single-layer feedforward neural network, whose input is the concatenated or summed projected vectors, for example... ,in Represented as importance coefficient, This is represented as an attention vector belonging to the document-ghost edge type. Represented as a linear rectified function, it outputs the input as is when the input is positive, and outputs a small slope (e.g., 0.01) multiplied by the input value when the input is negative. Then, it applies this slope to all such neighbors of v. Softmax normalization is performed to obtain the attention weights.

[0095] The calculated attention weights are used to perform a weighted summation of the projection vectors of neighboring nodes, which is then concatenated or added to the projection vector of the document node itself. A new representation of the node in the "document-ghost edge" edge type is obtained through a non-linear activation function.

[0096] After the above operations, each node obtains a new representation from different edge types. For each edge type of the node's new representation, the importance score for each edge type is calculated using an attention vector and a nonlinear transformation, as expressed by the formula: ,in, This is represented by the importance score for belonging to the i-th edge type j. and All parameters are shared. Represented as an attention vector, Let be a new representation of edge type j belonging to the i-th edge type. tanh represents the hyperbolic tangent activation function, which can perform a nonlinear mapping on the result after linear transformation, compressing the output value to the interval (-1, 1). Softmax normalization is applied to the importance scores of all edge types to obtain the fusion weight for each type, which can represent the contribution of different edge types to the node. The newly obtained fusion weights are multiplied and summed using the new representations of each edge type to obtain the aggregated text node vector, which possesses structural association and semantic information.

[0097] Step 3-3: Construct a similarity matrix based on the aggregated text node vectors, and apply a community detection algorithm to the similarity matrix to obtain the detection results.

[0098] The Louvain algorithm (community detection algorithm) is applied to directly cluster documents into A communities based on the similarity matrix formed by the aggregated text node vectors. The community segmentation results obtained by the above algorithm are used as the initial real community labels to initialize and train the discriminator. The community segmentation results are integrated with the A communities to form the detection result.

[0099] Steps 3-4 involve constructing and training an adversarial network containing a generator and a discriminator by discovering the results and aggregating text node vectors. Community labels and community centrality are then generated based on the adversarial network and output as community results.

[0100] Define a generator, which is a neural network. The input is an aggregated text node vector, and the output is an A-dimensional vector. After passing through a softmax layer, it represents the probability distribution of the document belonging to various communities.

[0101] Define a discriminator, which is also a neural network. Its input is the probability distribution output by the generator, and its output is a scalar representing the probability that the input comes from the true distribution.

[0102] Both the generator and the discriminator include loss functions. The discriminator includes a discriminator loss, which is trained to maximize its ability to distinguish between real and fake data. The discriminator loss is expressed as:

[0103] ;

[0104] in, Let E be the discriminator loss function. The training objective of discriminator D is to minimize this loss. E is the expectation operator, representing the average of the values ​​within the parentheses. The results of the community division This is represented by the probability distribution of real community tags. The output of the discriminator D is z, which is the aggregated text node vector. Let G(z) be the probability distribution of all aggregated text node vectors, and D(G(z)) be the discriminator's judgment result on the community tag G(z) forged by the generator.

[0105] The generator includes adversarial loss and community cohesion loss. The adversarial loss minimizes the discriminator's ability to identify fake labels, while the community cohesion loss optimizes the cohesion of the community.

[0106] The adversarial loss is represented as:

[0107] ,in To counteract the loss function.

[0108] Community cohesion loss is represented as:

[0109] ;

[0110] in, Let the community cohesion loss function be... For different documents, It is represented as the mean vector of community tags assigned to document d by generator G.

[0111] The generator's total loss is the weighted sum of the two: , These are hyperparameters, and then the process of generating a fixed set of parameters is repeated to update the discriminator's parameters in order to minimize them. The parameters of the generator are updated by fixing the discriminator to minimize The alternating operations continue until convergence.

[0112] Understandably, adversarial loss alone may only produce communities that conform to the topology but are loosely connected. Adding community cohesion loss as a regularization term to the generator forces documents in the same community to be similar to each other in the feature space. This ensures that the communities discovered are not only modules in the graph structure, but also groups that are cohesive in semantics and features, thus obtaining higher quality community partitioning.

[0113] After adversarial training is completed, for each document, the aggregated text node vector is input into the generator G to obtain the community probability distribution. The dimension with the largest probability value is taken as the final community label of the document. Then, the average cosine similarity between its aggregated text node vector and the aggregated text node vectors of all other documents in the community is calculated and used as the community centrality. The community label and community center are output as the community result.

[0114] Step 4: Construct a document dynamic graph based on community results and time series data. Obtain hidden states by processing the time and spatial dimensions of the document dynamic graph, and obtain maturity results by combining it with a maturity classifier.

[0115] In this embodiment, step 4 further includes:

[0116] Step 4-1: Obtain the time series data of the original document. For each time step in the time series data, construct an undirected graph by combining the original document and the community results to obtain the document dynamic graph.

[0117] Obtain the version number of the original document and sort it in order. Divide the sorted original document into T time steps, which are time series data. For each time step, construct an undirected graph independently. The node set of the time step consists of all time series data existing in the time step. Traverse all document pairs in the node set. If document 1 and document 2 are assigned to the same community in step 4, establish an undirected edge between them. The edge weight can be uniformly set to 1 to represent a simple connection relationship. Construct an undirected graph in this way, and combine the undirected graphs of each time step into a document dynamic graph.

[0118] Step 4-2: After assigning initial feature vectors to the document dynamic graph, a graph convolutional network is used to perform spatial community aggregation on the document dynamic graph, and a gated recurrent unit is used to perform temporal evolution on the document dynamic graph to output the hidden state at each time step.

[0119] For all nodes existing in the first time step, the aggregated text node vector obtained in step 3 is directly used as its initial feature vector, and graph convolution operation is performed on the document dynamic graph based on the initial feature vector. For the l-th layer of graph convolution at time step t, each node aggregates information from its first-order neighbors. The aggregation method adopts the basic operation of graph convolution network. After several layers of graph convolution, the spatial aggregation feature of time step t is obtained. The representation of each node in the spatial aggregation feature incorporates the messages of other documents in its current community.

[0120] Then, each node is treated as an independent time series. For the document at time step t, the spatial aggregation features and the hidden state of the previous step are input into the gated recurrent unit. The gated recurrent unit has update gate and reset gate mechanisms to determine how much historical information to retain and how much new information to absorb, thereby updating the spatial aggregation features, outputting the hidden state, and passing it to the next time step t+1 as the initial feature vector.

[0121] Starting from the first time step t, the concatenation operation of graph convolutional network → gated recurrent unit is performed sequentially for each time step. After all time steps are processed, the hidden state of each time step is output.

[0122] Step 4-3: Build and train a maturity classifier. Generate a maturity probability distribution and maturity label based on the hidden state and the maturity classifier, and output them as the maturity result.

[0123] Construct a fully connected neural network layer as a classifier. The input of the classifier is the hidden state, and the number of neurons in the output layer is equal to the number of maturity categories, such as 4 categories: draft, pre-release, official, and obsolete. After inputting the hidden state into the classifier, obtain the probability distribution of each maturity category through linear transformation and the Softmax activation function.

[0124] During classifier training, positive and negative sample pairs are extracted. A positive sample pair consists of the hidden states of the same document at two adjacent time steps. and Negative sample pairs are the hidden states of the same document at time step and the hidden states of other different documents in the same batch at the same time step. ( )constitute.

[0125] Using the InfoNCE loss function, for a document's representation at time step 1, its contrastive loss encourages that its similarity with positive samples is much higher than its similarity with all negative samples. The InfoNCE loss function is expressed as:

[0126] ;

[0127] in, Represented as positive sample pairs, It is represented as a temperature hyperparameter, used to adjust the degree of attention given to difficult negative samples, and sim is represented as a similarity function, such as cosine similarity.

[0128] Using the standard multi-class cross-entropy loss function Calculate the classification loss, which measures the difference between the probability distribution predicted by the classifier and the document's true maturity label (known during training). Then, linearly combine the classification loss and the contrastive loss to form the overall objective function for training: , is a hyperparameter used to balance the importance of the primary classification task and the auxiliary contrastive learning task.

[0129] The backpropagation algorithm is employed, and the gradient descent optimizer simultaneously optimizes all parameters in the time-map neural network, maturity classifier, and contrastive learning module. The optimization objective is to minimize the total loss. After training, the hidden state is input into the maturity classifier to obtain the maturity probability distribution and maturity label. The maturity label is the category with the highest probability value. The maturity probability distribution and maturity label are output as the maturity result, describing the probability of a predefined maturity level.

[0130] Step 5: Output the reasoning results, community results, and maturity results as parsing results.

[0131] Once the document has been parsed in steps 1 to 4, the results can be output.

[0132] Based on the above, the inference results include the mean and confidence interval labels. The mean quantifies the uncertainty of information caused by the existence of ghost references that cannot be linked within the document. The higher the value, the more severe the internal linking of the document is broken, and the lower the self-consistency and reliability of the content. After the specific value is output, users can evaluate the integrity of the document structure based on this value.

[0133] Community results include community tags and community centrality. Community centrality quantifies the representativeness or typicality of a single document within its community. A higher community centrality indicates that the document's characteristics are more consistent with the overall characteristics of its community. Conversely, a lower community centrality may indicate that the document deviates from the mainstream direction of the community or that the subject is not focused enough. Community tags indicate which community (domain) the document has been classified into.

[0134] The maturity results include a maturity probability distribution and a maturity label. For example, if a document outputs [Draft: 0.1, Pre-release: 0.15, Formal: 0.7, Outdated: 0.05] after step S4, then according to the maturity probability distribution, the document has a 70% probability of being evaluated as a formal document and having a formal maturity label.

[0135] It should also be noted that AI-based document parsing may result in occasional misjudgments. Therefore, the confidence interval label of the mean and the maturity probability distribution are retained in the final output so that users can comprehensively consider the final parsing result based on the actual situation.

[0136] Figure 2 The diagram illustrates an AI-based automatic multi-document parsing system that can realize the ideas of this application, according to some embodiments of this application. The following is a detailed description of this AI-based automatic multi-document parsing system.

[0137] Specifically, an AI-based multi-document automatic parsing system includes:

[0138] The AI-based automatic multi-document parsing method includes a processor and a memory, the memory and the processor being connected, the memory being used to store programs, instructions or code, and the processor being used to execute the programs, instructions or code in the memory to implement the AI-based automatic multi-document parsing method according to any one of claims 1-8.

[0139] Figure 3 The diagram illustrates an AI-based multi-document automatic parsing platform that can realize the ideas of this application, according to some embodiments of this application. The following is a detailed description of this AI-based multi-document automatic parsing platform.

[0140] Specifically, an AI-based multi-document automatic parsing platform includes:

[0141] The initial extraction sub-platform is used to extract a candidate set of ghost references from the original document and combine it with a conditional random field to perform cyclical serialization joint inference to obtain the ghost reference set;

[0142] The first parsing sub-platform is used to combine ghost reference sets, original documents and Bayesian networks to perform inference parsing in order to obtain inference results;

[0143] The second parsing sub-platform is used to perform community parsing by combining ghost reference sets, original documents, and community adversarial strategies to obtain community results.

[0144] The third analysis sub-platform is used to combine community results and time-series data to perform maturity analysis in order to obtain maturity results.

[0145] The results output platform is used to output reasoning results, community results, and maturity results as analytical results.

[0146] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. An AI-based automatic multi-document parsing method, characterized in that, include: The ghost reference candidate set is obtained by preprocessing the original document, a six-dimensional feature vector is extracted based on the ghost reference candidate set, and a cyclical serialization joint inference is performed in combination with a conditional random field to obtain the ghost reference set. A four-dimensional feature vector is obtained based on the ghost reference set and the original document. A Bayesian network is constructed by combining the four-dimensional feature vector. The inference result is obtained after inference through the Bayesian network. A document heterogeneous graph with multiple types of nodes and edges is created based on the ghost reference set and the original document. Attention aggregation is performed based on the multiple types of edges to generate aggregated text node vectors. Community adversarial strategies are combined to obtain community results. A document dynamic graph is constructed based on community results and time-series data. Hidden states are obtained after processing the time and spatial dimensions of the document dynamic graph, and maturity results are obtained by combining it with a maturity classifier. The inference results, community results, and maturity results are output as parsing results.

2. The AI-based automatic multi-document parsing method according to claim 1, characterized in that, A ghost reference candidate set is obtained by preprocessing the original document. A six-dimensional feature vector is extracted from the ghost reference candidate set, and a cyclical serialization joint inference is performed in conjunction with a conditional random field to obtain the ghost reference set, including: The original document undergoes preprocessing including document extraction, anchor point identification, and directional extraction to generate an anchor point candidate library and a directional expression set, which are then matched to construct a ghost reference candidate set. A six-dimensional feature vector, including matching features, text features, and reference features, is extracted by constructing a ghost reference candidate set and matching process, combined with an anchor point candidate library. Serialization joint inference is performed based on six-dimensional feature vectors and combined with conditional random fields, and the inference results are output. The conditional random fields include at least a graphical model based on linear chain conditional random fields, node potential functions and edge potential functions. Uncertainty assessment is performed on the inference results to obtain re-inference data. The conditional random field performs cyclic inference based on the re-inference data until the termination condition is met, at which point a ghost reference set is output.

3. The AI-based automatic multi-document parsing method according to claim 3, characterized in that, The original document undergoes preprocessing including at least document extraction, anchor point identification, and directional extraction to generate an anchor point candidate library and a directional expression set, which are then matched to construct a ghost reference candidate set, including: After determining the format of the original document, content is extracted based on the format to obtain text structure data; Based on text structure data, standard text fragments are obtained, and after semantic encoding of the standard text fragments, an index is built to generate an anchor point candidate library. After performing regular expression localization on the text structure data, dependency parsing is performed on the localization results based on natural language processing, and a set of directional expressions is generated through citation classification. A candidate set of directional statements is constructed by progressively matching the anchor point candidate library. The progressive matching includes first-level, second-level and third-level matching, with each level of matching being in a progressive relationship.

4. The AI-based automatic multi-document parsing method according to claim 1, characterized in that, A four-dimensional feature vector is obtained based on the ghost reference set and the original document. A Bayesian network is constructed using the four-dimensional feature vector. After inference through the Bayesian network, the inference result is obtained, including: The citation density feature, type distribution feature, semantic dispersion feature, and temporal concentration feature are calculated based on the ghost citation set and the original document, and a four-dimensional feature vector is constructed. The network structure of parent nodes, child nodes and corresponding dependencies is defined by four-dimensional feature vectors, and a conditional probability distribution model based on Gaussian mixture model is performed to generate a Bayesian network. Wherein, the parent node is a four-dimensional feature vector, the child node is a reference entropy, and the corresponding dependency relationship is that the parent node points to the child node; The original document is subjected to probabilistic inference using a Bayesian network, and the mean and confidence interval labels are calculated. The mean and confidence interval labels are then used as the inference results output.

5. The AI-based automatic multi-document parsing method according to claim 1, characterized in that, A heterogeneous document graph with multiple node and edge types is created based on the ghost reference set and the original document. Attention aggregation is performed based on the multiple edge types to generate aggregated text node vectors. Community adversarial strategies are then combined to obtain community results, including: Document nodes, ghost nodes, and content nodes are created based on the ghost reference set and the original document. At the same time, multiple types of edges are created to obtain a document heterogeneous graph, where the multiple types of edges are represented as an exhaustive combination of pairwise node types. For each type of edge in the document heterogeneous graph, find the neighboring nodes of the document node and calculate the attention weight between the document node and the neighboring nodes. After weighting, obtain the aggregated text node vector. A similarity matrix is ​​constructed based on aggregated text node vectors, and a community detection algorithm is applied to the similarity matrix to obtain the detection results; By discovering the results and aggregating text node vectors, an adversarial network containing a generator and a discriminator is constructed and trained. Based on the adversarial network, community labels and community centrality are generated and output as community results.

6. The AI-based automatic multi-document parsing method according to claim 5, characterized in that, The method also include: The generator includes adversarial loss and community cohesion loss, and the discriminator includes discriminator loss, and is trained using an alternating training mechanism.

7. The AI-based automatic multi-document parsing method according to claim 1, characterized in that, A document dynamic graph is constructed based on community results and time-series data. Hidden states are obtained after processing the time and spatial dimensions of the document dynamic graph. Maturity results are then obtained by combining this with a maturity classifier, including: Obtain the time series data of the original document. For each time step in the time series data, construct an undirected graph by combining the original document and community results to obtain the document dynamic graph. After assigning initial feature vectors to the document dynamic graph, a graph convolutional network is used to perform spatial community aggregation on the document dynamic graph, and a gated recurrent unit is used to perform temporal evolution on the document dynamic graph to output the hidden state at each time step. Construct and train a maturity classifier, and generate a maturity probability distribution and maturity label based on the hidden state and the maturity classifier, and output them as the maturity result.

8. The AI-based automatic multi-document parsing method according to claim 7, characterized in that, The method also include: The maturity classifier employs a fully connected neural network layer, including a contrastive loss based on the InfoNCE loss function and a classification loss based on the multi-class cross-entropy loss function.

9. An AI-based multi-document automatic parsing system, characterized in that, include: The AI-based multi-document automatic parsing system includes a processor and a memory, the memory and the processor being connected. The memory is used to store programs, instructions or code, and the processor is used to execute the programs, instructions or code in the memory to implement the AI-based multi-document automatic parsing method according to any one of claims 1-8.

10. An AI-based multi-document automatic parsing platform for implementing the method described in any one of claims 1-8, characterized in that, include: The initial extraction sub-platform is used to extract a candidate set of ghost references from the original document and combine it with a conditional random field to perform cyclical serialization joint inference to obtain the ghost reference set; The first parsing sub-platform is used to combine ghost reference sets, original documents and Bayesian networks to perform inference parsing in order to obtain inference results; The second parsing sub-platform is used to combine ghost reference sets, original documents, and community adversarial strategies to perform community parsing in order to obtain community results. The third analysis sub-platform is used to combine community results and time-series data to perform maturity analysis in order to obtain maturity results. The results output platform is used to output reasoning results, community results, and maturity results as analytical results.

Images