Heterogeneous document intelligent extraction method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computers, and discloses an intelligent heterogeneous document extraction method and system, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed heterogeneous structured document for analysis processing, and extracting path expressions of all data nodes in the document; inputting the path expressions into a pre-trained field classification model in batches; performing feature analysis and classification calculation on each path expression based on a field classification model, and outputting a corresponding target structured field prediction result; establishing a mapping relation from the path expressions to the target field according to the prediction result, and extracting node content data corresponding to the path expressions from the original structured document based on the mapping relation; and filling the node content data into the corresponding target structured fields according to the mapping relationship to generate a structured data record in a standard format. The method can automatically adapt to different document structures, and dependence on manual rules is reduced.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a method, system, electronic device, and storage medium for intelligent extraction of heterogeneous documents. Background Technology

[0002] With the rapid development of information technology, Extensible Markup Language (XML) has become a common format for data exchange and storage due to its self-descriptive and platform-independent nature. In practical applications, different data sources often employ their own custom structural specifications, leading to a large number of heterogeneous documents. These documents may use completely different path expressions for semantically identical fields, posing a significant challenge to the unified processing and extraction of data.

[0003] Currently, the industry generally uses a parsing method based on predefined templates to process such heterogeneous documents. This method requires writing specific parsing rules for each known structure and explicitly specifying the path expression corresponding to each target field. Although this method can guarantee accuracy in scenarios with fixed structures, it has obvious limitations: First, each time a new data source is connected, a large amount of human resources is required for structure analysis, rule writing, and testing and verification, resulting in long data access cycles and high costs; second, when any change occurs in the data source structure, the original parsing template will become invalid, requiring manual intervention for maintenance, resulting in poor system robustness; in addition, as the number of data sources increases, the number of templates and management complexity increase exponentially, severely restricting the scalability of the system.

[0004] Therefore, this application provides a method for intelligent extraction of heterogeneous documents to solve the above-mentioned technical problems. Summary of the Invention

[0005] The purpose of this invention is to provide a method, system, electronic device and storage medium for intelligent extraction of heterogeneous documents, so as to solve the efficiency bottleneck and maintenance problems faced by the prior art when processing large-scale heterogeneous data.

[0006] To address the aforementioned technical problems, this invention provides a method for intelligent extraction of heterogeneous documents, comprising:

[0007] The heterogeneous structured document to be processed is parsed and the path expressions of all data nodes in the document are extracted. The path expressions are represented in a standard syntax format and include tag names, attribute information and hierarchical relationships.

[0008] The path expressions are input into a pre-trained field classification model in batches. The field classification model is trained based on the correspondence between path expression samples and target structured field labels, learns the feature patterns of path expressions, and maps path expressions to a predefined set of structured fields.

[0009] Based on the field classification model, feature analysis and classification calculation are performed on each path expression, and the corresponding target structured field prediction results are output. The prediction results include the determined field labels and the corresponding confidence scores.

[0010] Based on the prediction results, establish a mapping relationship between path expressions and target fields, and extract the node content data corresponding to each path expression from the original structured document based on the mapping relationship;

[0011] The node content data is filled into the corresponding target structured fields according to the mapping relationship to generate a standard format structured data record. Among them, the prediction results with confidence scores higher than the preset threshold are filled with content, while the prediction results with confidence scores lower than the preset threshold are marked and reserved for manual review.

[0012] In some specific embodiments, the heterogeneous structured document to be processed is obtained and parsed to extract the path expressions of all data nodes in the document. The path expressions are represented using a standard syntax format and include tag names, attribute information, and hierarchical relationships. Further, they include:

[0013] Receive heterogeneous structured document input to be processed, and verify the integrity and validity of the document format;

[0014] A document parser is used to parse and process the document, and a document object model organized in a tree structure is constructed.

[0015] Access each data node in the document object model through a traversal algorithm, and record the node's type, attributes, and location information;

[0016] Standardized path expressions are generated based on the label names, attribute values, and hierarchical relationships of data nodes, and the association between the path expressions and the node content is stored.

[0017] In some specific embodiments, path expressions are input into a pre-trained field classification model in batches. The field classification model is trained based on the correspondence between path expression samples and target structured field labels, learning the feature patterns of the path expressions and mapping them to a predefined set of structured fields. This further includes:

[0018] The extracted path expressions are preprocessed, including format standardization, length adjustment, and outlier handling.

[0019] The preprocessed path expressions are grouped according to a preset batch size to form batch input data;

[0020] Feature extraction is performed on each batch of path expressions using a field classification model. The extracted features include structural features and semantic features.

[0021] The extracted features are input into the model's computation layer to obtain preliminary mapping results from path expressions to field labels.

[0022] In some specific embodiments, feature analysis and classification calculation are performed on each path expression based on a field classification model, and the corresponding target structured field prediction result is output. The prediction result includes a determined field label and a corresponding confidence score, and further includes:

[0023] The field classification model calculates the probability distribution of each path expression belonging to each target field and generates probability values ​​through the output layer activation function;

[0024] Select the field label corresponding to the highest probability value from the probability distribution as the main prediction result;

[0025] The confidence score of the prediction result is calculated based on the degree of dispersion of the probability distribution or the probability value output by the model.

[0026] The output includes complete prediction results with field labels and confidence scores, and is classified according to confidence intervals.

[0027] In some specific embodiments, a mapping relationship between path expressions and target fields is established based on the prediction results. Based on this mapping relationship, node content data corresponding to each path expression is extracted from the original structured document. This further includes:

[0028] Based on the field labels in the prediction results, construct a mapping table between path expressions and target fields;

[0029] Based on the mapping relationship, locate the data node corresponding to each path expression in the original structured document;

[0030] Extract the content information of the located data nodes, including text data, attribute values, and child node content;

[0031] The extracted information is cleaned and formatted, including character encoding standardization and filtering of invalid data.

[0032] In some specific embodiments, node content data is filled into corresponding target structured fields according to the mapping relationship to generate structured data records in a standard format. Specifically, prediction results with confidence scores higher than a preset threshold are populated with content, while prediction results with confidence scores lower than a preset threshold are marked and have a manual review channel reserved. Further, this includes:

[0033] The extracted node content data is allocated to the corresponding target field storage area according to the mapping relationship;

[0034] For predictions with confidence scores higher than a preset threshold, content filling and target field updates are automatically performed.

[0035] For predictions with a confidence score below a preset threshold, the path expression, prediction results, and extracted content are packaged and marked as samples to be reviewed.

[0036] Integrate all the populated field contents, assemble them into a complete structured data record, and output the results in a standard format.

[0037] In some specific embodiments, training the field classification model includes:

[0038] Collect historical structured document data, extract the correspondence between path expressions and field labels to construct a training dataset;

[0039] Feature engineering is performed on path expression samples in the training dataset to extract structural features and sequence feature vectors;

[0040] A machine learning algorithm is used to train a field classification model with feature vectors as input.

[0041] The model's accuracy, recall, and F1 score are evaluated using a validation set, and the model with the best overall performance is selected as the final field classification model.

[0042] Based on the same concept, the present invention also provides a heterogeneous document intelligent extraction system, comprising:

[0043] The parsing and processing module is configured to acquire heterogeneous structured documents to be processed, parse and process them, and extract the path expressions of all data nodes in the document. The path expressions are represented in a standard syntax format and include tag names, attribute information and hierarchical relationships.

[0044] The path expression input module is configured to input path expressions in batches into a pre-trained field classification model. The field classification model is trained based on the correspondence between path expression samples and target structured field labels, learns the feature patterns of path expressions, and maps path expressions to a predefined set of structured fields.

[0045] The prediction result output module is configured to perform feature analysis and classification calculation on each path expression based on the field classification model, and output the corresponding target structured field prediction results. The prediction results include the determined field labels and the corresponding confidence scores.

[0046] The mapping relationship establishment module is configured to establish a mapping relationship from path expressions to target fields based on the prediction results, and extract the node content data corresponding to each path expression from the original structured document based on the mapping relationship;

[0047] The confidence assessment and output selection module is configured to fill the node content data into the corresponding target structured fields according to the mapping relationship, and generate structured data records in a standard format. Among them, the prediction results with confidence scores higher than the preset threshold are filled with content, while the prediction results with confidence scores lower than the preset threshold are marked and reserved for manual review.

[0048] Based on the same concept, the present invention also provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a heterogeneous document intelligent extraction method.

[0049] Based on the same concept, the present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of a heterogeneous document intelligent extraction method.

[0050] Compared with existing technologies, its advantages are as follows:

[0051] This invention discloses a method, system, electronic device, and storage medium for intelligent extraction of heterogeneous documents. By transforming the structured extraction problem of heterogeneous documents into a classification problem in machine learning, it provides a new technical path for achieving automated document parsing.

[0052] Improved efficiency: When processing new data sources of the same type, there is no need for manual analysis of document structure and writing of parsing rules, shortening the data access cycle from several days or weeks to minutes or hours; when the document structure changes, as long as its path features are still within the model's recognition range, the system can maintain stable operation, reducing maintenance costs.

[0053] Accuracy and practicality are equally important: Machine learning models are used to learn the inherent patterns of path expressions from massive amounts of historical data, achieving a practical level of accuracy in identifying target fields; a confidence assessment mechanism is introduced to provide a manual review channel for predictions with low confidence, ensuring the reliability of data quality.

[0054] It boasts outstanding versatility and scalability: it does not depend on any specific document structure specification and can theoretically handle documents of any structure; the model supports incremental learning, can continuously incorporate new data features, and has self-optimization capabilities. Attached Figure Description

[0055] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0056] Figure 1 This is a flowchart illustrating some specific embodiments of the intelligent extraction method for heterogeneous documents according to the present invention;

[0057] Figure 2 This is one of the flowcharts of another embodiment of the intelligent extraction method for heterogeneous documents of the present invention;

[0058] Figure 3 This is a second flowchart illustrating another embodiment of the intelligent extraction method for heterogeneous documents according to the present invention;

[0059] Figure 4 This is a schematic diagram of the structure of a heterogeneous document intelligent extraction system according to some specific embodiments of the present invention;

[0060] Figure 5 This is a schematic diagram of the structure of an electronic device according to some specific embodiments of the present invention;

[0061] In the diagram, 710 is the processor; 720 is the memory; 730 is the input device; and 740 is the output device. Detailed Implementation

[0062] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0063] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “said,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0064] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0065] It should be understood that although the terms first, second, third, etc., may be used in the embodiments of this application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, first may also be referred to as second without departing from the scope of the embodiments of this application, and similarly, second may also be referred to as first.

[0066] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0067] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0068] It should be noted that any symbols and / or numbers present in the specification that are not marked in the accompanying drawings are not reference numerals.

[0069] Reference Figure 1 A method for intelligent extraction of heterogeneous documents, comprising:

[0070] S101, Obtain the heterogeneous structured document to be processed and parse it to extract the path expressions of all data nodes in the document. The path expressions are represented in a standard syntax format and include tag names, attribute information and hierarchical relationships.

[0071] S102, the path expressions are input into the pre-trained field classification model in batches. The field classification model is trained based on the correspondence between path expression samples and target structured field labels, learns the feature patterns of path expressions, and maps path expressions to a predefined set of structured fields.

[0072] S103, based on the field classification model, perform feature analysis and classification calculation on each path expression, and output the corresponding target structured field prediction results, wherein the prediction results include the determined field labels and the corresponding confidence scores;

[0073] S104, Establish a mapping relationship between path expressions and target fields based on the prediction results, and extract the node content data corresponding to each path expression from the original structured document based on the mapping relationship;

[0074] S105, fill the node content data into the corresponding target structured fields according to the mapping relationship to generate a standard format structured data record. Among them, the prediction results with confidence scores higher than the preset threshold are filled with content, and the prediction results with confidence scores lower than the preset threshold are marked and reserved for manual review.

[0075] Specifically, in this embodiment of the invention, a heterogeneous structured document to be processed is acquired and parsed. Complete path expressions for all data nodes in the document are extracted by traversing document nodes. These path expressions are represented using a standard syntax format and include tag names, attribute information, and hierarchical relationships. The extracted path expressions are input in batches into a pre-trained field classification model. This model is trained using a machine learning algorithm based on the correspondence between path expression samples and target structured field labels. It can automatically learn the feature patterns of path expressions and map them to a predefined set of structured fields. The field classification model performs feature analysis and classification calculations on each path expression, outputting the corresponding target structured field prediction result. The prediction result includes the determined field label and the corresponding confidence score. A complete mapping relationship from path expressions to target fields is established based on the prediction results. Based on this mapping relationship, node content data corresponding to each path expression is extracted from the original structured document. The extracted node content data is filled into the corresponding target structured fields according to the mapping relationship, generating a standard format structured data record. Prediction results with confidence scores higher than a preset threshold are directly filled with content, while prediction results with confidence scores lower than a preset threshold are marked and have a manual review channel reserved.

[0076] For example, a heterogeneous XML document from an academic publishing institution contains a data node with the path expression " / article / front / article-meta / title-group / article-title". This path expression is extracted and input into a field classification model. The model parses the data based on pre-trained feature patterns, where features include field names appearing in the path, such as "title", and a path depth of 4 levels. The field classification model uses a gradient boosting decision tree algorithm, calculating the feature vector of the path expression and comparing it with the feature weights in the training data to determine the probability that the path expression belongs to the "article title" field. The value is 95%, and the confidence score calculated by the model based on the dispersion of the probability distribution is 92%. The preset confidence threshold is 90%, so the prediction result is higher than the threshold. The node content "Research on XML Parsing Based on Machine Learning" corresponding to the path is directly extracted from the original document according to the mapping relationship and filled into the target structured field "Article Title". Conversely, if the prediction field of another path expression " / pub / article / body / sec" is "Abstract" but the confidence score is 85%, which is lower than the preset threshold, the result is marked as a sample to be reviewed and the manual review channel is reserved. All the filled field contents are assembled into a structured data record for output.

[0077] In some applications, the process involves acquiring heterogeneous structured documents to be processed, parsing them, and extracting path expressions for all data nodes in the document. These path expressions are represented using a standard syntax format and include tag names, attribute information, and hierarchical relationships. The process includes receiving the heterogeneous structured document input, verifying the integrity and validity of the document format, using a document parser to parse the document, constructing a document object model organized in a tree structure, accessing each data node in the document object model through a traversal algorithm, recording the node's type, attributes, and location information, generating standardized path expressions based on the data node's tag names, attribute values, and hierarchical relationships, and storing the association between the path expressions and the node content.

[0078] Understandably, the process involves receiving heterogeneous structured document inputs and verifying the integrity and validity of the document format to ensure compliance with processing specifications. A document parser is then used to parse the documents, constructing a document object model organized in a tree structure. This model accurately reflects the hierarchical structure and node relationships of the document. A traversal algorithm systematically accesses each data node in the document object model, fully recording the node's type characteristics, attribute information, and position information within the document tree. Based on the data node's label name, attribute value, and hierarchical relationship, path expressions conforming to standard syntax are generated, and these generated path expressions are associated with the corresponding node content and stored, forming a complete set of path expressions.

[0079] For example, when processing a 2MB XML document of journal articles, the process first verifies the validity of the document format, confirming that the document structure is complete and conforms to the XML 1.0 specification. A DOM parser is then used to parse the document, constructing a document object model containing 156 nodes, where the root node has a depth of 0 and the leaf nodes have a maximum depth of 7 levels. Each data node is accessed using a depth-first traversal algorithm, recording node types including element nodes, text nodes, and attribute nodes, with 89 element nodes, 52 text nodes, and 15 attribute nodes. Standard path expressions are generated based on the node information; for example, the path expression " / article / front / article-meta / title-group" is generated for nodes at the third level, while the corresponding node content "Machine Learning Research Topics" is recorded. Eighty-three path expressions are generated and associated with their corresponding node content, stored in a temporary database for subsequent processing.

[0080] In some applications, path expressions are input into a pre-trained field classification model in batches. The field classification model is trained based on the correspondence between path expression samples and target structured field labels, learning the feature patterns of path expressions and mapping them to a predefined set of structured fields. This includes preprocessing the extracted path expressions, such as format standardization, length adjustment, and outlier handling; grouping the preprocessed path expressions according to a preset batch size to form batch input data; extracting features from each batch of path expressions using the field classification model, including structural and semantic features; and inputting the extracted features into the model's computation layer to obtain preliminary mapping results from path expressions to field labels.

[0081] Understandably, the field classification model is trained based on the correspondence between path expression samples and target structured field labels. It learns the feature patterns of path expressions to map them to a predefined set of structured fields. This process includes preprocessing the extracted path expressions, standardizing the format to ensure grammatical consistency, adjusting the length to meet model input requirements, and handling outliers to eliminate non-compliant data. The preprocessed path expressions are then grouped according to a preset batch size to form batch input data to improve processing efficiency. The field classification model extracts features from each batch of path expressions, including structural features reflecting path hierarchy and semantic features embodying semantic information. The extracted features are then input into the model's computation layer, where the model's internal computational mechanism obtains the initial mapping results from path expressions to field labels.

[0082] For example, the 83 extracted path expressions are preprocessed. The path expression " / article / front / article-meta / title-group / article-title" is standardized to lowercase; paths longer than 256 characters are truncated, retaining only the first 256 characters; three abnormal paths containing special characters are identified and removed; the remaining 80 paths are grouped into three batches of input data, each consisting of 32 paths. A field classification model extracts features from the first batch of 32 paths, extracting structural features including a path depth of 4 levels and the number of nodes of 5. Semantic features are converted into 768-dimensional vectors using a word embedding model. The model's computation layer performs weighted calculations on the features to obtain preliminary mapping results from paths to fields. The confidence level of the path " / article / front / article-meta / title-group / article-title" mapping to the "title" field is 92%.

[0083] In some applications, feature analysis and classification calculations are performed on each path expression based on a field classification model, outputting the corresponding target structured field prediction results. The prediction results include determined field labels and corresponding confidence scores. This process involves the field classification model calculating the probability distribution of each path expression belonging to each target field, generating probability values ​​through the output layer activation function, selecting the field label corresponding to the highest probability value from the probability distribution as the main prediction result, calculating the confidence score of the prediction result based on the dispersion of the probability distribution or the probability value output by the model, and outputting a complete prediction result containing field labels and confidence scores, and classifying the data according to the confidence interval.

[0084] Understandably, the process involves calculating the probability distribution of each path expression belonging to a predefined target field using a field classification model, converting the raw numerical values ​​output by the model into standardized probability values ​​using an output layer activation function, selecting the field label corresponding to the highest probability value from the generated probability distribution as the main prediction result for that path expression, calculating the confidence score of the prediction result based on the dispersion feature of the probability distribution or the highest probability value output by the model, outputting a complete prediction result containing the determined field label and corresponding confidence score, and automatically classifying and labeling the prediction result according to a preset confidence interval range.

[0085] For example, for the path expression " / article / body / sec / title", the field classification model calculates its probability distribution across 12 target fields, obtains the probability value of each field through the softmax activation function, where the probability of the "Chapter Title" field is 90%, the probability of the "Abstract" field is 5%, and the probabilities of other fields are all below 3%. The "Chapter Title" corresponding to the highest probability value of 90% is selected as the main prediction result. The confidence score calculated based on the entropy value of the probability distribution is 95%, which is higher than the preset 90% high confidence threshold. This prediction is marked as a high-quality result and automatically adopted. Another path, " / article / back / ref-list / ref / mixed-citation", is predicted to have a probability of 65% for the "References" field, with a confidence score of 75%, which is in the medium confidence range of 70%-85%. This is marked as a sample that needs manual review. All prediction results are classified into three confidence ranges (high, medium, and low) according to their confidence scores and marked accordingly.

[0086] In some applications, a mapping relationship between path expressions and target fields is established based on the prediction results. Based on the mapping relationship, the node content data corresponding to each path expression is extracted from the original structured document. This includes constructing a mapping relationship table between path expressions and target fields based on the field labels in the prediction results; locating the data node corresponding to each path expression in the original structured document based on the mapping relationship; extracting the content information of the located data nodes, including text data, attribute values, and child node content; and performing data cleaning and formatting on the extracted content information, including character encoding unification and invalid data filtering.

[0087] Understandably, the process involves constructing a mapping table between path expressions and target fields based on the field labels in the prediction results, forming a complete field mapping specification. Based on this mapping, the data nodes corresponding to each path expression are precisely located in the original structured document to ensure accurate node positioning. Complete content information of the located data nodes is extracted, including text data, attribute values, and child node content, ensuring comprehensive information extraction. The extracted content information undergoes systematic data cleaning and formatting, including implementing unified character encoding to ensure data consistency, and filtering invalid data to remove blank characters and redundant information, thereby obtaining standardized content data.

[0088] For example, a mapping table containing 120 records is constructed based on the prediction results, where the path expression " / article / front / article-meta / title-group / article-title" maps to the "article title" field with a confidence level of 95%. Based on this mapping relationship, 15 corresponding nodes are located in the original XML document. The extracted node content includes the text data "Research on the Application of Deep Learning in Natural Language Processing", the attribute value "@xml:lang='zh-CN'", and the content of 3 child nodes. The extracted content is then cleaned, the character encoding is uniformly converted to UTF-8 format, and 12 blank nodes and 5 invalid attributes are filtered out to obtain standardized content data.

[0089] In some applications, node content data is filled into corresponding target structured fields according to mapping relationships to generate standard-format structured data records. Specifically, prediction results with confidence scores higher than a preset threshold are populated with content, while prediction results with confidence scores lower than the preset threshold are marked and reserved for manual review. This includes allocating extracted node content data to the corresponding target field storage area according to the mapping relationship; automatically completing content filling and target field updates for predictions with confidence scores higher than the preset threshold; and packaging and marking the path expression, prediction result, and extracted content as samples awaiting review for predictions with confidence scores lower than the preset threshold. All populated field content is then integrated to assemble a complete structured data record and output a standard-format result.

[0090] Understandably, the extracted node content data is accurately allocated to the corresponding target field storage area based on the established mapping relationship; for prediction results with a confidence score higher than the preset threshold, the content filling operation is automatically completed and the target field value is updated; for prediction results with a confidence score lower than the preset threshold, the corresponding path expression, prediction result and extracted content are packaged and marked as samples to be reviewed; all filled field contents are integrated and processed through data assembly to generate complete structured data records and output the final results in a standard format.

[0091] For example, when processing node content data containing 150 mapping relationships, the extracted node content is allocated to 25 target field storage areas based on the mapping relationships. Among them, 140 prediction results with a confidence score higher than the 90% threshold are automatically populated with content, including filling the "Artificial Intelligence Technology Research" text corresponding to the path expression " / article / title" into the "Article Title" field. Another 10 prediction results with a confidence score between 60% and 85% are packaged and marked as samples to be reviewed. Each sample contains a path expression, prediction field label, confidence score, and extracted content. All populated fields are integrated to assemble a complete structured data record, and the output is in standard JSON format, containing 25 field values.

[0092] In some applications, training a field classification model involves collecting historical structured document data, extracting the correspondence between path expressions and field labels to construct a training dataset; performing feature engineering on the path expression samples in the training dataset to extract structural features and sequence feature vectors; using machine learning algorithms with the feature vectors as input to train the field classification model; using a validation set to evaluate the model's accuracy, recall, and F1 score, and selecting the model with the best overall performance as the final field classification model.

[0093] Understandably, the process involves collecting historical structured document data and extracting the correspondence between path expressions and field labels to construct a training dataset; performing feature engineering on the path expression samples in the training dataset to extract structural features reflecting the hierarchical structure of the path and sequence feature vectors embodying semantic information; using machine learning algorithms with the extracted feature vectors as input to train a field classification model, and adjusting the model parameters through optimization algorithms; using an independent validation set to evaluate the accuracy, recall, and F1 score of the trained model, and selecting the optimal model as the final field classification model based on the comprehensive performance evaluation results.

[0094] For example, 5000 structured documents were collected from a historical document library, and 8500 path expressions and their corresponding field labels were extracted to construct a training dataset. Feature engineering was performed on the path expression samples to extract structural features, including a path depth of 3-8 levels and 4-12 nodes. Sequence features were converted into 512-dimensional vectors through word embeddings. The LightGBM algorithm was used to train the classification model with a learning rate of 0.05 and 200 training epochs. The model performance was evaluated on a validation set containing 1500 samples, achieving an accuracy of 94.2%, a recall of 92.8%, and an F1 score of 93.8%. This model was ultimately selected as the field classification model for the production environment.

[0095] The following is combined Figure 2 and Figure 3 Another embodiment of the intelligent extraction method for heterogeneous documents according to the present invention is described below:

[0096] like Figure 2 and Figure 3 As shown, this embodiment includes:

[0097] Training Phase: Building the Annotated Dataset: A high-quality labeled dataset is collected and built from template-processed XML documents. Each sample in this dataset is an (XPath path, target field label) pair. For example, (" / / article / title", "article_title").

[0098] Feature engineering: Targeting the textual characteristics of XPath paths, two types of features are designed and extracted: Structured features: Distinguishing elements are parsed from the path, such as specific field names (title, author, doi, etc.), attribute names (@language, @pub-type, etc.), and language codes (en, zh, etc.). Text sequence features: The complete XPath path is treated as a text sequence. After tokenization, it is input into a pre-trained language model (such as BERT) to capture the context and semantic relationships between tags in the path.

[0099] Model Training and Selection: Using the labeled dataset and extracted features described above, train one or more machine learning classification models. Selected models include gradient boosting decision tree models (such as LightGBM) and Transformer-based pre-trained models (such as BERT). Through comparative evaluation, select the model that performs best overall in terms of accuracy, F1 score, and inference speed as the production model.

[0100] Prediction and Application Phase: XML Parsing and XPath Extraction: Upon receiving a new XML document with an unknown structure, a general parsing is first performed, and the complete XPath paths of all data nodes in the document are traversed and extracted.

[0101] Field prediction: All extracted XPath paths are input into the pre-trained field classification model. The model will predict the most likely target field (such as "article title", "author", "abstract", etc.) for each path and provide the prediction confidence score.

[0102] Structured data assembly: Based on the model's prediction results, the content of the XML node pointed to by each XPath path is extracted and filled into the corresponding target field, thereby automatically assembling a complete and structured data record.

[0103] Engineering integration: The entire process, including XPath extraction and field prediction, is encapsulated into independent, reusable microservices (such as using the FastAPI framework) and deployed via Docker containers. This service seamlessly integrates with existing data processing pipelines through API interfaces.

[0104] The following describes this embodiment in conjunction with an application scenario:

[0105] The context is that CNKI's open access platform needs to process heterogeneous journal XML data from nearly 500 partner institutions.

[0106] Training data preparation: From the XML data and corresponding templates of over 500 institutions that were successfully parsed historically, approximately 4000 high-quality (XPath path-field label) samples were extracted. The field labels cover more than 30 core metadata fields, including article_title, author, abstract, keyword, doi, and journal_name.

[0107] Feature engineering and model training: Two types of models were trained in parallel for comparison and selection.

[0108] Model A (LightGBM): Features: Three types of structured features were extracted for each XPath path: Field name features: whether the path contains keywords such as title, author, and abstract. Attribute name features: whether the path contains attributes such as @language and @pub-type. Language features: whether the path contains language identifiers such as en and zh. Training parameters: The learning rate was set to 0.05, the maximum tree depth was 7, the iteration was 100 rounds, and multi_logloss was used as the loss function.

[0109] Model B (BERT-based): Features: The XPath path (e.g., / article / front / title) is converted into a sequence ["[CLS]", "article", "front", "title", "[SEP]"] using the WordPiece tokenizer, and then input into the bert-base-uncased pre-trained model. The output at the [CLS] position is taken as the representation of the entire path. Training parameters: Batch size 32, learning rate 2e-5, using the AdamW optimizer, fine-tuned for 15 epochs.

[0110] Model evaluation and selection: The model was evaluated on the retained test set, and the results are as follows:

[0111] The LightGBM model slightly outperforms the BERT model in both accuracy and F1 score. This is because XPath paths have relatively fixed and structured semantics, allowing LightGBM to efficiently utilize manually constructed strong features. Furthermore, LightGBM's training and prediction speeds are significantly faster than BERT, better meeting the real-time requirements of industrial scenarios. Therefore, the LightGBM model was chosen as the field classification model for online services.

[0112] Engineering deployment: Two core interfaces were developed using the FastAPI framework:

[0113] POST / xpath / extract: Receives an XML file or content and returns a list of its XPath paths and their text content.

[0114] POST / field / predict: Receives a list of XPath paths and returns the predicted field label and confidence score for each path, for example: {"xpath": " / / articles / body / journal / title", "field": "journal_title", "confidence": 0.98}.

[0115] Package these two services, model files, and their dependencies into a Docker image. In a production environment, deploy using Docker Compose and implement load balancing using Nginx.

[0116] Workflow: A new, unparsed XML dataset from a partner organization is received upstream in the data pipeline. The pipeline calls the ` / xpath / extract` interface to obtain all XPath paths of the XML. The pipeline then calls the ` / field / predict` interface to send all XPath paths to the deployed LightGBM model service for batch prediction. The service returns the prediction results. Predictions with a confidence level higher than 0.9 are directly adopted; those with a confidence level lower than 0.9 are marked for subsequent manual review. Based on the final field mapping relationships, content is extracted from the original XML, assembled into structured JSON or database records, and delivered to the downstream data validation and database entry module.

[0117] For the purpose of simplicity, the method steps disclosed in the above embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0118] like Figure 4 As shown, the present invention also provides a heterogeneous document intelligent extraction system, comprising:

[0119] The parsing and processing module 201 is configured to acquire the heterogeneous structured document to be processed, perform parsing and processing, and extract the path expressions of all data nodes in the document. The path expressions are represented in a standard syntax format and include tag names, attribute information and hierarchical relationships.

[0120] The path expression input module 202 is configured to input path expressions in batches into a pre-trained field classification model. The field classification model is trained based on the correspondence between path expression samples and target structured field labels, learns the feature patterns of path expressions, and maps path expressions to a predefined set of structured fields.

[0121] The prediction result output module 203 is configured to perform feature analysis and classification calculation on each path expression based on the field classification model, and output the corresponding target structured field prediction results. The prediction results include the determined field labels and the corresponding confidence scores.

[0122] The mapping relationship establishment module 204 is configured to establish a mapping relationship from the path expression to the target field based on the prediction result, and extract the node content data corresponding to each path expression from the original structured document based on the mapping relationship;

[0123] The confidence assessment and selection output module 205 is configured to fill the node content data into the corresponding target structured fields according to the mapping relationship, and generate a standard format structured data record. Among them, the prediction results with confidence scores higher than the preset threshold are filled with content, and the prediction results with confidence scores lower than the preset threshold are marked and reserved for manual review.

[0124] It is worth noting that although only some basic functional modules are disclosed in the embodiments of this invention, it does not mean that the composition of this system is limited to the above-mentioned basic functional modules. On the contrary, what this embodiment intends to express is that, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technology to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. The fact that this embodiment only discloses a few basic functional modules should not be considered as the scope of protection of the claims of this invention being limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above device is described separately according to its functions as various units and modules. Of course, in implementing this invention, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0125] like Figure 5 As shown, the present invention also provides an electronic device, including: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory stores a computer program, and when the computer program is executed by the processor, the processor performs the steps of a heterogeneous document intelligent extraction method.

[0126] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. For example... Figure 5 The structure shown in this embodiment of the invention includes an electronic device comprising one or more processors 710 and a memory 720; the processors 710 in this electronic device may be one or more. Figure 5 Taking a processor 710 as an example; the memory 720 is used to store one or more programs; the one or more programs are executed by the one or more processors 710, so that the one or more processors 710 implement a heterogeneous document intelligent extraction method as described in any one of the embodiments of the present invention.

[0127] The electronic device may also include an input device 730 and an output device 740.

[0128] The processor 710, memory 720, input device 730, and output device 740 in this electronic device can be connected via a bus or other means. Figure 5 Taking the example of a connection between China and Israel via a bus.

[0129] The memory 720 in this electronic device serves as a computer-readable storage medium, capable of storing one or more programs. These programs can be software programs, computer-executable programs, or modules, such as the program instructions / modules corresponding to the heterogeneous document intelligent extraction method provided in this embodiment of the invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 720, thereby implementing the heterogeneous document intelligent extraction method described in the above embodiment.

[0130] The memory 720 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device. Furthermore, the memory 720 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 720 may further include memory remotely located relative to the processor 710, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0131] Input device 730 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the electronic device. Output device 740 may include display devices such as a display screen.

[0132] The present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of a heterogeneous document intelligent extraction method.

[0133] Specifically, the computer storage medium in this embodiment of the invention can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be—but is not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0134] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for intelligent extraction of heterogeneous documents, characterized in that, include: The heterogeneous structured document to be processed is parsed and the path expressions of all data nodes in the document are extracted. The path expressions are represented in a standard syntax format and include tag names, attribute information and hierarchical relationships. The path expressions are input into a pre-trained field classification model in batches. The field classification model is trained based on the correspondence between path expression samples and target structured field labels, learns the feature patterns of path expressions, and maps path expressions to a predefined set of structured fields. Based on the field classification model, feature analysis and classification calculation are performed on each path expression, and the corresponding target structured field prediction results are output. The prediction results include the determined field labels and the corresponding confidence scores. Based on the prediction results, establish a mapping relationship between path expressions and target fields, and extract the node content data corresponding to each path expression from the original structured document based on the mapping relationship; The node content data is filled into the corresponding target structured fields according to the mapping relationship to generate a standard format structured data record. Among them, the prediction results with confidence scores higher than the preset threshold are filled with content, while the prediction results with confidence scores lower than the preset threshold are marked and reserved for manual review.

2. The method for intelligent extraction of heterogeneous documents according to claim 1, characterized in that, The heterogeneous structured document to be processed is parsed and processed to extract the path expressions of all data nodes in the document. The path expressions are represented in a standard syntax format and include tag names, attribute information, and hierarchical relationships, further including: Receive heterogeneous structured document input to be processed, and verify the integrity and validity of the document format; A document parser is used to parse and process the document, and a document object model organized in a tree structure is constructed. Access each data node in the document object model through a traversal algorithm, and record the node's type, attributes, and location information; Standardized path expressions are generated based on the label names, attribute values, and hierarchical relationships of data nodes, and the association between the path expressions and the node content is stored.

3. The method for intelligent extraction of heterogeneous documents according to claim 1, characterized in that, The path expressions are input in batches into a pre-trained field classification model. This model is trained based on the correspondence between path expression samples and target structured field labels, learning the feature patterns of the path expressions and mapping them to a predefined set of structured fields. Further steps include: The extracted path expressions are preprocessed, including format standardization, length adjustment, and outlier handling. The preprocessed path expressions are grouped according to a preset batch size to form batch input data; Feature extraction is performed on each batch of path expressions using a field classification model. The extracted features include structural features and semantic features. The extracted features are input into the model's computation layer to obtain preliminary mapping results from path expressions to field labels.

4. The method for intelligent extraction of heterogeneous documents according to claim 1, characterized in that, Based on a field classification model, feature analysis and classification calculation are performed on each path expression, outputting the corresponding target structured field prediction results. These prediction results include determined field labels and corresponding confidence scores, and further include: The field classification model calculates the probability distribution of each path expression belonging to each target field and generates probability values ​​through the output layer activation function; Select the field label corresponding to the highest probability value from the probability distribution as the main prediction result; The confidence score of the prediction result is calculated based on the degree of dispersion of the probability distribution or the probability value output by the model. The output includes complete prediction results with field labels and confidence scores, and is classified according to confidence intervals.

5. The method for intelligent extraction of heterogeneous documents according to claim 1, characterized in that, Based on the prediction results, a mapping relationship is established between path expressions and target fields. Then, based on this mapping relationship, node content data corresponding to each path expression is extracted from the original structured document. This further includes: Based on the field labels in the prediction results, construct a mapping table between path expressions and target fields; Based on the mapping relationship, locate the data node corresponding to each path expression in the original structured document; Extract the content information of the located data nodes, including text data, attribute values, and child node content; The extracted information is cleaned and formatted, including character encoding standardization and filtering of invalid data.

6. The method for intelligent extraction of heterogeneous documents according to claim 1, characterized in that, The node content data is populated into the corresponding target structured fields according to the mapping relationship to generate standard format structured data records. Specifically, prediction results with confidence scores higher than a preset threshold are populated with content, while prediction results with confidence scores lower than a preset threshold are marked and have a manual review channel reserved. Further steps include: The extracted node content data is allocated to the corresponding target field storage area according to the mapping relationship; For predictions with confidence scores higher than a preset threshold, content filling and target field updates are automatically performed. For predictions with a confidence score below a preset threshold, the path expression, prediction results, and extracted content are packaged and marked as samples to be reviewed. Integrate all the populated field contents, assemble them into a complete structured data record, and output the results in a standard format.

7. The method for intelligent extraction of heterogeneous documents according to claim 1, characterized in that, The training field classification model includes: Collect historical structured document data, extract the correspondence between path expressions and field labels to construct a training dataset; Feature engineering is performed on path expression samples in the training dataset to extract structural features and sequence feature vectors; A machine learning algorithm is used to train a field classification model with feature vectors as input. The model's accuracy, recall, and F1 score are evaluated using a validation set, and the model with the best overall performance is selected as the final field classification model.

8. A heterogeneous document intelligent extraction system, characterized in that, include: The parsing and processing module is configured to acquire heterogeneous structured documents to be processed, parse and process them, and extract the path expressions of all data nodes in the document. The path expressions are represented in a standard syntax format and include tag names, attribute information and hierarchical relationships. The path expression input module is configured to input path expressions in batches into a pre-trained field classification model. The field classification model is trained based on the correspondence between path expression samples and target structured field labels, learns the feature patterns of path expressions, and maps path expressions to a predefined set of structured fields. The prediction result output module is configured to perform feature analysis and classification calculation on each path expression based on the field classification model, and output the corresponding target structured field prediction results. The prediction results include the determined field labels and the corresponding confidence scores. The mapping relationship establishment module is configured to establish a mapping relationship from path expressions to target fields based on the prediction results, and extract the node content data corresponding to each path expression from the original structured document based on the mapping relationship; The confidence assessment and output selection module is configured to fill the node content data into the corresponding target structured fields according to the mapping relationship, and generate structured data records in a standard format. Among them, the prediction results with confidence scores higher than the preset threshold are filled with content, while the prediction results with confidence scores lower than the preset threshold are marked and reserved for manual review.

9. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus; the memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a computer program executable by an electronic device, which, when run on the electronic device, causes the electronic device to perform the steps of the method according to any one of claims 1 to 7.

Images