Algorithm for automatically generating table from important parameters in advance in document

Abstract

The invention relates to an algorithm for automatically generating a table from important parameters in advance from a document, which comprises the following steps of: preprocessing the document of which the table needs to be converted, and standardizing the format of the document; using a natural language processing technology to locate key parameters in the document, training a machine learning model to identify a specific type of parameters or using a rule engine to match a predefined mode; and specific numerical values and descriptions are extracted from the positioned key information, and text contexts are understood by using a regular expression and an analytical algorithm. According to the method, the pre-trained deep learning model is adopted, strong semantic understanding and context sensing capabilities are achieved, complex structures and semantic relationships in documents can be more accurately understood, so that the accuracy and correlation of parameter extraction are improved, rich language features and modes can be learned through pre-training on a large-scale corpus, and the method is suitable for large-scale application. Even for a newly appearing document format or parameter type, high extraction precision can be kept.

Description

Technical Field

[0001] The present invention relates to the technical field of text processing, and in particular to an algorithm for automatically generating a table from important parameters in advance from a document. Background Art

[0002] During the document processing process, there will be a need to generate a table from the content in the document. The current technology for automatically extracting parameters to generate tables is usually based on rule engines, regular expressions, traditional machine learning algorithms, etc.

[0003] However, the above method still has the following problems:

[0004] 1. Limitations of rule matching: Rule engines and regular expressions rely on predefined matching rules, which are more effective when processing documents with fixed formats and clear structures. However, when faced with documents with complex, diverse or noisy structures, the accuracy of rule matching will be greatly reduced.

[0005] 2. Insufficient generalization ability: Although traditional machine learning algorithms, such as support vector machines (SVM), random forests, or naive Bayes, can handle some classification and regression problems, their generalization ability is limited. Especially when dealing with new types of documents or parameters, the adaptability and accuracy of the model usually decrease.

[0006] 3. Challenges in processing long documents and understanding context: Technologies that do not rely on large models may find it difficult to capture the long-term dependencies and contextual information of the text when processing long documents. This may lead to misunderstanding or omission of important contextual information when extracting parameters, affecting the accuracy and relevance of parameter extraction.

[0007] 4. Limitations of iteration and optimization: The optimization and iteration of rule engines and traditional machine learning models usually require manual intervention, such as adjusting rules, selecting features, adjusting parameters, etc., which increases maintenance costs and cannot adapt to new document formats or parameter types in real time or automatically.

[0008] 5. Processing efficiency and resource consumption: When processing large-scale data sets, some rule-based or traditional machine learning technologies may consume a lot of computing resources, have slow processing speeds, and have difficulty responding in real time. Especially in resource-constrained environments, they may not be able to meet high-frequency data processing needs. Summary of the invention

[0009] In view of the deficiencies in the prior art, the present invention provides an algorithm for automatically generating a table of important parameters from a document in advance, which has the advantages of improving the accuracy and robustness of parameter extraction, enhancing processing efficiency and speed, and improving adaptability and flexibility.

[0010] To achieve the above object, the present invention provides the following technical solution: an algorithm for automatically generating a table from important parameters in advance from a document, comprising the following steps:

[0011] Step 1: Document preprocessing

[0012] Pre-process the documents that need to be converted and standardize the document format;

[0013] Step 2: Key information location

[0014] Use natural language processing techniques to locate key parameters in documents, train machine learning models to identify specific types of parameters or use rule engines to match predefined patterns;

[0015] Step 3: Parameter extraction

[0016] Extract specific values ​​and descriptions from the located key information, and use regular expressions and parsing algorithms to understand the text context;

[0017] Step 4: Structured data conversion

[0018] Convert the extracted parameters into structured data format, design data model, and define the categories and relationships of parameters;

[0019] Step 5: Table Generation

[0020] Use table generation software to generate a table from the structured data in step 4;

[0021] Step 6: Verification and optimization

[0022] The generated table needs to be verified;

[0023] Step 7: Output and Integration

[0024] Output the generated tables into the required format for integration with other systems or databases.

[0025] Preferably, the document preprocessing includes removing blank characters, segmenting words, adjusting the document format, and ensuring that punctuation conforms to Chinese and English and paragraph formats.

[0026] Preferably, the natural language processing technology includes entity recognition, keyword extraction and semantic analysis. The entity recognition content includes names of people, places, institutions and proper nouns. When extracting keywords, keywords are input according to needs, and then corresponding content is found based on the keywords. The semantic analysis is used to interpret and analyze the semantics of the document.

[0027] Preferably, Python is used for machine learning model training and evaluation in step 2, and the specific steps are as follows:

[0028] 1) Install the Scikit-learn library;

[0029] 2) Obtaining the preprocessed document data;

[0030] 3) Preprocess the document data, including handling missing values, encoding categorical variables, and standardizing data;

[0031] 4) Use logistic regression model for training;

[0032] 5) After the model training is completed, the accuracy, precision, recall and F1 score indicators are used to evaluate the model.

[0033] Preferably, the regular expression is used to describe and match the pattern of the string, and the content in the text is matched, searched, and replaced by using specific characters and patterns. The parsing algorithm includes data acquisition and processing, computing power optimization, corpus processing, and specific format document parsing.

[0034] Preferably, the parameters extracted in step 4 are value pairs or rows and columns in a table, and step 5 uses Python's Pandas to fill the data into the table to generate a table.

[0035] Preferably, the data model in step 4 describes the static characteristics, dynamic behaviors and constraints of the system at an abstract level, and provides an abstract framework for information representation and operation of the database system.

[0036] Preferably, the table is verified to ensure that important parameters are correctly extracted and displayed, and errors and omissions are corrected by combining manual inspection and automated inspection mechanisms.

[0037] Preferably, the structured data format is data logically expressed and implemented by a two-dimensional table structure, which is stored and managed through a relational database, including clear column definitions, a fixed data model and structure, and each row represents a data record. The table formats that can be output in step seven include Excel, CSV or database.

[0038] Compared with the prior art, the technical solution of this application has the following beneficial effects:

[0039] 1. The present invention has strong semantic understanding capabilities and adopts pre-trained deep learning models such as BERT, GPT, etc. It has strong semantic understanding and context perception capabilities and can more accurately understand the complex structures and semantic relationships in documents, thereby improving the accuracy and relevance of parameter extraction.

[0040] 2. The present invention has stronger generalization ability. By pre-training on a large-scale corpus, it can learn rich language features and patterns, which gives it better generalization ability and can maintain high extraction accuracy even in the face of new document formats or parameter types.

[0041] 3. The invention has the ability to process complex documents, can handle long documents and complex structures, and capture long-term dependencies and contextual information in the text, which is especially important for extracting parameters that rely on contextual understanding.

[0042] 4. The present invention automatically learns and iterates, and can optimize parameter extraction rules through automatic learning and iteration, reduce manual intervention, and improve efficiency and accuracy. This enables the model to adapt to new document formats and parameter types more quickly, and improve adaptability and flexibility.

[0043] 5. The parallel processing and high efficiency of the present invention can significantly improve the speed of processing a large number of documents by utilizing the parallel processing capabilities of large models, meeting the needs of real-time or high-frequency data processing, especially in a cloud computing environment, and can make full use of distributed computing resources to improve processing efficiency. DETAILED DESCRIPTION

[0044] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0045] In this embodiment, an algorithm for automatically generating a table from important parameters in advance from a document includes the following steps:

[0046] Step 1: Document preprocessing

[0047] Pre-process the documents that need to be converted and standardize the document format;

[0048] Step 2: Key information location

[0049] Use natural language processing techniques to locate key parameters in documents, train machine learning models to identify specific types of parameters or use rule engines to match predefined patterns;

[0050] Step 3: Parameter extraction

[0051] Extract specific values ​​and descriptions from the located key information, and use regular expressions and parsing algorithms to understand the text context;

[0052] Step 4: Structured data conversion

[0053] Convert the extracted parameters into structured data format, design data model, and define the categories and relationships of parameters;

[0054] Step 5: Table Generation

[0055] Use table generation software to generate a table from the structured data in step 4;

[0056] Step 6: Verification and optimization

[0057] The generated table needs to be verified;

[0058] Step 7: Output and Integration

[0059] Output the generated tables into the required format for integration with other systems or databases.

[0060] In an embodiment, the format in the preprocessed document is standardized, the blank characters, word segmentation, document format adjustment, and punctuation in the document are adjusted to ensure that they conform to Chinese, English, and paragraph formats. The document format is strictly controlled to ensure a clear document structure for easy subsequent processing.

[0061] In this embodiment, the natural language processing technology includes entity recognition, keyword extraction and semantic analysis. The entity recognition content includes human names, place names, organization names and proper nouns. When extracting keywords, keywords are input according to needs, and then corresponding content is found according to the keywords. Semantic analysis is used to interpret and analyze the semantics of the document, specifically information extraction and automatic summarization, where information extraction is the process of extracting unstructured information embedded in the text and converting it into structured data. The main process of information extraction has three steps: first, automatic processing of unstructured data, second, targeted extraction of text information, and finally structured representation of the extracted information. Automatic summarization is an information compression technology that uses a computer to automatically extract text information according to a certain rule and assemble it into a short summary, aiming to achieve two goals: first, make the language brief, and second, retain important information.

[0062] In this embodiment, Python is used to train and evaluate the machine learning model in step 2. The specific steps are as follows:

[0063] 1) Install the Scikit-learn library;

[0064] 2) Obtaining the preprocessed document data;

[0065] 3) Preprocess the document data, including handling missing values, encoding categorical variables, and standardizing data;

[0066] 4) Use logistic regression model for training;

[0067] 5) After the model training is completed, the accuracy, precision, recall and F1 score indicators are used to evaluate the model.

[0068] In this embodiment, regular expressions are used to describe and match string patterns, and specific characters and patterns are used to match, search, and replace content in text. The parsing algorithm includes data acquisition and processing, computing power optimization, corpus processing, and specific format document parsing.

[0069] In this embodiment, Python's Pandas mainly provides two data structures, namely Series and DataFrame. Series is a one-dimensional data structure suitable for storing a single column of data sequences and has an index associated with it. DataFrame is a two-dimensional, variable-size, potentially heterogeneous tabular data structure that is very suitable for processing data sets containing multiple columns. The design of these two structures makes Pandas particularly suitable for processing tabular data, such as Excel tables or tables in SQL databases. In terms of data processing, Pandas can easily perform data cleaning and conversion tasks, such as processing of missing values, data alignment and manipulation, etc. These functions are essential for preparing the preprocessing steps before data analysis. In terms of data analysis, in addition to basic data operations, Pandas also supports data aggregation and statistics, as well as time series analysis and other functions. It allows users to group, merge, and filter data based on complex logic, making data analysis more efficient and flexible. In terms of data visualization, Pandas is closely integrated with graphics libraries such as Matplotitb, providing a simple interface to create various types of charts to help users better understand data distribution and trends. In terms of ease of use, Pandas was designed with user experience in mind. Its API is concise and clear, so even beginners can easily get started and apply it to practical problem solving. Community support: As an open source project, Pandas has active community support and rich documentation resources, which is very helpful for users to learn and solve problems.

[0070] In this embodiment, the data model in step 4 describes the static characteristics, dynamic behaviors and constraints of the system at an abstract level, provides an abstract framework for information representation and operation of the database system, ensures that important parameters are correctly extracted and displayed during table verification, and uses a combination of manual inspection and automated inspection mechanisms to correct errors and omissions. The structured data format is a data logically expressed and implemented by a two-dimensional table structure, which is stored and managed through a relational database. It includes clear column definitions, a fixed data model and structure, and each row represents a data record. The table formats that can be output in step 7 include Excel, CSV or database.

[0071] By adopting the above technical solution, the present invention has a strong semantic understanding ability, adopts a pre-trained deep learning model, such as BERT, GPT, etc., has a strong semantic understanding and context perception ability, can more accurately understand the complex structure and semantic relationship in the document, thereby improving the accuracy and relevance of parameter extraction. The generalization ability is stronger. By pre-training on a large-scale corpus, it can learn rich language features and patterns, which makes it have better generalization ability, even in the face of new document formats or parameter types, it can maintain a high extraction accuracy. In terms of the ability to process complex documents, it can handle long documents and complex structures, capture long-term dependencies and context information in the text, which is particularly important for extracting parameters that rely on context understanding. It has automatic learning and iteration capabilities, can optimize parameter extraction rules through automatic learning and iteration, reduce manual intervention, improve efficiency and accuracy, which enables the model to adapt to new document formats and parameter types more quickly, and improve adaptability and flexibility. It has parallel processing and high efficiency. By using the parallel processing capabilities of large models, the speed of processing a large number of documents can be significantly improved to meet the needs of real-time or high-frequency data processing, especially in cloud computing environments, it can make full use of distributed computing resources and improve processing efficiency.

[0072] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0073] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. An algorithm for automatically generating a table from important parameters in advance from a document, characterized in that: The steps include: Step 1: Document preprocessing Pre-process the documents that need to be converted and standardize the document format; Step 2: Key information location Use natural language processing techniques to locate key parameters in documents, train machine learning models to identify specific types of parameters or use rule engines to match predefined patterns; Step 3: Parameter extraction Extract specific values ​​and descriptions from the located key information, and use regular expressions and parsing algorithms to understand the text context; Step 4: Structured data conversion Convert the extracted parameters into structured data format, design data model, and define the categories and relationships of parameters; Step 5: Table Generation Use table generation software to generate a table from the structured data in step 4; Step 6: Verification and optimization The generated table needs to be verified; Step 7: Output and Integration Output the generated tables into the required format for integration with other systems or databases.

2. The algorithm for automatically generating a table of important parameters from a document according to claim 1, characterized in that: The document preprocessing includes removing blank characters, segmenting words, adjusting the document format, and ensuring that punctuation conforms to Chinese and English and paragraph formats.

3. The algorithm for automatically generating a table of important parameters from a document according to claim 1, characterized in that: The natural language processing technology includes entity recognition, keyword extraction and semantic analysis. The entity recognition content includes names of people, places, institutions and proper nouns. When extracting keywords, keywords are input according to needs, and then corresponding content is found based on the keywords. The semantic analysis is used to interpret and analyze the semantics of the document.

4. The algorithm for automatically generating a table of important parameters from a document according to claim 1, characterized in that: In step 2, Python is used to train and evaluate the machine learning model. The specific steps are as follows: 1) Install the Scikit-learn library; 2) Obtaining the preprocessed document data; 3) Preprocess the document data, including handling missing values, encoding categorical variables, and standardizing data; 4) Use logistic regression model for training; 5) After the model training is completed, the accuracy, precision, recall and F1 score indicators are used to evaluate the model.

5. The algorithm for automatically generating a table of important parameters from a document according to claim 1, characterized in that: The regular expression is used to describe and match the pattern of the string, and matches, searches, and replaces the content in the text by using specific characters and patterns. The parsing algorithm includes data acquisition and processing, computing power optimization, corpus processing, and parsing of documents in specific formats.

6. The algorithm for automatically generating a table of important parameters from a document according to claim 1, characterized in that: The parameters extracted in step 4 are value pairs or rows and columns in a table, and step 5 uses Python's Pandas to fill the data into the table to generate a table.

7. The algorithm for automatically generating a table from important parameters in advance from a document according to claim 1, characterized in that: The data model in step 4 describes the static characteristics, dynamic behaviors and constraints of the system at an abstract level, and provides an abstract framework for information representation and operation of the database system.

8. The algorithm for automatically generating a table from important parameters in advance from a document according to claim 1, characterized in that: The table verification ensures that important parameters are correctly extracted and displayed, and adopts a combination of manual and automated checking mechanisms to correct errors and omissions.

9. The algorithm for automatically generating a table from important parameters in advance from a document according to claim 1, characterized in that: The structured data format is data logically expressed and implemented by a two-dimensional table structure, which is stored and managed through a relational database. It includes clear column definitions, a fixed data model and structure, and each row represents a data record. The table formats that can be output in step seven include Excel, CSV or database.