Natural language rule table information extraction system based on large model

Abstract

The invention discloses a natural language rule table information extraction system based on a large model, and relates to the technical field of table information extraction, and the system comprises a data input module which obtains original data from a file uploaded by a user; the data analysis and information extraction module is used for dynamically identifying and extracting field information in the table based on a natural language rule and an intelligent analysis model; the data conversion module is used for formatting the field information into a uniform cost template and carrying out data verification and formatting processing; the data output module is used for uploading the generated template to an internal system; the user feedback and correction module is used for correcting the template based on user feedback; and the self-learning and rule optimization module is used for optimizing data analysis rules and models through feedback and correction of the user on the basis of a self-learning mechanism. According to the method, the field information in the table can be dynamically identified through the intelligent analysis model and the natural language rule, so that the efficiency is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of table information extraction, and in particular to a natural language rule table information extraction system based on a large model. Background Art

[0002] Tabular information extraction is widely used in various business scenarios, especially when processing structured data and generating reports. As a common way of presenting data, tables often carry a large amount of important information. Traditional tabular information extraction methods mainly rely on rules and templates, such as regular expressions, field name matching, keyword extraction and other technologies, but these methods are usually difficult to handle complex, dynamic, and ambiguous table structures. Especially when the table content is diverse and the data type is rich, the accuracy and efficiency of traditional methods are often limited.

[0003] With the development of natural language processing, deep learning and big model technology, table information extraction based on intelligent models has gradually become an effective solution. These intelligent systems can automatically learn table structure, associations between fields, and contextual semantics based on big data and deep learning models, thereby achieving accurate table data extraction. However, although the current table information extraction technology based on big models has great potential, its effectiveness still depends on how to effectively utilize user feedback, corrections and self-learning mechanisms to continuously optimize data parsing rules and improve accuracy and flexibility. Therefore, in the existing technology, how to combine the natural language rules of the big model with the table structure and use the self-learning mechanism for real-time optimization has become a key technical challenge to improve the effectiveness of the table information extraction system. Summary of the invention

[0004] In order to solve the above technical problems, a natural language rule table information extraction system based on a large model is provided. This technical solution solves the above problems.

[0005] In order to achieve the above purpose, the technical solution adopted by the present invention is:

[0006] A natural language rule table information extraction system based on a large model, including:

[0007] Data input module: obtains raw data from files uploaded by users;

[0008] Data parsing and information extraction module: The data parsing and information extraction module is electrically connected to the data input module, and is used to dynamically identify and extract field information in the table based on natural language rules and intelligent parsing models;

[0009] Data conversion module: The data conversion module is electrically connected to the data analysis and information extraction module, and is used to format the field information into a unified cost template and perform data verification and formatting processing;

[0010] Data output module: The data output module is electrically connected to the data conversion module, and is used to upload the generated template to the internal system and collect user feedback at the same time;

[0011] User feedback and correction module: the user feedback and correction module is electrically connected to the data output module, and the user feedback and correction module is used to correct the template based on user feedback;

[0012] Self-learning and rule optimization module: The self-learning and rule optimization module is electrically connected to the user feedback and correction module. The self-learning and rule optimization module is used to optimize data analysis rules and models based on a self-learning mechanism through user feedback and correction.

[0013] Preferably, the data input module specifically includes:

[0014] File upload and receiving unit: provides an interface for users to select and upload files, supports multiple file formats, including PDF, Excel, Word, CSV, pictures, and text files, verifies whether the uploaded files conform to the supported file format standards, and prompts users to re-upload if the format does not meet the requirements;

[0015] File storage and management unit: temporarily save the files uploaded by users in the designated temporary directory of the server, uniquely name the files, and deduplicate the files.

[0016] Preferably, the data parsing and information extraction module specifically includes:

[0017] Data preprocessing unit: cleans the original data and unifies the character encoding, and restores the table structure based on the table layout analysis algorithm;

[0018] Field identification and rule application unit: Based on natural language rules, keyword lists and field name recognition mechanisms, key fields in the table are identified, and data in a specific format is extracted from the data based on regular expressions;

[0019] Field type inference unit: Through intelligent parsing technologies such as context understanding and semantic analysis, it dynamically identifies the field relationships in the table and determines the semantic associations between fields. It identifies fields based on surrounding field information and context, uses deep learning models to semantically classify fields, and predicts possible fields and field values ​​in the table based on historical data and model training.

[0020] Field relationship reasoning unit: Based on the row and column position analysis of the table, the contextual relationship between fields is inferred, rules are set to identify and infer data associations in the table, and the NLP model is used to analyze and infer the semantic associations between fields to identify semantic relationships.

[0021] Preferably, the method of dynamically identifying the field relationship in the table and determining the semantic association between the fields through intelligent parsing technologies such as context understanding and semantic analysis, performing field identification based on surrounding field information and context, using a deep learning model to perform semantic classification of the fields, and predicting possible fields and field values ​​in the table based on historical data and model training specifically includes:

[0022] Clean field names, perform preliminary keyword matching, identify potential fields, and infer the semantics and type of fields by analyzing field context and adjacent fields;

[0023] Use the trained deep learning model to classify the fields;

[0024] Use historical datasets to train predictive models to predict field values ​​based on field names and context;

[0025] Based on feedback from actual applications, we continue to optimize the field type inference model;

[0026] Outputs the inferred field type and predicted field value.

[0027] Preferably, the prediction model is trained using the historical data set, and the field value is predicted based on the field name and context, specifically including:

[0028] Among them, the prediction model formula is:

[0029] y=β 0 +β 1 x 1 +β 2 x 2 +…+β n x n

[0030] In the formula, x 1 x 2 x n is the value of each field in the context, β 1 β 2 β n is the regression coefficient of the corresponding field, y is the predicted field value, β 0 is the intercept term.

[0031] Preferably, the above-mentioned analysis of the row and column positions of the table, reasoning about the contextual relationship between the fields, setting rules to identify and reason about the data association in the table, using the NLP model to analyze and reason about the semantic association between the fields, and identifying the semantic relationship specifically includes:

[0032]

[0033] Where P(p|x) ​​represents the probability that a text has positive sentiment given an input text x, x is the input text, is the scoring function for the input text x corresponding to the positive sentiment category, is the scoring function for the input text x corresponding to negative sentiment, is the scoring function corresponding to the neutral sentiment of the input text x.

[0034] Preferably, the data conversion module specifically includes:

[0035] Field mapping unit: Design cost templates based on business requirements and target formats. The templates define the data format, field name, data type, and field order.

[0036] Map original fields to target fields unit: Map the fields in the original data with the fields in the target template;

[0037] Data validation unit: Checks whether the field data conforms to the expected format, including date format, numeric format, and string length, checks whether the numeric field is within the set range, ensures that all required fields have valid values, throws errors and warnings when required fields are empty, and checks whether the consistency rules are met between fields;

[0038] Data cleaning unit: Use default values ​​to handle missing values, find and delete duplicate records, and detect and handle outliers in the data based on Z-Score;

[0039] Data formatting unit: according to the requirements of the target template, convert the fields into the required format, standardize the data, convert all text fields into uniform uppercase, remove spaces and special characters in the string, and uniformly convert fields of different units;

[0040] Data merging and splitting unit: Merge multiple fields into one field, summarize and group data, and generate statistical information.

[0041] Preferably, the data output module specifically includes:

[0042] Template upload unit: encapsulates template data into json format and uploads it to the internal storage system;

[0043] User feedback collection unit: provides a feedback interface for users, and collects user feedback on the template through feedback forms, ratings, and comments filled in by users;

[0044] Feedback data storage: Convert user feedback data into structured data and store the feedback data in the database.

[0045] Preferably, the user feedback and correction module specifically includes:

[0046] Feedback data analysis unit: conduct sentiment analysis on user feedback to identify user satisfaction, pain points and needs. Based on the analysis results, identify the core issues reflected by users and assign a priority for correction to each issue based on the severity, frequency and impact on user experience.

[0047] Template modification and update unit: set modification targets, make actual modifications to templates according to modification plans, verify and test modified templates to ensure that modified templates meet expected functions and effects, perform version control on modified templates, and update version numbers;

[0048] Feedback and Correction Documentation Unit: Record all feedback issues, analysis results, correction plans and solutions in documents, generate correction logs, and record the content, time and correction personnel information of each correction.

[0049] Preferably, the self-learning and rule optimization module specifically includes:

[0050] Error pattern analysis and identification unit: Analyze the types of errors reported by users, identify which error patterns are most common, obtain the impact of different types of errors on user experience, and prioritize them;

[0051] Data parsing rule optimization unit: based on the analysis of error patterns, correct existing data parsing rules and add new parsing rules based on user feedback and new data requirements;

[0052] Self-learning mechanism unit: Based on user feedback and revised historical data, the model is optimized using machine learning algorithms. Based on new data and feedback, the model automatically adjusts its parameters;

[0053] Real-time feedback loop and update unit: When rules and models are optimized, the versions in the system are updated.

[0054] Compared with the prior art, the present invention has the following beneficial effects:

[0055] The present invention proposes that through intelligent parsing models and natural language rules, field information in a table can be dynamically identified without relying on the absolute position of the field. This dynamic parsing mechanism can ensure accurate extraction of key data even when the table format changes. It provides a way to define fields, conditions and processing logic through natural language rules, supports high flexibility and generalization of field parsing, and uses pre-trained language models to automatically identify and parse common terms and professional terms, avoiding the limitations of traditional methods that require manual declaration and enumeration of terms. It supports parallel processing of large-scale, multi-format quotations, is suitable for scenarios where a large number of supplier quotations need to be processed, and greatly improves efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Figure 1 It is a system framework diagram of the present invention. DETAILED DESCRIPTION

[0057] The following description is used to disclose the present invention so that those skilled in the art can implement the present invention. The preferred embodiments described below are only examples, and those skilled in the art may think of other obvious variations.

[0058] Reference Figure 1 As shown, the natural language rule table information extraction system based on the large model includes:

[0059] Data input module:

[0060] File upload and receiving unit: provides an interface for users to select and upload files, supports multiple file formats, including PDF, Excel, Word, CSV, pictures, and text files, verifies whether the uploaded files conform to the supported file format standards, and prompts users to re-upload if the format does not meet the requirements;

[0061] File storage and management unit: temporarily save the files uploaded by users in the designated temporary directory of the server, uniquely name the files, and deduplicate the files.

[0062] Data parsing and information extraction module:

[0063] Data preprocessing unit: cleans the original data and unifies the character encoding, and restores the table structure based on the table layout analysis algorithm;

[0064] Field identification and rule application unit: Based on natural language rules, keyword lists and field name recognition mechanisms, key fields in the table are identified, and data in a specific format is extracted from the data based on regular expressions;

[0065] Field type inference unit: Through intelligent parsing technologies such as context understanding and semantic analysis, it dynamically identifies the field relationships in the table and determines the semantic associations between fields. It identifies fields based on surrounding field information and context, uses deep learning models to semantically classify fields, and predicts possible fields and field values ​​in the table based on historical data and model training.

[0066] Clean field names, perform preliminary keyword matching, identify potential fields, and infer the semantics and type of fields by analyzing field context and adjacent fields;

[0067] Use the trained deep learning model to classify the fields;

[0068] Use historical datasets to train predictive models to predict field values ​​based on field names and context;

[0069] Among them, the prediction model formula is:

[0070] y=β 0 +β 1 x 1 +β 2 x 2 +…+β n x n

[0071] In the formula, x 1 x 2 x n is the value of each field in the context, β 1 β 2 β n is the regression coefficient of the corresponding field, y is the predicted field value, β 0 is the intercept term;

[0072] Based on feedback from actual applications, we continue to optimize the field type inference model;

[0073] Output the inferred field type and predicted field value;

[0074] Field relationship reasoning unit: Based on the row and column position analysis of the table, the contextual relationship between fields is inferred, rules are set to identify and infer the data association in the table, and the NLP model is used to analyze and infer the semantic association between fields to identify the semantic relationship;

[0075]

[0076] Where P(p|x) ​​represents the probability that a text has positive sentiment given an input text x, x is the input text, is the scoring function for the input text x corresponding to the positive sentiment category, is the scoring function for the input text x corresponding to negative sentiment, is the scoring function corresponding to the neutral sentiment of the input text x.

[0077] Data conversion module:

[0078] Field mapping unit: Design cost templates based on business requirements and target formats. The templates define the data format, field name, data type, and field order.

[0079] Map original fields to target fields unit: Map the fields in the original data with the fields in the target template;

[0080] Data validation unit: Checks whether the field data conforms to the expected format, including date format, numeric format, and string length, checks whether the numeric field is within the set range, ensures that all required fields have valid values, throws errors and warnings when required fields are empty, and checks whether the consistency rules are met between fields;

[0081] Data cleaning unit: Use default values ​​to handle missing values, find and delete duplicate records, and detect and handle outliers in the data based on Z-Score;

[0082] Data formatting unit: according to the requirements of the target template, convert the fields into the required format, standardize the data, convert all text fields into uniform uppercase, remove spaces and special characters in the string, and uniformly convert fields of different units;

[0083] Data merging and splitting unit: Merge multiple fields into one field, summarize and group data, and generate statistical information.

[0084] Data output module:

[0085] Template upload unit: encapsulates template data into json format and uploads it to the internal storage system;

[0086] User feedback collection unit: provides a feedback interface for users, and collects user feedback on the template through feedback forms, ratings, and comments filled in by users;

[0087] Feedback data storage: Convert user feedback data into structured data and store the feedback data in the database.

[0088] User feedback and correction module:

[0089] Feedback data analysis unit: conduct sentiment analysis on user feedback to identify user satisfaction, pain points and needs. Based on the analysis results, identify the core issues reflected by users and assign a priority for correction to each issue based on the severity, frequency and impact on user experience.

[0090] Template modification and update unit: set modification targets, make actual modifications to templates according to modification plans, verify and test modified templates to ensure that modified templates meet expected functions and effects, perform version control on modified templates, and update version numbers;

[0091] Feedback and Correction Documentation Unit: Record all feedback issues, analysis results, correction plans and solutions in documents, generate correction logs, and record the content, time and correction personnel information of each correction.

[0092] Self-learning and rule optimization module:

[0093] Error pattern analysis and identification unit: Analyze the types of errors reported by users, identify which error patterns are most common, obtain the impact of different types of errors on user experience, and prioritize them;

[0094] Data parsing rule optimization unit: based on the analysis of error patterns, correct existing data parsing rules and add new parsing rules based on user feedback and new data requirements;

[0095] Self-learning mechanism unit: Based on user feedback and revised historical data, the model is optimized using machine learning algorithms. Based on new data and feedback, the model automatically adjusts its parameters;

[0096] Real-time feedback loop and update unit: When rules and models are optimized, the versions in the system are updated.

[0097] In summary, the advantages of the present invention are:

[0098] Through natural language understanding technology based on large models, it is possible to identify complex field structures in tables, infer the relationship between fields, and dynamically parse table contents. Through deep learning and natural language processing technology, the system can effectively improve the accuracy of extracting table information.

[0099] It uses context understanding and semantic analysis technology to automatically infer field types and associations, and can process tables of different formats, complexity, and semantics, especially for files of different sources and formats.

[0100] The data conversion module can quickly and accurately format and verify the extracted data according to business needs and target formats. The field data will be automatically mapped to a unified template. The verification process can ensure that the field data conforms to the expected format, improving data consistency and accuracy.

[0101] The system automatically handles data cleaning, outlier detection, missing value filling and other issues, effectively reducing manual intervention and ensuring data quality;

[0102] The self-learning and rule optimization module can continuously learn and improve the data parsing model through user feedback. When users provide correction suggestions for the template, the system can automatically adjust the parsing rules and continuously train and optimize the model through historical data. This can maintain flexibility in different data sets and demand changes and optimize system performance. The system can automatically adjust parameters, identify and optimize the most common error modes, gradually reduce the error rate, and improve overall performance.

[0103] The data input module supports multiple file formats and can extract table information from various files, which makes the system applicable to a variety of practical scenarios and solves the problem that traditional rule systems cannot handle various file formats. Through image recognition and text extraction technology, the system can also process tables in picture format, providing support for more diversified application scenarios.

[0104] The user feedback and correction module can effectively collect and analyze user feedback information, thereby realizing dynamic optimization of the system. The feedback provided by users is not only used to correct templates, but also provides more training data for the self-learning mechanism. The feedback data analysis and sentiment analysis units can effectively identify user pain points and needs, quickly respond and optimize template design and parsing rules, thereby improving user satisfaction.

[0105] The seamless connection of multiple links from data input, analysis, conversion to output, feedback and correction ensures the efficiency and reliability of the process. Each link has been carefully designed to ensure that efficient table information extraction services can be provided under different scenarios and needs. Data processing functions such as data merging, splitting, and formatting make the extraction of table information not only accurate, but also convenient for subsequent operations, such as generating reports and data analysis.

[0106] Through the self-learning mechanism, the system can continuously improve based on feedback, gradually accumulate experience, enhance the accuracy and parsing ability of rules, and adapt to more application scenarios. Error pattern analysis and identification can continuously optimize data parsing rules, so that the model can handle more complex tables and data variants.

[0107] The system is based on a large model and self-learning mechanism, has strong scalability and adaptability, can handle a large number of file uploads, processing and parallel computing tasks, and shows good performance in large-scale data processing and high-frequency applications.

[0108] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions only describe the principles of the present invention. The present invention may be subject to various changes and improvements without departing from the spirit and scope of the present invention. These changes and improvements fall within the scope of the present invention. The scope of protection claimed by the present invention is defined by the attached claims and their equivalents.

Claims

1. A natural language rule table information extraction system based on a large model, characterized by: include: Data input module: obtains raw data from files uploaded by users; Data parsing and information extraction module: The data parsing and information extraction module is electrically connected to the data input module, and is used to dynamically identify and extract field information in the table based on natural language rules and intelligent parsing models; Data conversion module: The data conversion module is electrically connected to the data analysis and information extraction module, and is used to format the field information into a unified cost template and perform data verification and formatting processing; Data output module: The data output module is electrically connected to the data conversion module, and is used to upload the generated template to the internal system and collect user feedback at the same time; User feedback and correction module: the user feedback and correction module is electrically connected to the data output module, and the user feedback and correction module is used to correct the template based on user feedback; Self-learning and rule optimization module: The self-learning and rule optimization module is electrically connected to the user feedback and correction module. The self-learning and rule optimization module is used to optimize data analysis rules and models based on a self-learning mechanism through user feedback and correction.

2. The natural language rule table information extraction system based on a large model according to claim 1 is characterized in that: The data input module specifically includes: File upload and receiving unit: provides an interface for users to select and upload files, supports multiple file formats, including PDF, Excel, Word, CSV, pictures, and text files, verifies whether the uploaded files conform to the supported file format standards, and prompts users to re-upload if the format does not meet the requirements; File storage and management unit: temporarily save the files uploaded by users in the designated temporary directory of the server, uniquely name the files, and deduplicate the files.

3. The natural language rule table information extraction system based on a large model according to claim 2 is characterized in that: The data analysis and information extraction module specifically includes: Data preprocessing unit: cleans the original data and unifies the character encoding, and restores the table structure based on the table layout analysis algorithm; Field identification and rule application unit: Based on natural language rules, keyword lists and field name recognition mechanisms, key fields in the table are identified, and data in a specific format is extracted from the data based on regular expressions; Field type inference unit: Through intelligent parsing technologies such as context understanding and semantic analysis, it dynamically identifies the field relationships in the table and determines the semantic associations between fields. It identifies fields based on surrounding field information and context, uses deep learning models to semantically classify fields, and predicts possible fields and field values ​​in the table based on historical data and model training. Field relationship reasoning unit: Based on the row and column position analysis of the table, the contextual relationship between fields is inferred, rules are set to identify and infer data associations in the table, and the NLP model is used to analyze and infer the semantic associations between fields to identify semantic relationships.

4. The natural language rule table information extraction system based on a large model according to claim 3 is characterized in that: The intelligent parsing technologies such as context understanding and semantic analysis are used to dynamically identify the field relationships in the table and determine the semantic associations between the fields. Fields are identified based on surrounding field information and context, and deep learning models are used to semantically classify fields. Based on historical data and model training, possible fields and field values ​​in the table are predicted, including: Clean field names, perform preliminary keyword matching, identify potential fields, and infer the semantics and type of fields by analyzing field context and adjacent fields; Use the trained deep learning model to classify the fields; Use historical datasets to train predictive models to predict field values ​​based on field names and context; Based on feedback from actual applications, we continue to optimize the field type inference model; Outputs the inferred field type and predicted field value.

5. The natural language rule table information extraction system based on a large model according to claim 4 is characterized in that: Use historical data sets to train prediction models to predict field values ​​based on field names and context include: Among them, the prediction model formula is: y=β0+β1x1+β2x2+…+β n x n In the formula, x1x2x n is the value of each field in the context, β1β2β n is the regression coefficient of the corresponding field, y is the predicted field value, and β0 is the intercept term.

6. The natural language rule table information extraction system based on a large model according to claim 5 is characterized in that: The above-mentioned analysis of the row and column positions of the table, reasoning about the contextual relationship between fields, setting rules to identify and reason about the data association in the table, using the NLP model to analyze and reason about the semantic association between fields, and identifying the semantic relationship specifically includes: Where P(p|x) ​​represents the probability that a text has positive sentiment given an input text x, x is the input text, is the scoring function for the input text x corresponding to the positive sentiment category, is the scoring function for the input text x corresponding to negative sentiment, is the scoring function corresponding to the neutral sentiment of the input text x.

7. The natural language rule table information extraction system based on a large model according to claim 6 is characterized in that: The data conversion module specifically includes: Field mapping unit: Design cost templates based on business requirements and target formats. The templates define the data format, field name, data type, and field order. Map original fields to target fields unit: Map the fields in the original data with the fields in the target template; Data validation unit: Checks whether the field data conforms to the expected format, including date format, numeric format, and string length, checks whether the numeric field is within the set range, ensures that all required fields have valid values, throws errors and warnings when required fields are empty, and checks whether the consistency rules are met between fields; Data cleaning unit: Use default values ​​to handle missing values, find and delete duplicate records, and detect and handle outliers in the data based on Z-Score; Data formatting unit: according to the requirements of the target template, convert the fields into the required format, standardize the data, convert all text fields into uniform uppercase, remove spaces and special characters in the string, and uniformly convert fields of different units; Data merging and splitting unit: Merge multiple fields into one field, summarize and group data, and generate statistical information.

8. The natural language rule table information extraction system based on a large model according to claim 7 is characterized in that: The data output module specifically includes: Template upload unit: encapsulates template data into json format and uploads it to the internal storage system; User feedback collection unit: provides a feedback interface for users, and collects user feedback on the template through feedback forms, ratings, and comments filled in by users; Feedback data storage: Convert user feedback data into structured data and store the feedback data in the database.

9. The natural language rule table information extraction system based on a large model according to claim 8 is characterized in that: The user feedback and correction module specifically includes: Feedback data analysis unit: conduct sentiment analysis on user feedback to identify user satisfaction, pain points and needs. Based on the analysis results, identify the core issues reflected by users and assign a priority for correction to each issue based on the severity, frequency and impact on user experience. Template modification and update unit: set modification targets, make actual modifications to templates according to modification plans, verify and test modified templates to ensure that modified templates meet expected functions and effects, perform version control on modified templates, and update version numbers; Feedback and Correction Documentation Unit: Record all feedback issues, analysis results, correction plans and solutions in documents, generate correction logs, and record the content, time and correction personnel information of each correction.

10. The natural language rule table information extraction system based on a large model according to claim 9 is characterized in that: The self-learning and rule optimization module specifically includes: Error pattern analysis and identification unit: Analyze the types of errors reported by users, identify which error patterns are most common, obtain the impact of different types of errors on user experience, and prioritize them; Data parsing rule optimization unit: Based on the analysis of error patterns, the existing data parsing rules are revised, and new parsing rules are added according to user feedback and new data requirements; Self-learning mechanism unit: Based on user feedback and revised historical data, the model is optimized using machine learning algorithms. Based on new data and feedback, the model automatically adjusts its parameters; Real-time feedback loop and update unit: When rules and models are optimized, the versions in the system are updated.

Images