Non-template table intelligent recognition and storage method and system

Abstract

The application belongs to the technical field of data processing and big data warehousing. A non-templated table intelligent recognition and warehousing method and system are proposed. First, the table file is received and preprocessed, and the processed Sheet dataset is output. Then, the dataset is parsed by a large model to obtain a table structure set, and the set is globally matched with the metadata table to obtain a table-level mapping result. Field-level mapping results are generated in parallel and an artificial confirmation entry is provided. According to the confirmed mapping result, data type conversion is completed in batches with the help of a large model, and compliant data is screened by combining confidence verification. The dynamic data table is written through the update insertion mode. For data that does not pass the verification, an error file in the form of cell annotation is generated to complete the archiving. The application does not need to preset a fixed template, and can continuously complete table parsing, semantic mapping, data standardization warehousing and abnormal archiving operations. It is suitable for table files with different structures, and optimizes the operation process of table data import.

Description

Technical Field

[0001] This invention relates to the field of data processing and big data warehousing technology, specifically to a non-template-based table intelligent recognition and warehousing method and system. Background Technology

[0002] In digital business scenarios, various business data are generally transferred and exchanged across entities using spreadsheet files. Daily operations, qualification applications, patent collection, and other business processes all generate spreadsheet files with varying structures, making various office parsing components standard support tools for data import. Metadata systems are gradually being used to standardize the definition of business database table structures. Dynamic data tables can automatically create physical tables based on preset field rules. Large-scale model semantic parsing capabilities are increasingly being applied to the recognition of unstructured and semi-structured table content, enabling basic processing operations such as text semantic interpretation and content classification. Various data import programs are gradually integrating auxiliary functions such as parallel scheduling, anomaly labeling, and incremental writing to adapt to the automated processing needs of batch spreadsheet files. The fusion technology of table recognition and database writing has become an important research direction for enterprise data governance.

[0003] Current standard table import programs lack a complete hierarchical recognition and verification chain when handling table files without a unified standard. This makes it difficult to seamlessly complete the entire process, from file preprocessing, automatic splitting of multiple tables, global database-table matching, human-machine collaborative field calibration, data standardization conversion, and archiving of abnormal files. Existing tools can only execute single processing steps independently, failing to connect automatic table structure extraction and global metadata table matching. They lack a manual confirmation step for field mapping, a semantic adaptation verification mechanism in the data conversion stage, incremental update logic in the database import operation, and only return text prompts after data anomalies, failing to generate original error files with marked cells. The overall process is fragmented and difficult to adapt to the integrated recognition and database import needs of diverse table files. Summary of the Invention

[0004] To address the technical problems of existing table import tools, such as reliance on fixed templates, inability to automatically recognize multiple tables within a single sheet, inability to intelligently match column name semantics, poor data conversion tolerance, lack of closed-loop error location, and the need for manual code modification and deployment when data sources change, this invention provides a non-template-based table intelligent recognition and import method and system. It relies on large-model semantic parsing combined with metadata constraints to build a multi-stage human-machine collaborative recognition link. Through global table field weighted confidence matching, cell-level error annotation, and a dynamic data table automatic creation mechanism, it achieves automatic parsing of tables of any format without templates, semantic field mapping, and standardized batch import, improving the universality of data import from heterogeneous table files, the accuracy of imported data, and the efficiency of business iteration response.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: In one aspect, the present invention provides a method for intelligent recognition and database entry of non-template-based tables.

[0006] A non-template-based intelligent table recognition and database entry method includes the following process: Receive a table file, perform preprocessing to obtain the processed Sheet dataset; The large model is invoked to parse the processed Sheet dataset and generate a set of table structures. The table structure set is globally matched with the metadata table to obtain the table-level mapping result; The table-level mapping results are used to generate field-level mapping results in parallel and displayed for manual confirmation. Based on the confirmed field-level mapping results, the large model is called in batches to complete the data type conversion. After confidence verification, the data is written to the dynamic data table using the update and insert mode. Generate and archive error files with labeled incorrect cells for data that failed to be validated.

[0007] In one implementation of the first aspect of the present invention, the preprocessing operation for obtaining the processed Sheet dataset includes: Filter the hidden worksheets to obtain the retained worksheets, expand all merged cells in the retained worksheets, distinguish blank rows, header rows, and data rows, extract cell text, and integrate and output the processed Sheet dataset.

[0008] In one implementation of the first aspect of the present invention, when generating the set of table structures, multiple worksheets in the processed Sheet dataset are executed in parallel using virtual threads to call a large model; a multi-level cleaning and retry mechanism is performed on the text returned by the large model; and standardized table structures are extracted and summarized to form the set of table structures.

[0009] In one implementation of the first aspect of the present invention, the confidence score associated with generating the table-level mapping result is obtained by weighted calculation of four dimensions, namely, semantic similarity of table title, column name matching coverage, data type compatibility score, and metadata description alignment score; the confidence score is obtained by dynamically adjusting the weight of each dimension according to the business stage.

[0010] In one implementation of the first aspect of the present invention, when generating the field-level mapping result, supplementary prompt text input manually is received and injected into the large model as prompt words; the supplementary prompt text has a higher priority than the large model's autonomous semantic matching logic; the field-level mapping result is regenerated after switching the target metadata table.

[0011] In one implementation of the first aspect of the present invention, the large model prompt words adopt a three-layer hierarchical architecture, the three layers being a system prompt layer with fixed rules, a context prompt layer that loads metadata information, and a user supplementary prompt layer that carries the supplementary prompt text; the content of the three layers is concatenated and input into the large model to complete the mapping calculation.

[0012] In one implementation of the first aspect of the present invention, the data type conversion operation includes: converting cell text into five standard formats—integer, decimal, date, boolean, and text—based on the field definitions of the metadata table; and appending the confidence level and low confidence reason text to each conversion result.

[0013] In one implementation of the first aspect of the present invention, the update insertion mode execution logic includes: reading the unique key combination configured in the metadata table; querying the matching record in the dynamic data table; performing an update operation if the match is successful, and performing an insertion operation if the match fails; and synchronously injecting a multi-tenant identifier into each written data to complete data isolation. The steps to generate the error file include: creating two types of cell styles: abnormal cell style and error description style; locating the cells corresponding to the failed data and filling them with the abnormal cell style; appending the low confidence reason text to the end of the row; clearing the rows of successful data and then exporting the error file.

[0014] In one implementation of the first aspect of the present invention, a task reset operation is further included: receiving a reset instruction; deleting the data already written in the dynamic data table according to the task identifier; retaining the table-level mapping result and the field-level mapping result; and re-executing the entire data entry process after resetting the task status.

[0015] Secondly, the present invention provides a non-template-based table intelligent recognition and database entry system.

[0016] A non-template-based table intelligent recognition and database entry system includes: The file preprocessing unit is configured to receive a table file, perform preprocessing, and obtain a processed Sheet dataset. The table parsing unit is configured to: call the large model to parse the processed Sheet dataset and generate a set of table structures; The global table matching unit is configured to globally match the set of table structures with the metadata table to obtain a table-level mapping result. The field mapping unit is configured to generate field-level mapping results in parallel from the table-level mapping results and display them for manual confirmation. The data entry unit is configured to: call the large model in batches to complete the data type conversion based on the confirmed field-level mapping results, and write the data into the dynamic data table using the update insertion mode after confidence verification; The error generation unit is configured to generate and archive error files that annotate the cells with errors for data that failed to be validated.

[0017] Compared with the prior art, the beneficial effects of the present invention are: This invention sequentially executes file preprocessing, intelligent table parsing, global table matching, field mapping confirmation, data conversion and import, and error file archiving. It can seamlessly complete the entire processing chain from raw reading of diverse table files to importing and archiving errors, improving upon the fragmented workflows of existing tools. The preprocessing stage standardizes table cell content, facilitating the extraction of the complete table structure from subsequent large-scale models. The global matching mode simultaneously compares all identified tables with the metadata table, reducing adaptation bias caused by partial matching. Field mapping results are visualized for users and support manual correction, mitigating adaptation bias caused by simple machine semantic matching. Batch large-scale model data conversion combined with configuration verification can filter out data content with format deviations. The update and insertion mode, combined with dynamic data table writing, adapts to incremental data import scenarios, reducing storage redundancy caused by duplicate data. Error files with cell annotations are generated for failed verification, intuitively showing the location of data anomalies and reducing the operational cost for users to locate and modify them. The entire set of layered processing steps works together to adapt to various table file processing scenarios without fixed templates, broadening the applicability of table import programs and simultaneously optimizing the user experience of verification, correction, and archiving operations during data import.

[0018] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0019] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0020] Figure 1 A flowchart illustrating a non-template-based table intelligent recognition and database entry method provided as an exemplary embodiment of the present invention; Figure 2 A schematic diagram of a non-template-based table intelligent recognition and database entry system provided as an exemplary embodiment of the present invention; Figure 3 A schematic diagram of a computer device provided for an exemplary embodiment of the present invention. Detailed Implementation

[0021] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0022] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0023] This implementation proposes a non-template-based table intelligent recognition and database entry method. The following is a brief introduction to the technical terms and related concepts involved in this solution: Large Language Model (LLM): refers to a natural language processing model based on deep learning, which has the ability to understand semantics and generate text. In this invention, it is used for table structure parsing, table-level mapping, field-level mapping and data type conversion.

[0024] Meta Table: The system pre-configured business table structure definition, including fields such as table code, table name, and table description, which is the structural blueprint of dynamic data tables.

[0025] Meta Column: A pre-configured field definition that includes information such as column code, column name, column description, and field type (INT / DECIMAL / DATE / DATETIME / SMALLINT / VARCHAR / TEXT).

[0026] Dynamic data tables: Physical database tables that are automatically created at runtime using DDL based on the metadata table definition. The table name is mapped by the metadata table encoding, and the fields are driven by the metadata field definition.

[0027] Expand merged cells: This function fills all cells in a table with the value of the top-left cell from a merged cell range that spans multiple rows and columns, giving each cell an independent value.

[0028] Upsert mode: When importing data, query the unique key group to see if a record already exists. If a match is found, execute UPDATE to update the record; otherwise, execute INSERT to add the record, thus avoiding conflicts caused by duplicate imports.

[0029] Virtual thread parallelism: Based on a lightweight thread parallel execution model, each AI call task is allocated an independent virtual thread to achieve high concurrency and low overhead batch AI calls.

[0030] Human-machine collaborative mapping: A mapping paradigm that combines AI autonomous decision-making with human verification. AI outputs the initial mapping results and confidence levels, while humans verify, correct, and supplement at key nodes, forming a human-in-the-Loop quality control closed loop.

[0031] Multi-tenant data isolation: Queries and writes to dynamic data tables are isolated by tenant ID (company ID), ensuring that data from different tenants is not visible to each other.

[0032] More specifically, it includes the following steps: Includes the following steps: Step S1: File upload and preprocessing.

[0033] Receive the table file (.xlsx / .xls) uploaded by the user and perform the following preprocessing operations: S1.1 File Persistence: Store the uploaded file to Object Storage Service (OSS), generate a file ID, create an identification task record, and set the task status to "Mapping".

[0034] S1.2 Sheet Reading and Filtering: Use Apache POI to read the workbook, traverse all sheets, filter out hidden sheets (including those in the hidden and very hidden states), and only process visible sheets.

[0035] S1.3 Expanding Merged Cells: Perform a merged cell expansion operation on each visible sheet. Iterate through all merged areas (Cell Range Addresses) in the sheet, extract the value of the top-left cell, fill all cells within the merged area with that value, and then remove the merged area markers. After expansion, each cell has an independent value, facilitating subsequent column-by-column reading.

[0036] S1.4 Row Data Extraction and Classification: Read the Sheet data row by row and perform the following processing on each row: Calculate the number of non-empty cells in the row. If the number is 0, skip it (completely empty rows are used as implicit delimiters and are not included in AI input); determine whether all non-empty cells in the row have the same value (distinct count==1). If so, it is determined as the header row, and the multi-column values ​​are compressed into a single column value to reduce AI input redundancy; the remaining rows are treated as data rows, and the cell values ​​are concatenated into a CSV format string, recording the row number.

[0037] S1.5 Cell Value Extraction: Extracts cell string values ​​using the WYSIWYG (What You See Is What You Get) principle. Data Formatter is prioritized for output formatting; for numeric cells, the readability of the output is validated. If the output is unreadable (containing reserved characters, control characters, etc.), it is converted to a table date sequence number or as a fallback number. Specific rules include: For non-numeric cells, the Data Formatter result is directly reliable; for numeric cells with readable output, it is returned directly; for numeric cells with unreadable output, it is checked whether it is a valid table date (num Val > 1). If so, it is output in date format (including hours, minutes, and seconds, output yyyy-MM-dd HH:mm:ss; otherwise, output yyyy-MM-dd); for non-date cells, it is output as a pure number (integer values ​​are output as long, floating-point values ​​are output as double).

[0038] Step S2: AI table structure parsing.

[0039] For each sheet's list of rows, the large language model is used to parse the table structure. S2.1 Prompt Construction: The line data of the sheet (including line numbers and CSV content) is serialized into JSON and sent as the user prompt to the large language model. The system prompt instructs the AI ​​to recognize the following information: Table Title: The title text above the table. If there is no explicit title, it should be inferred from the content. It cannot be empty or null. Header row position (start Row): The row number (1-based) of the column name. End Row: The row number of the last row of valid data; Column information: The header name (table header) and column number (col Num, such as A / B / C) for each column, excluding the serial number column; Note the distinction between the header row and the table header row. A single sheet may contain multiple tables.

[0040] S2.2 Parallel Parsing: The table structure parsing task for multiple sheets is executed in parallel based on virtual threads. Each sheet corresponds to an independent AI call task, and the parsing results are summarized into a global table list. The virtual thread model allows each AI call to have its own independent thread, eliminating the need for thread pool management overhead, making it suitable for I / O-intensive LLM call scenarios.

[0041] S2.3 JSON Response Cleaning: A multi-level degradation strategy is used to extract JSON from the text returned by the AI. Level 1 cleaning: Use regular expression matching to strip out Markdown code block wrappers and remove code block tags that AI may have added; Secondary cleaning: If there is no code block wrapping, locate the first JSON starting character ([ or {) to extract the valid content, and remove any prefix text and explanations that the AI ​​may have added; Three-level degradation: When parsing fails, a retry mechanism is triggered, with a maximum of N retries. The retry interval increases incrementally (2 seconds × number of retries). After all retries fail, an exception is thrown and the task status is set to error.

[0042] This multi-level degradation strategy is applicable to AI response cleaning in the S3 table-level mapping, S4 field-level mapping, and S6 data type conversion stages.

[0043] Step S3: AI global table-level mapping.

[0044] Perform a global match between all table structures identified in step S2 and the system metadata table: S3.1 Metadata Table Loading: Loads all metadata table definitions from the database, including table code, table name, and table description. Simultaneously, it batch loads the field name list for each metadata table (avoiding N+1 queries) and appends it to the metadata table information.

[0045] S3.2 Global Mapping Call: Sends all identified table structure information and candidate metadata table information to the large language model at once. System prompts instruct the AI ​​to: determine the business meaning of the table based on the semantics of the table title and column names; select the best matching target table from the candidate metadata table, and output the table code and confidence (0.0-1.0); if no suitable match is found, set the table code to null and the confidence to 0.0; specifically instruct the AI ​​to pay attention to cases where the table is divided into segments but is actually a single table, and make a global decision.

[0046] The confidence level assessment uses a multi-factor weighted model: ,in, The semantic similarity between the table title and the metadata table name. For semantic matching coverage of the column name set and field name set, Rate the data type compatibility. The model assigns a score to the semantic alignment of field descriptions, with w1~w4 representing the weights of each factor. This model is applicable to both table-level mapping (S3) and field-level mapping (S4) confidence outputs.

[0047] S3.3 Global Decision Advantage: Compared with table-by-table independent mapping, global mapping ensures that AI can see all table information in a single call, and can identify the same business tables across sheets or delimiters, avoiding local optima problems.

[0048] Step S4: AI field-level mapping.

[0049] For each table that has matched the target table, perform a field-level mapping between its column headers and the field definitions of the target metadata table: S4.1 Metadata Field Loading: Loads all field definitions (MetaColumn) of the target metadata table from the database, including the column code, column name, column description, and field type.

[0050] S4.2 Field Mapping Invocation: The table column information (table header + col Num) and target field definitions are sent to the large language model. System prompts instruct the AI ​​to: match the most suitable target field based on the semantics of the table column names; consider factors such as the business meaning, data type, and field description of the fields; one table column can only be mapped to one target field; for unmatched fields, the table column code is set to null, and confidence is set to 0.0; the confidence assessment uses the same multi-factor weighted model as S3.2, where... It focuses on semantic matching coverage of column names and field names. It focuses on the compatibility between table data types and target field types.

[0051] S4.3 Parallel Mapping: Field-level mapping tasks for multiple tables are executed in parallel, with each table corresponding to an independent AI invocation task.

[0052] S4.4 User Hint: Allows users to input supplementary hints during the confirmation phase. When AI field mapping is re-executed after switching target tables, the hint content takes precedence over AI's autonomous judgment, such as business context information like filenames containing patent numbers. This supplementary hint is injected as part of the user supplementary hint layer of the structured hint framework.

[0053] S4.5 Mapping Result Persistence: Save the table-level mapping (Recognize Table Mapping) and column-level mapping (Recognize Column Mapping) results to the database, update the task status to "Mapped", and wait for user confirmation.

[0054] Step S5: User confirms mapping.

[0055] The front end displays the AI ​​mapping results (table level + column level), and users can perform the following operations: S5.1 Confirm Mapping: After the user confirms that the AI ​​mapping result is correct, the data entry process is triggered.

[0056] S5.2 Modify Target Table: Users can switch the target table for a specific table. The system calls the `remap Columns` method to physically delete the original column mapping records, reconstruct the Table Structure using the original table's column header information, call AI to perform field-level mapping, generate new column mapping records, and persist them. S5.3 Modify field mapping: Users can manually modify the target field corresponding to a table column, or delete unnecessary column mappings (soft delete, del Flag).

[0057] S5.4 Supplementary Prompts: Users can enter supplementary prompts to assist AI in remapping.

[0058] Step S6: Data parsing and data import.

[0059] After the user confirms the mapping, the system asynchronously performs data parsing and database entry: S6.1 File Reload: Download the original spreadsheet file from OSS, create two Workbook instances: a Processing Workbook (for reading data) and an Error Workbook (for writing error annotations, preserving the original formatting including merged cells). Perform merged cell expansion on all affected sheets (only once to avoid duplicate expansions).

[0060] S6.2 Read Data by Mapped Range: Based on the start / end row of the table-level mapping and the col Num of the column-level mapping, read data from the processing workbook for a specified range of rows and columns. Specific steps: Convert table column numbers (e.g., A / AB) to 0-based column subscripts; Assemble a Map structure based on column mapping relationships, connecting field codes to cell values. Perform JSON security cleanup on cell values, including trimming leading and trailing whitespace, removing control characters such as NUL (while retaining tab / cr / lf), and uniformly escaping backslashes; Skip rows where the entire target column is empty and header rows where all values ​​are the same.

[0061] S6.3 Batch AI Data Conversion: Data is split into batches of 10 rows, and the large language model is called to perform data type conversion for each batch. The system prompts define strict type conversion rules: INT / BIGINT → number (without quotes); DECIMAL → number (without quotes); DATE → yyyy-MM-dd format string; DATETIME → yyyy-MM-dd HH:mm:ss format string; SMALLINT → 1 (yes) or 0 (no), supports inferring Boolean values ​​from explanatory text such as XXX; TEXT / VARCHAR → Preserve original value (trim leading and trailing whitespace); Date parsing supports Java Date.toString() formats such as Tue Jan 29 00:00:00 CST 2024, and numeric formats such as 2024.0; Output null for empty values, confidence = 1.0 (confidence). AI needs to remove non-data rows such as title rows and explanation rows mixed in with the data.

[0062] S6.4 Confidence Verification: AI outputs a confidence value (0.0-1.0) for each row of data. Rows with a confidence value below 0.7 are marked as erroneous and not entered into the database. AI also outputs a "low Confidence Reason" field describing the reason for the low confidence (using Chinese field names, including original data values, and without complex formatting). Confidence assessment is based on the multi-factor weighted model described in Section 3.1, focusing on two factors during the data type conversion stage: S_type (data type compatibility) and S_desc (semantic alignment of field descriptions).

[0063] S6.5 Dictionary Label Conversion: For SELECT type fields (dictionary fields), the system automatically converts the dictionary labels (such as high-tech enterprises) output by AI into dictionary values ​​(such as 1). If the conversion fails, an error message is generated.

[0064] S6.6 Data Ingestion: The dynamic data service is invoked to write the converted JSON records into the target dynamic table. Each record is automatically injected with a primary key prefixed with the task ID (taskId - sequence number) and a tenant identifier (company Id) to achieve multi-tenant data isolation and ensure that data from different tenants is not visible to each other. Upsert mode is supported. Configure the parsing of unique key groups (unique Cols) from the metadata table, in the format of a JSON array of column name combinations; Before data entry, check if a record already exists by using the unique key group; If a single record is matched, an UPDATE statement is executed; if multiple different records are matched, a conflict exception is thrown; otherwise, an INSERT statement is executed.

[0065] S6.7 Database Error Translation: Inbound exceptions are translated into human-readable information using the Research Error Adapter. This includes parsing unique key conflict error messages, extracting conflict field codes and values, loading Chinese field names, translating dictionary values ​​into labels, generating user-friendly prompts such as "Data already exists," "Unique fields [Company Name, Patent Number] are duplicated," and "Current value: Hangzhou XX Technology, GR20243300ZZZZ," followed by "Please check and resubmit."

[0066] S6.8 Virtual Thread Parallel Processing: Data insertion tasks for multiple tables are executed in parallel using virtual threads. Within each table, AI transformation is called in parallel in batches (10 rows / batch). The virtual thread model achieves high concurrency and low overhead batch LLM calls, avoiding resource contention and context switching overhead of traditional thread pools. Small batches (less than half the batch size) are automatically merged into the previous batch to avoid fragmented calls.

[0067] Step S7: Error labeling and write-back.

[0068] S7.1 Error Style Creation: Create two error cell styles in the Error Workbook: the Exception Cell Style (red background + white bold text, used to mark the error cell itself) and the Error Description Style (yellow background + red bold text, used to append error descriptions), both of which are set to automatic line wrapping.

[0069] S7.2 Error Cell Location and Writing: For each error row, the system analyzes the cause of the exception and extracts the exception field. Combining this with the column location information (col Num) from the table-level mapping, it locates the specific cell where the error occurred. An exception cell style (red background + bold white text) is applied directly to the error cell. An error description cell is appended after the last content cell in that row. The error message is truncated to 300 characters, written to the cell, and the error description style is applied. The error column width is set to 160×256 (approximately 160 characters wide).

[0070] S7.3 Clear Successful Rows: Remove rows that were successfully entered from the error Workbook. First, delete all cell contents of the target row (remove Row), then move all rows below that row up one row (shift Rows). Process all successful rows in descending order of row number to avoid misalignment of row numbers after shifting.

[0071] S7.4 Remove Sheets Without Errors: Iterate through all sheets in the error workbook. If a sheet has no error lines, remove the entire sheet from the error file (remove Sheet At), reducing file size.

[0072] S7.5 Independent Upload of Error Files: If an error line exists, the error workbook is written to a temporary file and uploaded to the OSS research-result directory with the original file name. An independent file ID is returned, and the file is stored in isolation from the original file.

[0073] S7.6 Task Result Record: The `results` field in the task table records the following JSON data: `{success Count: N, error Count: M, error FileId: xxx}`. The front-end uses this information to display a summary of the import results and a download link for the error file. The task status is updated to "Completed".

[0074] Step S8: Reset and re-import the task.

[0075] S8.1 Reset Task: When a user performs a reset operation on a completed task, the system verifies that the task status is completed or importing, deletes all imported data for the task (deleting data by matching the task ID prefix), resets the task status to mapped (preserving the mapping relationship), clears the task results, and deletes the asynchronous task record.

[0076] S8.2 Re-import: When the user performs the import operation again on the reset task, the system downloads the original file (not the error file) from OSS again, performs data parsing and database entry again according to the existing mapping relationship, and generates a new error file (if any).

[0077] S8.3 Upsert Mode: Users can choose to enable upsert mode. When re-importing, the system checks whether the data already exists based on the unique key group. If it matches, it updates; otherwise, it inserts, avoiding conflicts caused by repeated imports.

[0078] S8.4 Re-import after Mapping Modification: Users can modify table-level or column-level mappings and then re-import. When modifying table-level mappings, switching the target table and calling `remap Columns` to re-execute AI field mappings; when modifying column-level mappings, the column mapping records are updated directly. After modification and re-execution of the import, the system parses the data according to the new mapping relationship.

[0079] The system adopts a layered prompt keyword architecture, dividing prompt keywords into three layers according to their responsibilities and lifecycles: (1) System Prompt Layer: Solidifies role definitions, task rules, and output format constraints, enabling cross-task reuse. Separate system prompts are defined for each of the three stages: table structure parsing, table-level mapping, and field-level mapping. This clarifies the AI's role (e.g., you are a professional table structure analysis expert), task rules (e.g., column name matching rules, output JSON format constraints), and constraints (e.g., a table column can only map to one target field), ensuring the consistency and controllability of the AI's output.

[0080] (2) Context Prompt Layer: Dynamically injects runtime data such as metadata definitions, candidate table information, and historical mapping results. Each time the AI ​​is invoked, the context information such as the current task's metadata table definitions, field definitions, and identified table structures is serialized into JSON and injected into the user prompt, enabling the AI ​​to make decisions based on specific business contexts rather than relying on general knowledge.

[0081] (3) User-provided hints (Hint): Business context hints entered by the user during the confirmation phase have higher priority than AI's autonomous judgment. For example, if the user enters a file name that is a patent number or other business rule, the system will inject it into the field-level mapping hint words to guide the AI ​​to match according to the user's intent. When switching target tables, the hint words are passed in along with the remapping request to ensure that the user's intent continues to be effective in the secondary matching.

[0082] The three-layered prompts are assembled in each AI call in the order of system prompt layer → context prompt layer → user supplementary prompt layer. The upper layer constrains the lower layer, and the user supplementary prompt layer can cover the AI's autonomous judgment, realizing human-machine collaborative mapping decision-making.

[0083] In summary, by adopting the above solution, users do not need to pre-select or configure import templates. The system automatically understands the column header semantics of Excel files of any format through a large language model and completes the mapping to the target database fields, significantly improving ease of use and versatility. Adding a new business table only requires configuring the metadata table and metadata field definitions; no modification to the import code or template is required. The Excel-to-database mapping process is decomposed into four independent stages (structure parsing → table-level mapping → field-level mapping → data type conversion). Each stage can be executed and validated independently, and a manual confirmation step is provided after field-level mapping. Compared to a one-time end-to-end extraction, the multi-stage solution allows users to correct mapping errors before data entry, preventing erroneous data from polluting the database. All identified tables are globally matched with all candidate metadata tables in a single AI call. The AI ​​can identify the same business tables across sheets or delimiters, avoiding the local optimum problem of independent table-by-table mapping. Leveraging the semantic understanding capabilities of a large language model, it can handle non-standard data formats, including Java date formats such as Tue Jan 29 00:00:00 CST 2024, numeric formats such as 2024.0, and explanatory Boolean values ​​such as "Yes" and "Because XXX". Each row of data is accompanied by a confidence score, and rows with low confidence scores are automatically intercepted and the reason for the error is indicated.

[0084] The preprocessing stage of this invention automatically expands merged cells, giving each cell an independent value, thus solving the column reading misalignment problem caused by complex formats such as multi-level headers and cross-column headings. Table structure parsing of multiple sheets, field-level mapping of multiple tables, and batch data conversion of multiple tables can all be performed in parallel, significantly reducing the processing time for large files. The system analyzes the cause of errors, extracts abnormal fields, and, combined with column location information within the table, precisely marks the error to the erroneous cell itself (red background + bold white text), and appends an error description (yellow background + bold red text) to the end of the row. Successful rows are removed from the error file, and error-free sheets are removed entirely. After downloading the error file, users only see the rows that need correction and the specific erroneous cells. After correction, the task can be reset and re-imported, forming a complete error correction loop. Automatic database table creation (DDL) is driven by metadata tables and metadata field definitions. Adding new business tables does not require manual table creation; the system automatically executes DDL statements to create physical tables and fields based on the metadata definitions.

[0085] The following three examples further illustrate the solution of the present invention: Example 1: Importing patent data from Excel.

[0086] The user uploads an Excel file containing three sheets: Sheet1 (Patents) contains columns such as patent name, patent type, and application date; Sheet2 (Qualifications) contains columns such as company name, certification type, and certification time, and Sheet2 contains two tables separated by blank lines; Sheet3 is a hidden sheet.

[0087] S1 Preprocessing: Read the workbook, filter and hide Sheet3, expand merged cells in Sheet1 and Sheet2, extract data row by row and categorize (skip blank rows, compress header rows, and retain data rows).

[0088] S2 Structure Parsing: AI is called in parallel to parse the table structure of Sheet1 and Sheet2. Sheet1 identifies 1 table, and Sheet2 identifies 2 tables (separated by blank lines), for a total of 3 table structures.

[0089] S3 Table-level Mapping: Send the three table structures to the AI ​​at once. The AI ​​makes a global decision to map the Sheet1 table to the patent metadata table, and both Sheet2 tables to the qualification metadata table (the AI ​​recognizes that they are cross-page splits of the same table).

[0090] S4 Field Mapping: Parallel calls to AI to perform field-level mapping on 3 matched tables, mapping Excel column headers to metadata field codes.

[0091] S5 User Confirmation: The front end displays the mapping results. Users can confirm that the results are correct and then trigger the data entry. Alternatively, they can manually switch the target table (after switching, the system will automatically re-execute the AI ​​field mapping), or fill in supplementary prompts for secondary AI matching (the prompts take precedence over the AI's autonomous judgment). After modifying the field mapping or deleting unnecessary columns, the data entry can be triggered.

[0092] S6 data ingestion: Download raw files from OSS, read data according to the mapped range, call AI in batches for type conversion, and write the data to a dynamic table after the confidence check is passed.

[0093] S7 Error Labeling: If a row fails to be imported (e.g., data format error, unique key conflict), the system analyzes the reason for the error, extracts the abnormal field, and marks the error style (red background + white bold text) on the cell with the column location information. An error description is appended to the end of the row. Successful rows are cleared, and the error file is uploaded to OSS.

[0094] S8 Reset and Re-import: After the user downloads and corrects the faulty file, the reset task is re-imported.

[0095] Example 2: Importing a multi-level header Excel file containing merged cells.

[0096] A user uploads an Excel file with a multi-level header structure. Row 1 contains the company name, certification information, certification information, and certification information (B1 / C1 / D1 contain certification information from merged cells). Row 2 contains (empty) certification type, certification time, and certificate number.

[0097] Expand the merged cells B1 / C1 / D1 and fill all three cells with certification information. AI identifies row 1 as the group header row and row 2 as the table header row. start Row=3 (data start row). AI maps the enterprise name to enterprise_name, certification type to qualification_type, certification time to qualification_date, and certificate number to certificate_no. The subsequent data entry process is the same as in Example 1.

[0098] Example 3: Low-confidence data interception.

[0099] A user uploaded an Excel file, in which the value of the application date column in a certain row is March 19th.

[0100] AI Data Conversion: The AI ​​outputs March 19th for the `apply_date` field in this row, with confidence=0.0. The lowConfidence reason is: the application date should be in date format, but the current value is March 19th, making it impossible to determine the year. Confidence Verification: Since confidence < 0.7, this row is marked as an error and will not be entered into the database. Error Labeling: The system analyzes the reason for the anomaly (low confidence), extracts the abnormal field (`apply_date`), and, based on column location information, adds an anomaly style (red background + bold white text) to the cell containing the application date. An error description is appended to the end of the row: "The recognition result has low confidence; please check the data. The application date should be in date format, but Chinese characters appear in the content to be processed (yellow background + bold red text)." See Example 1 for other processes.

[0101] Figure 2 This paper presents a non-template-based table intelligent recognition and database entry system, including: The file preprocessing unit 201 is configured to receive a table file, perform preprocessing, and obtain a processed Sheet dataset. Table parsing unit 202 is configured to: call the large model to parse the processed Sheet dataset to generate a set of table structures; The global table matching unit 203 is configured to: globally match the set of table structures with the metadata table to obtain a table-level mapping result; The field mapping unit 204 is configured to generate field-level mapping results in parallel from the table-level mapping results and display them for manual confirmation. Data entry unit 205 is configured to: call the large model in batches to complete the data type conversion based on the confirmed field-level mapping results, and write the data into the dynamic data table in update insertion mode after confidence verification; Error generation unit 206 is configured to generate and archive error files that annotate the cells with errors for data that failed to be validated.

[0102] It is understood that the aforementioned units can be individually or entirely merged into one or more other units, or some of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of the present invention. The aforementioned units are based on logical functional division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of the present invention, the system may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0103] According to another embodiment of the present invention, the system described in this embodiment can be constructed by running a computer program (including program code) capable of performing the steps involved in the corresponding method of the present invention on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM). The computer program can be recorded on, for example, a computer-readable recording medium, loaded into the aforementioned computing device through the computer-readable recording medium, and run therein.

[0104] Figure 3 A computer device is shown, which includes a processor 301, a communication interface 302, and a computer-readable storage medium 303. The processor 301, communication interface 302, and computer-readable storage medium 303 can be connected via a bus or other means.

[0105] The communication interface 302 is used to receive and send data. The computer-readable storage medium 303 can be stored in the memory of the electronic device. The computer-readable storage medium 303 is used to store computer programs, which include program instructions. The processor 301 is used to execute the program instructions stored in the computer-readable storage medium 303.

[0106] The processor 301 is the computing and control core of the electronic device. It is suitable for implementing one or more instructions, specifically for loading and executing one or more instructions to achieve the corresponding method flow or corresponding function.

[0107] The processor 301 is configured to perform the following process: Receive a table file, perform preprocessing to obtain the processed Sheet dataset; The large model is invoked to parse the processed Sheet dataset and generate a set of table structures. The table structure set is globally matched with the metadata table to obtain the table-level mapping result; The table-level mapping results are used to generate field-level mapping results in parallel and displayed for manual confirmation. Based on the confirmed field-level mapping results, the large model is called in batches to complete the data type conversion. After confidence verification, the data is written to the dynamic data table using the update and insert mode. Generate and archive error files with labeled incorrect cells for data that failed to be validated.

[0108] This invention also provides a computer-readable storage medium, which is a memory device in an electronic device for storing programs and data. It is understood that the computer-readable storage medium here may include both built-in storage media in the electronic device and extended storage media supported by the electronic device. The computer-readable storage medium provides storage space for storing the processing system of the electronic device.

[0109] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory; alternatively, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0110] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes the one or more instructions stored in the computer-readable storage medium to perform the following process: Receive a table file, perform preprocessing to obtain the processed Sheet dataset; The large model is invoked to parse the processed Sheet dataset and generate a set of table structures. The table structure set is globally matched with the metadata table to obtain the table-level mapping result; The table-level mapping results are used to generate field-level mapping results in parallel and displayed for manual confirmation. Based on the confirmed field-level mapping results, the large model is called in batches to complete the data type conversion. After confidence verification, the data is written to the dynamic data table using the update and insert mode. Generate and archive error files with labeled incorrect cells for data that failed to be validated.

[0111] The present invention also provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the following process: Receive a table file, perform preprocessing to obtain the processed Sheet dataset; The large model is invoked to parse the processed Sheet dataset and generate a set of table structures. The table structure set is globally matched with the metadata table to obtain the table-level mapping result; The table-level mapping results are used to generate field-level mapping results in parallel and displayed for manual confirmation. Based on the confirmed field-level mapping results, the large model is called in batches to complete the data type conversion. After confidence verification, the data is written to the dynamic data table using the update and insert mode. Generate and archive error files with labeled incorrect cells for data that failed to be validated.

[0112] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this invention can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0113] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of the present invention is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic cable, digital cable) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive), etc.

[0114] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A non-templated table intelligent identification warehousing method, characterized in that, Includes the following processes: Receive a table file, perform preprocessing to obtain the processed Sheet dataset; The large model is invoked to parse the processed Sheet dataset and generate a set of table structures. The table structure set is globally matched with the metadata table to obtain the table-level mapping result; The table-level mapping results are used to generate field-level mapping results in parallel and displayed for manual confirmation. Based on the confirmed field-level mapping results, the large model is called in batches to complete the data type conversion. After confidence verification, the data is written to the dynamic data table using the update and insert mode. Generate and archive error files with labeled incorrect cells for data that failed to be validated.

2. The non-template-based table intelligent recognition and database entry method as described in claim 1, characterized in that, The preprocessing operations for obtaining the processed Sheet dataset include: Filter the hidden worksheets to obtain the retained worksheets, expand all merged cells in the retained worksheets, distinguish blank rows, header rows, and data rows, extract cell text, and integrate and output the processed Sheet dataset.

3. The non-template-based table intelligent recognition and database entry method as described in claim 1, characterized in that, When generating the table structure set, multiple worksheets in the processed Sheet dataset are executed in parallel using virtual threads to call the large model; a multi-level cleaning and retry mechanism is performed on the text returned by the large model; and the standardized table structures are extracted and summarized to form the table structure set.

4. The non-template-based table intelligent recognition and database entry method as described in claim 1, characterized in that, The confidence score associated with the generated table-level mapping result is obtained by weighted calculation across four dimensions: table title semantic similarity, column name matching coverage, data type compatibility score, and metadata description alignment score. The confidence score is obtained by dynamically adjusting the weights of each dimension according to the business stage.

5. The non-template-based table intelligent recognition and database entry method as described in claim 1, characterized in that, When generating the field-level mapping result, supplementary prompt text input by the user is injected into the large model as prompt words; the supplementary prompt text has a higher priority than the large model's autonomous semantic matching logic; the field-level mapping result is regenerated after switching the target metadata table.

6. The non-template-based table intelligent recognition and database entry method as described in claim 1, characterized in that, The large model prompts adopt a three-layer architecture, consisting of a system prompt layer with fixed rules, a context prompt layer that loads metadata information, and a user-added prompt layer that carries the supplementary prompt text. The content of the three layers is concatenated and input into the large model to complete the mapping calculation.

7. The non-template-based table intelligent recognition and database entry method as described in claim 1, characterized in that, The data type conversion operation includes: converting cell text into five standard formats—integer, decimal, date, boolean, and text—based on the field definitions of the metadata table; and appending the confidence level and the reason for low confidence to each conversion result.

8. The non-template-based table intelligent recognition and database entry method as described in claim 1, characterized in that, The update / insert mode execution logic includes: reading the unique key combination configured in the metadata table; querying the matching records in the dynamic data table; executing the update operation if the match is successful, and executing the insert operation if the match fails; synchronously injecting the multi-tenant identifier into each written data to complete data isolation; The steps to generate the error file include: creating two types of cell styles: abnormal cell style and error description style; locating the cells corresponding to the failed data and filling them with the abnormal cell style; appending the low confidence reason text to the end of the row; clearing the rows of successful data and then exporting the error file.

9. The non-template-based table intelligent recognition and database entry method as described in claim 1, characterized in that, It also includes a task reset operation: receiving a reset command; deleting the data already written to the dynamic data table according to the task identifier; retaining the table-level mapping result and the field-level mapping result; and re-executing the entire data entry process after resetting the task status.

10. A non-templated table intelligent recognition warehousing system, characterized in that, include: The file preprocessing unit is configured to receive a table file, perform preprocessing, and obtain a processed Sheet dataset. The table parsing unit is configured to: call the large model to parse the processed Sheet dataset and generate a set of table structures; The global table matching unit is configured to globally match the set of table structures with the metadata table to obtain a table-level mapping result. The field mapping unit is configured to generate field-level mapping results in parallel from the table-level mapping results and display them for manual confirmation. The data entry unit is configured to: call the large model in batches to complete the data type conversion based on the confirmed field-level mapping results, and write the data into the dynamic data table using the update insertion mode after confidence verification; The error generation unit is configured to generate and archive error files that annotate the cells with errors for data that failed to be validated.

Images