Big model-based target file table header and multi-file table header similarity identification and associated data filling method

Abstract

The application discloses a target file table header and multi-file table header similarity identification and associated data filling method based on a large model, and relates to the technical field of data processing and artificial intelligence. The application analyzes the table to be filled and the multi-data set table, identifies the table header structure and column name information through a large model, and generates standardized column names; based on a two-stage method of vector retrieval and large model semantic fine correction, semantic matching of target column names and multi-source column names is realized, and a mapping relationship is constructed; the multi-data source is searched row by row in combination with the primary key field, and automatic filling or user decision triggering is realized through a deduplication and dynamic judgment mechanism; finally, the result is output in the original format and the completed data is marked, and the efficiency and stability are improved through parallel processing and a dynamic vector library. The method can improve the accuracy and automation level of complex table data filling.

Description

Technical Field

[0001] This invention relates to the fields of data processing and artificial intelligence technology, and in particular to a method for identifying the similarity between the header of a target file and the header of multiple files based on a large model, and for filling in associated data. Background Technology

[0002] With the continuous improvement of the informatization level of various business systems, data management and exchange based on Excel spreadsheets remains one of the most common data processing methods in actual work. In practical application scenarios, users often need to match and populate information from multiple datasets into a target table according to specific primary key fields—such as ID card number, user ID, etc.—to achieve data integration and information completion.

[0003] In existing technologies, this type of data population work mainly relies on the following methods:

[0004] One method is manual processing. Users manually open multiple dataset tables one by one, identify the corresponding fields, and then search for and populate the target data row by row based on the primary key. This method is tedious, inefficient, and prone to omissions or errors when dealing with large amounts of data or inconsistent field naming, making it difficult to meet the needs of batch data processing.

[0005] The second method is automation based on rule or keyword matching. This involves pre-setting field mapping rules or matching based on keywords to achieve partial autofill. However, this type of method typically relies on exact matching of field names or simple string similarity, and cannot effectively handle situations where fields are semantically similar but have different expressions. For example, the semantic correspondence between fields such as "ID number," "document number," and "ID code" is difficult to accurately identify.

[0006] Thirdly, methods based on single vector similarity or simple machine learning models can improve the flexibility of field matching to some extent, but the following problems still exist: First, relying solely on semantic similarity lacks a comprehensive judgment of business scenarios, data types, and keyword information, which can easily lead to mismatches; second, it cannot distinguish the different meanings of the same field name in different business scenarios; third, when faced with complex Excel structures (such as multi-row headers, merged cells, non-standard formats, etc.), it is difficult to accurately parse the header structure and field information.

[0007] In addition, existing technologies for Excel parsing typically use fixed rules, such as defaulting to column names in the first row, which lacks adaptability to complex table structures. For cases with multi-level headers, cross-row column names, and comment rows, the parsing accuracy is low.

[0008] During the data population process, existing solutions typically lack effective mechanisms for handling multi-source matching results. When multiple candidate data exist, they cannot provide reasonable decision support; at the same time, they lack identification and traceability mechanisms for the populated data, making it difficult for users to verify the data source and the accuracy of the population results. Summary of the Invention

[0009] Therefore, how to achieve semantic-level accurate matching of column names in complex Excel structures and achieve efficient, accurate and controllable data filling in multi-data source environments has become an urgent technical problem to be solved.

[0010] To achieve the above objectives, this invention proposes a method for identifying the similarity between the header of a target file and the headers of multiple files based on a large model, and for filling in associated data, comprising the following steps:

[0011] S1. File Reception and Integrity Verification: Receive the file path (filePath) of the table to be filled, the data path (dataPaths) of the dataset table folder, and the primary key column name (knownField) of the table to be filled submitted by the user. Verify in turn whether filePath is a valid file, whether knownField exists in the table to be filled, and whether dataPaths is a valid folder. After successful verification, proceed to the next processing step.

[0012] S2. Excel File Parsing: For all files in the table to be filled and the dataset table folder, the corresponding dedicated parsing engine is automatically called based on the file type. The first 10 rows of data in each file are extracted as core example samples. The large model is called to perform deep semantic reasoning on the example samples, automatically locating the row index of the table header area, the row where the column name field is located, and the starting row of the real business data. The structured parsing results are output in standardized JSON format. Based on the parsing results, the column name information is read, and the split column names distributed across multiple rows are semantically associated and fused to generate standardized column names. The standardized column names of the dataset table are stored in a vector database.

[0013] S3. Column Correlation Analysis: For all blank column names in the table to be filled, firstly, high-similarity candidate column names are retrieved from the vector database through vector retrieval, and then a large model is called to perform deep semantic refinement on the candidate column names to generate a mapping relationship between the column names of the table to be filled and the column names of the dataset table.

[0014] S4. Data Integration: Based on the correspondence mapping generated in step S3, retrieve the target data that matches the primary key value in the dataset table row by row; if only one matching data is found in a blank cell, directly fill and mark the cell; if multiple matching data are found, generate a candidate data list and trigger user decision-making.

[0015] S5. Output Results: Write the filled data to the target file in the original file format, highlight the cells automatically completed by the system in red, and return the processing status and output file path.

[0016] The semantic association fusion processing of split column names across multiple rows described in step S2 specifically involves: extracting multimodal features of hierarchical column names, including text features (word segmentation vectors of column name content), structural features (parent-child cell relationships in merged cells), and visual features (border thickness, font bold / centering, row height); and calculating hierarchical row and column names. With header row and column names Semantic similarity:

[0017] ;

[0018] in Semantic vectors generated for large models; similarity normalized to association weights. :

[0019] ;

[0020] By integrating multimodal features and association weights, standardized column names are generated:

[0021] ;

[0022] in For structural constraint indicator functions, only if and The value is 1 when the cells are in the same merged cell range and have continuous borders; the final standardized column names are output in the format of "parent column name-child column name".

[0023] Step S2 also includes non-standard format adaptive cleaning processing: Let the starting row index of the merged cells be... The end row index is The starting column index is The index of the end column is The cell content is Then each sub-cell after splitting ( , The inherited content is as follows:

[0024] ;

[0025] The filtering rules for blank lines and comment lines are as follows: Let the first line be... The number of non-empty cells in the row is The total number of columns is The border style is marked as The condition for filtering this row is:

[0026] ;

[0027] in A threshold is set for blank rows; simultaneously, based on the large model, the probability of each row serving as the table header is output, along with the format feature vector of each row. ( To decide whether to bold, To determine whether it is centered, (font size deviation value) and semantic feature vector (The 1024-dimensional field name saliency vector generated based on the large model) is concatenated and input into a multilayer perceptron, outputting the probability of the header row:

[0028] ;

[0029] Select Maximum and The rows that are used as header rows must have a positioning accuracy of no less than 98%.

[0030] The two-stage column correlation analysis of vector retrieval and large model refinement mentioned in step S3 specifically includes:

[0031] First stage: Vector coarse screening: Generating column name vectors to be filled based on the large model. Vector of column names in the dataset Introducing One-Hot encoding for business scenario tags Generate contextualized semantic vectors:

[0032] ;

[0033] in The scene weight matrix has a vector dimension of 1024; cosine similarity is calculated as follows:

[0034] ;

[0035] Sort by similarity in descending order, retrieve the top 20 candidate column names based on FAISS, an open-source "vector retrieval library"; the retrieval time is less than 10ms.

[0036] The second stage involves refining the large model: inputting the 20 candidate column names into the large model, and combining them with the primary key column names. Calculate the overall matching score based on the business context. :

[0037] ;

[0038] in , For semantic similarity, For data type matching degree, The column name with the highest overall matching score and that meets the matching threshold is output as the corresponding relationship mapping result.

[0039] The calculation methods for each component in the overall matching score are as follows:

[0040] Data type matching degree :

[0041] ;

[0042] Business keyword matching degree :set up The number of keywords contained is , The number of keywords contained is The two have a total of [number] keywords. ,but:

[0043] ;

[0044] Dynamic matching threshold: The matching threshold is dynamically adjusted based on the column name type. If the column name to be filled If it is the primary key column (identified by user annotation or automatic recognition from a large model, with an accuracy rate of no less than 95%), then... If it is a regular column, then The matching result is determined as follows:

[0045] ;

[0046] The parsing of the dataset table file in step S2 is performed in parallel using a thread pool, and the column correlation analysis of the column names to be populated in step S3 is performed in parallel using a thread pool with a maximum of 50 worker threads. This parallel processing uses a Future mechanism to asynchronously submit tasks. The method monitors the task completion status. If a single-column correlation analysis task fails, it will not affect the parallel processing of other columns. After recording the error information, the failed task will continue to execute the remaining tasks.

[0047] The data integration described in step S4 specifically involves: using the primary key column name The corresponding primary key value of the current row Using the search criteria, the target data is retrieved column by column in the dataset table according to the correspondence mapping generated in step S3; the search results for each blank cell are then processed. Perform the following processing:

[0048] when When the unique matching value is entered directly into the current cell, the cell is added to the set of cells to be highlighted in red, in the format of "column name + Excel row number". ;

[0049] when When this happens, the cell is encapsulated as ambiguous data and recorded as follows:

[0050] ;

[0051] Then add the ambiguous data set. Do not perform the data filling operation yet. Wait until the user selects and confirms from the candidate data list before performing the data filling and adding the data. ;

[0052] when If the current cell is empty, record it in the unmatched log.

[0053] After all cells have been processed, The candidate data is deduplicated and filtered for null values ​​before being filled.

[0054] The red marking process described in step S5 specifically involves: after integration, the system... The cell identifiers (column name + Excel row number) recorded in the collection are marked in red only for target cells automatically completed by the system, while maintaining the original file's formatting specifications and structural integrity. Cells completed by user manual decision-making are also marked in red. Cells with existing data in the original file are not formatted in any way. After the red highlighting is completed, the results are written to the output file in the original file format, and a success status and output file path are returned. If an exception occurs during the processing, a failure status and specific reason for failure are returned.

[0055] The vector database described in step S3 adopts a dynamic vector library mechanism: a unique set identifier is generated for each processing task, and an independent vector set is created with the identifier. The standardized column name vectors of all dataset tables for the current task are written into this independent set. After the task is completed, the set is released to achieve vector data isolation in multi-task concurrent scenarios and avoid column name vector pollution between different tasks. When the column name vector is written, the file table name and Sheet name to which the column name belongs are recorded synchronously as metadata for the large model fine-tuning stage to extract the association context information between the table name and the Sheet.

[0056] When the large model performs semantic reasoning on the example samples in step S2, the output format of the large model is constrained by the structured output parser. The output fields include: the number of header rows. (Indicates the number of consecutive unstructured description rows starting from the first row, an integer), column name row number (Indicates the number of rows of field names immediately following the table header, an integer), number of sample data rows. (Number of rows that meet the criteria of including format placeholders or having explanatory content, integer) and the number of actual data rows. (The number of rows containing specific business data after the sample data, an integer); the sum of the above four types of row counts equals the total number of sample rows; the system automatically calculates the row offset of each type of data based on the above structured output, and uses the data region parsing mechanism based on the row offset to preferentially read the data of the corresponding row range by pandas, so as to achieve accurate segmentation of the header area, column name area, sample data area and real data area without manually specifying any row numbers. pandas is one of the most commonly used data processing libraries in Python.

[0057] Compared with existing technologies, the method for similarity recognition and associated data filling of target file headers and multiple file headers based on large models provided by this invention has the following beneficial effects:

[0058] Firstly, in terms of column name identification and matching, this invention employs a two-stage column name relevance analysis mechanism that combines large-scale model semantic understanding, vector retrieval for coarse screening, and multi-dimensional feature refinement. This mechanism effectively identifies column names that are semantically similar but have different expressions. Furthermore, it combines data type matching degree and business keyword matching degree for comprehensive judgment, significantly improving the accuracy of column name matching and reducing the false matching rate.

[0059] Secondly, in terms of parsing complex Excel structures, this invention introduces multimodal features, including text features, structural features, and visual features, and combines them with a large model to make probability judgments on the header area. This enables adaptive parsing of multi-row headers, merged cells, and non-standard format tables, eliminating the need for manual specification of header row positions and significantly improving parsing accuracy and versatility.

[0060] Furthermore, in terms of multi-source data integration, this invention achieves unified integration of multi-source data through a primary key-based row-by-row retrieval mechanism, combined with deduplication and candidate data analysis; it automatically fills in when a unique matching result exists, improving processing efficiency; and when multiple candidate results exist, it introduces a user decision-making mechanism to achieve controllable selection of key data, balancing automation and accuracy.

[0061] Furthermore, this invention uses a dynamic matching threshold mechanism to adaptively adjust the matching criteria based on the column name type (primary key column or ordinary column), which improves the flexibility of ordinary field matching while ensuring high accuracy of primary key matching, thereby achieving a balance between accuracy and coverage in different scenarios.

[0062] Furthermore, this invention accelerates the file parsing and column correlation analysis process through a parallel processing mechanism (thread pool), significantly improving overall processing efficiency while ensuring the accuracy of the results, making it suitable for large-scale data processing scenarios.

[0063] Meanwhile, by constructing a dynamic vector database, this invention achieves data isolation between different tasks, avoids vector data pollution, and improves the stability and reliability of the system in multi-task concurrent scenarios.

[0064] Finally, in terms of result output, this invention marks the automatically filled and manually filled cells in red and retains the original file format, thereby making the filling results visible and traceable, facilitating user verification of results, and improving the usability and reliability of the system.

[0065] In summary, this invention significantly outperforms existing technologies in terms of data processing efficiency, column name matching accuracy, adaptability to complex tables, and result controllability, and has good application value. Attached Figure Description

[0066] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. Obviously, the drawings described below are merely some embodiments of the present invention, and those skilled in the art can obtain other drawings based on these drawings without creative effort. In the drawings:

[0067] Figure 1 This is an overall flowchart of a method for identifying the similarity between the header of a target file and the header of multiple files, and for filling in associated data, based on a large model. Detailed Implementation

[0068] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0069] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0070] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0071] Example 1

[0072] This embodiment provides a method for identifying the similarity of target file headers and multi-file headers based on a large model, and for filling in associated data, applied to a government data completion scenario. A unit needs to batch fill in the mobile phone numbers of its personnel, which are scattered across multiple Excel spreadsheets (a dataset spreadsheet folder, containing 8 files and a total of 23 sheets), into a personnel information summary table (the table to be filled). The primary key column of the table to be filled is named "ID Number," and the target column to be filled is "Mobile Number." The dataset spreadsheets are provided by different departments, resulting in inconsistent column names (such as "ID Code," "Document Number," "Contact Number," "Mobile Phone Number," etc.), and non-standard formats such as hierarchical headers, merged cells, and comment rows exist.

[0073] This embodiment uses the above scenario as a consistent example to provide a detailed explanation of the complete execution process of the five core steps.

[0074] I. Document Receipt and Integrity Verification;

[0075] The system receives three basic pieces of information submitted by the user: the path to the file containing the form to be filled. (Pointing to the Personnel Information Summary Table.xlsx), Dataset Table Folder Path (Point to the dataset folder containing 8 Excel files) and the primary key column names of the table to be populated. (The value is "ID number").

[0076] The system performs the following verifications in sequence: First, verify The first step is to check if the file points to a valid Excel file. In this example, the file exists and is in a valid format, so the verification passes. The second step is to parse the table to be filled and verify if the "ID Number" column exists in the table's column names. In this example, the second row of the table to be filled is the column name row, and the "ID Number" column exists, so the verification passes. The third step is to verify... The system checks whether the folder contains at least one valid Excel file. In this example, the folder contains eight .xlsx files, so the verification passes. After all three verifications pass, the system proceeds to the file parsing process.

[0077] II. Excel file parsing module;

[0078] (I) Parsing the table to be filled; The system calls the xlsx dedicated parsing engine to read the first 10 rows of data from all sheets of the personnel information summary table (this example contains only 1 sheet) as a sample. The sample content is as follows: The first row is "2024 Personnel Information Summary Table of a Certain City Human Resources Management Center" (cells are merged across columns, no borders, bold and centered); the second row is the column name row, containing fields such as "Serial Number", "Name", "ID Number", "Department", "Mobile Number", and "Position" (with borders, bold font); the third to fifth rows are sample data rows, with the mobile number column containing placeholders in the format "138xxxxxx67"; the sixth row onwards are the actual data rows.

[0079] The system feeds example samples into the large model, using a structured output parser to constrain the output format. The large model performs deep semantic reasoning on the example samples, comprehensively analyzing the format feature vector and semantic feature vector of each row, and calculating the probability of each row serving as the table header:

[0080] ;

[0081] Feature vector in row 1 format (Bold, centered, larger font), semantic feature vector The field names have low significance (they are organization names rather than field names), combined with the borderless feature ( ),determination The second line serves as a filter for the remarks line; the second line is a formatted feature vector. semantic feature vector Field names are highly salient and have borders ( ),determination This is positioned as the header row. The large model outputs structured JSON, indicating 1 row of headers, 1 row of column names, 3 rows of sample data, and 245 rows of actual data.

[0082] The system uses JSON results, and the row offsets are calculated as follows: the header area is row 0 (index), the column name area is row 1, the sample data area is rows 2 to 4, and the actual data area starts from row 5. Pandas is used to read the corresponding row ranges and extract the column name list as ["Serial Number", "Name", "ID Number", "Department", "Mobile Number", "Position"]. The "Mobile Number" column has 245 rows of data that are all empty, and is confirmed as the target column to be filled.

[0083] (ii) Parallel parsing of dataset tables; the system submits 8 dataset tables to a thread pool (maximum number of worker threads 50) for parallel processing. Taking one of the dataset tables, "Social Security Personnel Information Table.xlsx" from the social security department, as an example, this file contains 2 sheets. We will use Sheet1 ("Employed Personnel") as an example for explanation.

[0084] The system reads the first 10 rows of data from Sheet1. The sample structure is relatively complex: the first two rows are hierarchical headers ("User Basic Information" merged cells span the three columns "Name", "ID Number" and "Contact Information", "User Basic Information" is the parent column name, and "Contact Information" is further split into two sub-columns "Mobile Phone" and "Landline Phone"), the third row is a single row of column name supplementary descriptions, and the fourth row onwards contains the actual data.

[0085] The large model outputs JSON, with column names in rows 0 to 2 (row 3 column names).

[0086] The system enters the hierarchical column name semantic association fusion processing stage. For the sub-columns "Mobile Phone" and "Landline Phone" under the parent column "Contact Information," multimodal features are extracted: In text features, the semantic similarity of the word segmentation vectors for "Contact Information" and "Mobile Phone" and "Landline Phone" is calculated separately; in structural features, it is confirmed that the merged cell area of ​​"Contact Information" covers both the "Mobile Phone" and "Landline Phone" columns. Visual features confirm that the borders of the parent column name and the child column name are continuous.

[0087] Calculate semantic similarity and normalize it into association weights:

[0088] ;

[0089] ;

[0090] Normalized association weights , Combined with structural constraint indicator functions Both generate standardized column names, namely "Contact Information - Mobile Phone" and "Contact Information - Landline".

[0091] At the same time, the system recognized that the sheet needed to split merged cells, and performed splitting processing on the parent merged cell "User Basic Information". After splitting, each child cell inherited the content: ( , ).

[0092] Finally, the standardized column names extracted from Sheet1 are ["User Basic Information - Name", "User Basic Information - ID Number", "User Basic Information - Contact Information - Mobile Phone", "User Basic Information - Contact Information - Landline", "Social Security Number", "Payment Status"], and written to the vector database (set identifier). The metadata record file name is "Social Security Personnel Information Table.xlsx" and the sheet name is "Employed Personnel". (This is the UUID for this task).

[0093] After the parallel parsing of all 23 sheets in 8 datasets was completed, the vector database stored 186 standardized column name vectors and their metadata.

[0094] III. Column Correlation Analysis Module;

[0095] The system initiates column correlation analysis for the two columns to be added to the table ("ID number" as the primary key column and "phone number" as the target column), and submits them to the thread pool for parallel execution. The analysis process for the target column "phone number" is described in detail below.

[0096] (I) First stage: Vector coarse screening. The system first introduces business scenario tags. In this embodiment, the scenario is human resources management, and the scenario is One-Hot coding. (Personnel scenario dimension set to 1). Generate a contextualized semantic vector for "phone number":

[0097] ;

[0098] Based on FAISS, a similarity retrieval was performed on 186 standardized column name vectors in the vector database, recalling the top 20 candidate column names with the highest similarity. The retrieval time was 7ms. The top 5 candidate column names and their cosine similarities are as follows: "Contact Information - Mobile Phone" (0.91), "Mobile Phone" (0.89), "Contact Number" (0.84), "Contact Information - Landline" (0.76), and "Telephone Number" (0.74). The similarities of the remaining candidate column names ranged from 0.51 to 0.70.

[0099] (II) Second Stage: Large Model Refinement. The 20 candidate column names, along with the primary key column name "ID Number" and the business context (human resources management scenario, the table to be filled is a summary table of personnel information), are input into the large model for deep semantic refinement, and the comprehensive matching score of each candidate column name is calculated:

[0100] ;

[0101] In this embodiment, the weight is taken as , , For example, using "Contact Information - Mobile Phone":

[0102] semantic similarity ;

[0103] Data type matching: The data type of the column to be populated, "Mobile Number," is a string-numeric type, and the actual data in the "Contact Information - Mobile Phone" column is also an 11-digit string-numeric type; the data types are consistent. ;

[0104] Business keyword matching: The column name to be populated, "Mobile Number", contains the keywords ["Mobile", "Number"]. The candidate column name "Contact Information - Mobile Phones" contains the keywords ["Contact", "Mobile", "Telephone"]. The large model determines that "mobile phone" and "telephone" are semantically equivalent in a personnel scenario, based on the total number of keywords. ;

[0105] ;

[0106] Overall Match Score:

[0107] ;

[0108] Taking "Contact Information - Landline" as an example: The large model determines that "Mobile Number" and "Landline" are not equivalent in terms of business semantics. , Overall matching score:

[0109] ;

[0110] Let the column to be analyzed in the target table be... Its candidate column set is For any candidate column Its comprehensive matching score is defined as:

[0111] ;

[0112] in, Indicates the semantic similarity between the target column and the candidate column. Indicates the degree of data type matching. Indicates the business keyword matching degree, and satisfies: ;

[0113] For different types of target columns, the system employs a dynamic threshold determination mechanism. Let the target column type determination function be... ;

[0114] The corresponding threshold is defined as:

[0115] ;

[0116] When the candidate column satisfies: Then determine the candidate column. For target column Valid matching column names; otherwise, exclude them. For the target column "phone number", since it is not a primary key column, we have: ;

[0117] Therefore, its dynamic threshold is: ;

[0118] For the candidate column "Contact Information - Mobile Phone", the overall matching score is: Since 0.835 ≥ 0.7, "Contact Information - Mobile Phone" is determined to be the matching column name for "Mobile Number".

[0119] For the candidate column "Contact Information - Landline", the overall matching score is:

[0120] ;

[0121] Since 0.620 < 0.7, "Contact Information - Landline" is not a matching column name for "Mobile Number" and is therefore excluded from the candidate set. For the other candidate column, "Mobile Phone," the overall matching score is: Since 0.801 ≥ 0.7, "Mobile Phone" is also determined to be the matching column name for "Mobile Number". When the matching column comes from different dataset tables, the system retains it as a valid mapping of the same type from different sources.

[0122] For the primary key column "ID Number", the system performs the same matching analysis; since "ID Number" is a primary key column, we have: Type("ID Number")="Primary Key Column"; the corresponding dynamic threshold is: Thresh("ID Number")=0.9; for the candidate column "ID Number", its comprehensive matching score is: Since 0.93 ≥ 0.9, "ID Number" is determined to be a valid matching column name for "ID Number".

[0123] For the candidate column "ID card code", the overall matching score is: Since 0.91 ≥ 0.9, "ID card code" is determined to be a valid matching column name for "ID card number".

[0124] Let the set of all columns to be analyzed in the target table be defined. After column name matching is completed, the system generates a mapping of the correspondence between the target column and the source column path set: ;

[0125] in A set of three-element path strings representing "source file name / worksheet name / column name". It represents its power set.

[0126] In this embodiment, the generated mapping relationship can be represented as Map(phone number):

[0127] ;

[0128] This contains a total of 7 source mapping relationships. Similarly, the mapping relationship for the primary key column "ID number" can be represented as Map(ID number):

[0129] ;

[0130] Thus, the system completes the mapping construction from the target table's columns to be populated and the primary key columns to the source columns of multiple dataset tables, providing input conditions for subsequent data retrieval and automatic population based on primary key values.

[0131] IV. Data Integration Module;

[0132] Let the set of actual data behaviors in the table to be filled be defined. Let the first The primary key value of the row is Let the source dataset table set be... ;

[0133] For each source dataset table Let the primary key mapping column between it and the target table be... The data to be extracted is listed as follows: Then, for the first... The set of candidate search results for a row is defined as follows:

[0134] ;

[0135] in, Indicates the source table name, Indicates the source worksheet name. Indicates matching line number, Indicates the source data column name, This represents the retrieved data values; the candidate search results are deduplicated based on the result values, and the set of unique values ​​after deduplication is defined as:

[0136] ;

[0137] The number of candidate data after deduplication is: ;

[0138] when ; then it means the first If a row has only one valid candidate value, then the unique value is denoted as: The system will then directly perform the fill operation. ;

[0139] Add the corresponding filled cell identifier to the set to be marked: ;in, Indicates the first The actual row number of each piece of real data in Excel;

[0140] With primary key value For example, the system searches through the seven source dataset tables one by one to obtain the candidate set:

[0141] ;

[0142] Of the remaining matching sources, no results were found for 2 datasets, and while results were found for 3 datasets, all of them contained the phone number 1381234xxxx; therefore, after deduplication, the results are as follows: ;

[0143] then: The system will directly: And update the set to be labeled as follows: ;when: ; then it means the first If a row has multiple distinct candidate values, the system triggers an ambiguous decision-making mechanism and constructs an ambiguous candidate encapsulation result: ;

[0144] in This is the set of candidate source-value pairs after deduplication; taking the actual Excel row number 30 corresponding to the real data in row 28 as an example, let its primary key value be: The system retrieved two different candidate mobile phone numbers: Therefore:

[0145] ;

[0146] In this case, the system does not perform automatic filling, but instead constructs ambiguous data records. The format is: ("ID number", "3701021985072xxxxx", 30,"Mobile number", [(Personnel file.xlsx, 1398765xxxx), (Medical insurance information.xlsx, 1369876xxxx)]);

[0147] And mark that row and column as pending manual confirmation, that is: ;

[0148] The system skips the current fill of the cell and continues processing other rows; after the user completes all ambiguous data selections, a second round of fill is performed on the ambiguous cells; if the user selects the first... The final confirmed value of the row is ;

[0149] Then we have: And update simultaneously: In this embodiment, out of 245 rows of real data, 242 rows satisfy the following: ;

[0150] And it was directly filled by the system; there are 3 rows that meet the requirements: ;

[0151] This triggers the ambiguity decision-making mechanism; after the user confirms the three ambiguous records, the system executes the second round of supplementary filling, and finally completes the filling of all 245 rows of target data.

[0152] After all data integration is completed, the system will proceed according to... The set (containing 245 cell identifiers) is used to highlight the output file: The xlsx write engine is invoked, and while maintaining all original file formatting (merged cells, font, row height, borders, etc.), only the formatting is highlighted. The 245 cells in the record are marked in red. Cells that already contain data in the original file (columns such as "Serial Number", "Name", "ID Number", "Department", "Position", etc.) will not be formatted in any way.

[0153] The system returns the processing results: success status, output file path, and processing summary (a total of 245 rows were processed, 242 rows were automatically filled, 3 rows were filled by user decision, and 0 rows were not matched). Users can open the output file and see that all filled cells in the "Phone Number" column are displayed in red, facilitating quick verification of the source and accuracy of the fill results.

[0154] Example 2

[0155] Based on the method framework described in Example 1, this embodiment focuses on verifying the parsing robustness in a non-standard hierarchical header scenario, and provides a detailed explanation of the parsing process for a more complex dataset table (Construction Project Personnel Information Table.xlsx).

[0156] In this embodiment, the "Construction Project Personnel Information Table.xlsx" contains 3 sheets, taking Sheet2 ("Labor Personnel") as an example. The structure of this sheet is as follows: The first row is a blank row (all cells are empty, no border); the second row is the report name row "Construction Project Labor Personnel Information Registration Table (2024)" (spanning all columns, merged cells, no border, bold font); the third and fourth rows are the hierarchical column name rows, the third row contains the merged cells "Basic Personnel Information" (spanning columns 1 to 5) and "Contact and Identification Information" (spanning columns 6 to 9), the fourth row is the corresponding sub-column names: "Name", "Gender", "Age", "Place of Origin", "Occupation", "Mobile Phone Number", "ID Card Number", "Emergency Contact Person", "Emergency Contact Number"; the fifth and sixth rows are sample data (the mobile phone number column content is a placeholder in the format "138xxxxxxxx"); the seventh row onwards is the actual data, a total of 412 rows.

[0157] (a) Adaptive cleaning of non-standard formats;

[0158] The system reads the first 10 lines of data and performs filtering and judgment on blank lines and comments for each line.

[0159] Row 1: Number of non-empty cells Total number of columns Blank rate (Not satisfied), but (No border) Filter this line.

[0160] Row 2: Number of non-empty cells (Merged cells count as 1), Blank rate , (No border, report name row) Filter this line.

[0161] Row 3: Number of non-empty cells (Two merged cells) , (with border) Keep this line. Similarly, keep line 4.

[0162] Merge cells and split: split the third row "Basic Personnel Information" ( After splitting, the content of each cell in row 3, columns 0 to 4 will inherit from "Basic Personnel Information" and "Contact and Document Information". Similarly, split the process.

[0163] (ii) Integration of header row positioning and hierarchical column names;

[0164] The large model performs inference on the filtered and split sample data, outputting JSON. The system performs hierarchical association fusion on the sub-column name "Mobile Number" in the 4th row. Multimodal features are extracted: semantic similarity between "Mobile Number" and "Contact and ID Information" in the text features.

[0165] ;

[0166] The structural features confirm that the "Mobile Number" cell is within the merged cell area of ​​"Contact and Document Information". ), continuous borders; normalized association weights (Normalized together with other sub-column names in the same region). Because the structural constraints satisfy ( Generate a standardized column name "Contact and Document Information - Mobile Number" for "ID Card Number"; generate a standardized column name "Contact and Document Information - ID Card Number" for "ID Card Number".

[0167] Finally, Sheet2 extracts standardized columns named ["Personnel Basic Information - Name", "Personnel Basic Information - Gender", "Personnel Basic Information - Age", "Personnel Basic Information - Place of Origin", "Personnel Basic Information - Occupation", "Contact and Identification Information - Mobile Phone Number", "Contact and Identification Information - ID Card Number", "Contact and Identification Information - Emergency Contact Person", "Contact and Identification Information - Emergency Contact Number"], writes them to the vector database, and attaches metadata "Construction Project Personnel Information Table.xlsx / Labor Personnel".

[0168] (iii) Contextualized semantic matching in column correlation analysis;

[0169] During the column relevance analysis phase, when the system performed a coarse vector screening for the column "Mobile Number" to be filled, the cosine similarity of the contextualized semantic vector of "Contact and ID Information - Mobile Number" was 0.88, ranking 3rd among the 20 candidate column names recalled.

[0170] During the large model refinement stage, the comprehensive matching score is calculated:

[0171] ;

[0172] Data type matching: Both are 11-digit string numbers. ;

[0173] Business keyword matching: The column to be populated, "Mobile Number", contains keywords ["Mobile", "Number"]. Candidate column names with keywords ["Contact", "ID", "Information", "Mobile", "Number"] The keywords are ["mobile phone", "number"] ("number" and "number" are considered synonyms by the large model). ;

[0174] ;

[0175] Overall Match Score:

[0176] ;

[0177] A score of 0.854, which is greater than or equal to the threshold of 0.7 for ordinary columns, was determined to be a matching column name and included in the corresponding relationship mapping. In the subsequent data integration stage, it successfully supplemented the 245 rows of personnel data in Example 1 with mobile phone numbers from construction project archives, increasing the overall matching coverage from 92% to 97%.

[0178] This embodiment verifies the robust handling capability of the WxExcelAnalysis module of the present invention for complex non-standard formats such as blank rows, merged headers, and hierarchical column names, as well as the ability of the WxCorrelationAnalysis module to accurately identify semantic correspondences under cross-scenario standardized column names ("Contact and Document Information - Mobile Phone Number"). The two modules work together to effectively improve the accuracy of cross-table field matching and the coverage of data filling.

[0179] Example 3

[0180] This embodiment introduces a multi-source consistency confidence correction mechanism and a user decision feedback adaptive weight correction mechanism;

[0181] First, the overall matching score in step S3. It only reflects the semantic matching degree between a single data column and the column to be filled, without making use of redundant information between multiple dataset tables. If multiple sources independently return the same value, the confidence of that value is obviously higher than that of a value returned by only a single source. However, the existing algorithm treats both equally, resulting in an uneven distribution of confidence in the filling decision.

[0182] Second, the weights in the overall matching score in step S3. , , If the process remains fixed throughout the same task, it cannot be corrected in real time based on the user's actual selection behavior in the S4 ambiguity decision-making stage. Each time the user confirms a piece of ambiguous data, it essentially provides a label of "which candidate column name should have a higher comprehensive matching score". If this information is included in the weight correction, it can make the matching of subsequent columns more accurate.

[0183] This embodiment uses a scenario example of a task to batch complete the contact information of property owners at a municipal real estate registration center (the table to be filled contains 1200 industry master data entries, and the dataset table folder contains 12 source files) to provide a detailed explanation of the complete execution process of the two correction mechanisms mentioned above.

[0184] In step S4, the data integration stage, after the system performs a search on each blank cell across all matching sources, it additionally records the number of sources and value distribution of the original search results before deduplication and null value filtering.

[0185] Suppose a blank cell is in Results were retrieved from a total of [number] matching sources (excluding sources where no matching rows were found), including the return value. The number of sources is The multi-source consistency score of this value is defined as:

[0186] ;

[0187] when When all sources consistently return the same value, the confidence level is highest; when... At that time, multiple sources of conflict exist, and the confidence level decreases.

[0188] In the original filling decision rule ( Fill directly. Based on triggering user decisions, a consistency score correction is introduced to form a three-level decision-making mechanism:

[0189] In the original filling decision rule ( Fill directly. Based on triggering user decisions, a consistency score correction is introduced to form a three-level decision-making mechanism:

[0190] in The default value is set as the consistency confidence threshold. ; This is the set of candidate values ​​after deduplication. The candidate value with the highest consistency score.

[0191] ;

[0192] in The default value is set as the consistency confidence threshold. ; This is the set of candidate values ​​after deduplication. The candidate value with the highest consistency score.

[0193] The second level, "high-confidence direct filling," is the core enhancement of this mechanism: when multiple distinct values ​​exist after deduplication ( However, the multi-source consistency score of a certain value When more than 60% of the sources return the same value, the system determines that the value is sufficiently reliable, and without triggering a user decision, directly fills in and adds the cell. The set also includes a consistency confidence field in the output structure for user verification.

[0194] Taking the primary key value (real estate certificate number) "XXXXXXX20230156" as an example, where the "owner's mobile phone number" is filled in, the system retrieved the primary key value from 9 out of 12 sources. The original search results are as follows: 7 sources returned "1375678xxx2", 1 source returned "1375678xxx3" (suspected typo, the last digit is off by 1), and 1 source returned a null value (the "mobile phone number" column of this source has no data for this row).

[0195] After deduplication and null value filtering, "1375678xxx2","1375678xxx3" , According to the original rules, this should trigger a user decision. The number of valid sources after introducing consistency correction... (Excluding one source that returns a null value);

[0196] ;

[0197] ;

[0198] The condition is determined to meet the second-level "high-confidence direct filling" condition. "1375678xxxx" is directly filled in, and the output structure includes additional fields to allow users to verify that the filled value comes from consistent support from 7 out of 8 sources.

[0199] In this embodiment, out of 1200 rows of data, 312 cells triggered the second-level high-confidence direct fill, accounting for 26%. Compared with the pure rule decision in embodiment one, this reduces the number of ambiguous user decisions by about 71%, while retaining the data traceability basis through the consistency confidence field.

[0200] In step S4, each time the user makes a selection decision regarding ambiguous data, an implicit annotation is essentially provided regarding the quality of the candidate column name matching: the user selects the source. Instead This means that in the current business scenario, The overall matching score should be higher than The system extracts information from this decision-making process. and The differences in scores for each sub-item were analyzed and their weights were determined. , , In which dimension, increasing this will... This serves as the direction for weight adjustment.

[0201] Suppose the user completes the first Secondary ambiguous decision-making, choosing a source Refuse source Extract the difference between the scores of each item:

[0202] ;

[0203] ;

[0204] ;

[0205] Define the correction direction vector For the current weight vector Perform gradient direction correction:

[0206] ;

[0207] in To correct the step size, take ; This represents the score difference between the two under the current weights. To trigger the conditional indicator function, only when the score difference is less than a threshold. Correction is only triggered when the current weight fails to effectively distinguish the source of a user's choice from their rejection, thus avoiding unnecessary disturbances when the weight can already distinguish correctly. Guarantee after correction And all weights are non-negative.

[0208] The initial task weight in this embodiment is the same as that in embodiment 1. .

[0209] First user decision (row 83, "Owner's Mobile Number", candidate values ​​after deduplication come from source A and source B): The user selects source A ("Contact Number" column, value "1582345xxx9"), and rejects source B ("Alternative Contact Information" column, value "1582345xxx8", last digit difference 1). Extract scores for each item:

[0210] Source A: , , ;

[0211] Source B: , , ;

[0212] Score difference under current weights:

[0213] ;

[0214] Trigger the correction condition and calculate the correction direction vector: ;

[0215] Perform weight adjustment:

[0216] ;

[0217] Revised The (business keyword matching weight) has slightly improved, reflecting that keyword matching has a stronger ability to distinguish the source of ambiguity in the current business scenario.

[0218] After a total of 7 user decisions in this example task, the weights finally converge to:

[0219] ;

[0220] Business keyword matching weight The increase from 0.2 to 0.245 reflects that the distinguishability of keywords such as "mobile phone number", "contact number" and "mobile" in real estate registration business scenarios is significantly higher than in general scenarios.

[0221] The system will weight The comprehensive matching score calculation is applied to the column ("Emergency Contact Phone Number" column) for which subsequent column relevance analysis has not yet been completed in the task of this embodiment. Taking the two candidate column names "Emergency Contact Phone Number" (one source) and "Backup Mobile Phone" (another source) as examples:

[0222] Using initial weights :

[0223] ;

[0224] ;

[0225] The difference is 0.050, indicating weak differentiation.

[0226] Use the corrected weights :

[0227] ;

[0228] ;

[0229] The difference of 0.065 represents a 30% improvement in discrimination, making the "emergency contact number" more advantageous over "backup phone" in the overall matching score, and the matching results are more in line with the actual semantic needs of the current business scenario.

[0230] The complete processing results of the 1200 industry master data in this embodiment are as follows: The multi-source consistency correction mechanism reduced the number of ambiguous decisions requiring user intervention from the estimated 53 under the rules in Embodiment 1 to 15 (7 of which also triggered weight correction as user decision feedback samples); the adaptive weight correction mechanism improved the discrimination of the column correlation analysis comprehensive matching score of the "emergency contact phone number" column by 30%, avoiding 2 potential column name mismatches; the final data filling coverage rate was 98.7% (1184 / 1200 rows), with 15 manual decisions, reducing user operations by approximately 72% compared to the pure rule method.

[0231] Both mechanisms enhance the comprehensive matching score in step S3 and the filling decision in step S4 without modifying the existing core algorithm framework by introducing correction terms and feedback loops, thus verifying the scalability of the invention in scenarios with redundant data from multiple sources and user feedback learning scenarios.

Claims

1. A method for identifying the similarity between the header of a target file and the headers of multiple files based on a large model, and for filling in associated data, characterized in that... Includes the following steps: S1. File reception and integrity verification. After successful verification, proceed to the next processing step. S2, Excel file parsing: For the files in the table to be filled and the dataset table folder, the corresponding parsing engine is automatically called based on the file type, the data of each file is extracted as the core example sample, the large model is called to perform deep semantic reasoning on the example sample, the row index of the table header area, the row where the column name field is located and the starting row of the real business data are automatically located, and the structured parsing results are output in standardized JSON format; Based on the parsing results, the column name information is read, and semantic association and fusion processing is performed on the split column names distributed across multiple rows; The semantic association and fusion processing of the split column names distributed across multiple rows specifically involves: extracting multimodal features of the hierarchical column names, including text features, structural features, and visual features; Calculate the row and column names of the hierarchy With header row and column names Semantic similarity: ； in Semantic vectors generated for large models; similarity normalized to association weights. : ； By integrating multimodal features and association weights, standardized column names are generated: ； in For structural constraint indicator functions, only if and The value is 1 when the columns are in the same merged cell range and have continuous borders; the final standardized column names are output in the format "parent column name-child column name"; Store the standardized column names of the dataset table into a vector database; S3. Column Correlation Analysis: For all blank column names in the table to be filled, high similarity candidate column names are retrieved from the vector database through vector retrieval. Then, a large model is called to perform deep semantic refinement on the candidate column names, generating a mapping relationship between the column names in the table to be filled and the column names in the dataset table. The two-stage column correlation analysis of vector retrieval and large model refinement specifically includes: First stage: Vector coarse screening: Generating column name vectors to be filled based on the large model. Vector of column names in the dataset Introduce business scenario tags to generate scenario-based semantic vectors: ； in This is the scene weight matrix; Calculate cosine similarity: ； Candidate column names are retrieved by sorting them in descending order of similarity and using a vector similarity retrieval method. The second stage involves refining the large model: inputting the candidate column names into the large model, combining the primary key column names and business context, and calculating the comprehensive matching score. : ； in , For semantic similarity, For data type matching degree, The column name with the highest overall matching score and that meets the matching threshold is output as the corresponding relationship mapping result. The calculation methods for each component in the overall matching score are as follows: Data type matching degree : ； Business keyword matching degree :set up The number of keywords contained is , The number of keywords contained is The two have a total of [number] keywords. ,but: ； Dynamic matching threshold: The matching threshold is dynamically adjusted based on the column name type. If the column name to be filled If it is a primary key column, then ; If it is a regular column, then The matching result is determined as follows: ； S4. Data Integration: Based on the correspondence mapping generated in step S3, retrieve the target data in the dataset table that matches the primary key value; If only one matching data is found in a blank cell, the cell is filled and marked directly; if multiple matching data are found, a candidate data list is generated and the user's decision is triggered. S5. Output Results: Write the filled data to the target file in the original file format, mark the cells automatically completed by the system, and return the processing status and output file path.

2. The method for similarity recognition and associated data filling of target file headers and multiple file headers based on a large model according to claim 1, characterized in that, Step S2 also includes non-standard format adaptive cleaning processing: Let the starting row index of the merged cells be... The end row index is The starting column index is The index of the end column is The cell content is Then each sub-cell after splitting ( , The inherited content is as follows: ； The filtering rules for blank lines and comment lines are as follows: Let the first line be... The number of non-empty cells in the row is The total number of columns is The border style is marked as The condition for filtering this row is: ； in A threshold is set for blank rows; simultaneously, based on the large model, the probability of each row serving as the table header is output, along with the format feature vector of each row. The data is concatenated with the semantic feature vector and then input into a multilayer perceptron, which outputs the probability of the header row.

3. The method for similarity recognition and associated data filling of target file headers and multiple file headers based on a large model according to claim 1, characterized in that, The parsing of the dataset table file in step S2 is performed in parallel by a thread pool, and the column correlation analysis of the column names to be filled in step S3 is performed in parallel by a thread pool; the parallel processing adopts the Future mechanism to submit tasks asynchronously.

4. The method for similarity recognition and associated data filling of target file headers and multiple file headers based on a large model according to claim 1, characterized in that, The data integration described in step S4 specifically involves: using the primary key column name The corresponding primary key value of the current row Using the search criteria, the target data is retrieved column by column in the dataset table according to the correspondence mapping generated in step S3; Search results for each blank cell Perform the following processing: when When a unique matching value is entered directly into the current cell, the cell is added to the cell collection in the format of "column name + Excel row number". ; when When this happens, the cell is encapsulated as ambiguous data and recorded as follows: ； Then add the ambiguous data set. Do not perform the data filling operation yet. Wait until the user selects and confirms from the candidate data list before performing the data filling and adding the data. ; when If the current cell is empty, record it in the unmatched log. After all cells have been processed, The candidate data is deduplicated and filtered for null values ​​before being filled.

5. The method for similarity recognition and associated data filling of target file headers and multiple file headers based on a large model according to claim 1, characterized in that, The marking process described in step S5 includes: after the data integration is completed, the system adds distinguishable identifiers to the target cells filled automatically by the system or confirmed by the user, based on the set of cells to be marked, so as to distinguish the filled data from the original data while keeping the original table structure and the format of non-target cells unchanged; the distinguishable identifiers include at least one: font style change, color change, cell attribute identifier or additional metadata information, so as to achieve the traceability of the filling results.

6. The method for similarity recognition and associated data filling of target file headers and multiple file headers based on a large model according to claim 1, characterized in that, The vector database described in step S3 adopts a dynamic vector library mechanism: a unique set identifier is generated for each processing task, an independent vector set is created with the identifier, and the standardized column name vectors of all dataset tables for the current task are written into the independent vector set; the set is released after the task is completed, realizing vector data isolation in multi-task concurrent scenarios and avoiding column name vector pollution between different tasks; when the column name vector is written, the file table name and Sheet name to which the column name belongs are recorded synchronously as metadata, which is used to extract the association context information between the table name and the Sheet during the large model refinement stage.

7. The method for similarity recognition and associated data filling of target file headers and multi-file headers based on a large model according to claim 1, characterized in that, When the large model performs semantic reasoning on the example samples in step S2, the output format of the large model is constrained by the structured output parser. The output fields include: the number of header rows, the number of column name rows, the number of sample data rows, and the number of real data rows. The sum of the number of rows in the above four categories equals the total number of rows in the example sample. Based on the above structured output, the system automatically calculates the row offset of each type of data and reads the data in the corresponding row range using a data region parsing mechanism based on the row offset, thereby achieving the separation of the header area, column name area, example data area and real data area.

Images