Intelligent matching method, system and device for table header field based on semantic index segmentation

Abstract

The application discloses a kind of table header field intelligent pair of mark methods and systems based on semantic index segmentation, it is related to data management technical field, the present application is to the problems of prior art in table header field standardization relies on artificial, low efficiency and it is difficult to handle semantic ambiguity and non-Chinese field, an intelligent pair of mark scheme of fusion semantic understanding and vector matching is presented.The method generates initial word library by large language model decomposition table header field, constructs historical pair of mark database to realize fast matching;Semantics is divided using fine-tuning BERT model, and vector library retrieval and Top-K recommendation are combined to complete accurate pair of mark;For non-Chinese field, regular rules and semantic logic are used to realize code set identification.The system includes word library construction, historical matching, semantic segmentation and vector matching modules, and the device includes processor, memory and other hardware units.The application realizes the automatic standardization processing of table header field, and improves the data management efficiency.

Description

Technical Field

[0001] This invention belongs to the field of data governance technology, and in particular relates to a method and system for intelligent alignment of table header fields based on semantic index segmentation. Background Technology

[0002] In the field of data governance, the standardization of header fields is a crucial step in achieving data integration, exchange, and analysis. Currently, the alignment of data elements with qualifiers primarily relies on manual work, requiring maintenance personnel to compare and input each data item against the standard fields one by one. This method is not only inefficient, especially when dealing with large-scale data sources, but also prone to omissions and errors due to the sheer number of fields or naming differences, severely impacting the accuracy and efficiency of data standardization.

[0003] Currently, some systems attempt to achieve semi-automatic matching of data fields using rule engines, but this approach has significant limitations in practical applications. When dealing with large-scale, complex, and heterogeneous data sources, these solutions often fail to achieve ideal intelligent matching results due to a lack of deep semantic understanding and context-aware mechanisms. The matching accuracy of traditional methods drops significantly, especially when dealing with data items that are ambiguous, ambiguous, or synonymous. Existing technologies struggle to effectively identify and process fields lacking Chinese names or using abbreviated headers. These problems are particularly pronounced in industries with high data accuracy requirements, such as finance and healthcare, severely hindering the efficient flow and value extraction of data.

[0004] The root cause of these problems lies in the inability of traditional methods to effectively parse the semantic structure of header fields. Data items are typically composed of qualifiers and data elements, but current technologies lack the ability to automatically decompose composite fields into these two components. Furthermore, the construction and maintenance of standard thesauruses also face challenges, failing to cover variations in terminology across different business scenarios. In addition, for the identification of special types such as code set fields, existing solutions largely rely on fixed rule matching, lacking flexibility and scalability. These technical deficiencies have resulted in data governance efforts remaining in a state of high cost and low efficiency for a long time, failing to meet the needs of modern enterprises for data standardization and intelligence.

[0005] In practical applications, these problems manifest as long data integration cycles, heavy manual verification workloads, and inconsistent standardization results. Particularly in scenarios involving cross-system data migration or heterogeneous data source integration, the standardization of header fields often becomes a critical bottleneck in project implementation. Therefore, there is an urgent need for a novel solution that combines semantic understanding, intelligent matching, and historical data to improve the automation and accuracy of data field standardization and meet the demands of modern data governance. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention proposes a method and system for intelligent alignment of table header fields based on semantic index segmentation, thereby resolving the issues present in the prior art.

[0007] Firstly, to achieve the above objectives, this invention provides a method for intelligent alignment of header fields based on semantic index segmentation, comprising the following steps:

[0008] Collect data from the header fields, decompose it into pairs of qualifiers and data elements using a large language model, and generate an initial lexicon.

[0009] The initial vocabulary is stored in a relational database to build a historical benchmarking module;

[0010] For Chinese header fields, retrieve matching standard terms from historical databases based on field name and table name similarity;

[0011] If historical matching fails, the semantic segmentation model is used to segment the field into limiting words and data elements, and candidate combinations are retrieved through the vector library to output Top-K recommendation results.

[0012] For non-Chinese fields, sample values ​​are extracted and regular expression rules and semantic logic are applied to map them into standard data elements;

[0013] The completed benchmarking data will be stored in the historical database to update the thesaurus.

[0014] Optionally, the process of generating the initial lexicon includes:

[0015] Extract data from the header fields and convert it to JSON format;

[0016] The input large language model is decomposed into pairs of qualifiers and data elements;

[0017] High-quality term pairs are filtered and categorized for storage through semantic validation.

[0018] Optionally, the construction process of the historical benchmarking module includes:

[0019] The approved qualifiers and data elements are stored in a relational database by field name and table name;

[0020] Create full-text and vector indexes for field names, table names, qualifiers, and data elements.

[0021] Optionally, the semantic segmentation model's operation includes:

[0022] Fine-tuning the BERT model to identify field qualifiers and data element boundaries;

[0023] The segmentation results are output using the BIO tagging system;

[0024] Combine regular expression rule base to verify edge cases.

[0025] Optionally, the process of retrieving candidate combinations from the vector library includes:

[0026] Encode the segmented qualifiers and data elements into vectors;

[0027] Search for similar candidates in the Faiss database and calculate the cosine similarity.

[0028] Expand the candidate set by synonym replacement and generate Top-K recommendations.

[0029] Optionally, the mapping process for the non-Chinese fields includes:

[0030] Extract sample values ​​from the fields and perform normalization processing;

[0031] Apply regular expression rules sequentially to match date, number, and email format;

[0032] Perform semantic logic rule verification on unmatched samples.

[0033] Secondly, the present invention also provides a semantic index-based intelligent alignment system for table header fields, used to implement a semantic index-based intelligent alignment method for table header fields, the system comprising:

[0034] The lexicon building module is used to collect data from the header fields and decompose it into pairs of qualifying words and data elements using a large language model to generate an initial lexicon.

[0035] The historical benchmarking database module is used to store standard term pairs in the initial thesaurus and to create full-text indexes and vector indexes for field names, table names, qualifiers and data elements;

[0036] The historical matching module is used to retrieve matching standard terms from the historical database based on the similarity of field names and table names;

[0037] The semantic segmentation and vector matching module is used to fine-tune the BERT model's segmentation fields into limiting words and data elements, and to retrieve candidate combinations from the vector library to output Top-K recommendation results;

[0038] The non-Chinese field processing module is used to extract sample values ​​of non-Chinese fields and apply regular expression rules and semantic logic mapping to standard data elements;

[0039] The dynamic update module is used to store the completed benchmarking data into the historical database to update the thesaurus.

[0040] Thirdly, the present invention also provides a semantic index-based intelligent alignment device for header fields, used to implement a semantic index-based intelligent alignment method for header fields, the device comprising:

[0041] The processor is used to execute the steps of the intelligent mapping method for header fields based on semantic index segmentation;

[0042] The memory is used to store the initial lexicon, historical benchmark database, and vector index;

[0043] Input / output interfaces are used to receive data from the header fields and output the benchmarking results;

[0044] The communication module is used to interact with external data sources and governance platforms to obtain and update data.

[0045] Fourthly, the present invention also provides a computer terminal device, comprising:

[0046] One or more processors;

[0047] A memory, coupled to the processor, for storing one or more programs;

[0048] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the semantic index-based intelligent alignment method for header fields in the first aspect described above.

[0049] Fifthly, the present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the intelligent alignment method for header fields based on semantic index segmentation in the first aspect described above.

[0050] Compared with the prior art, the present invention has the following advantages and technical effects:

[0051] This invention provides a method and system for intelligent alignment of table header fields based on semantic index segmentation. By combining a semantic segmentation model with vector matching technology, this invention achieves automated decomposition and accurate alignment of table header fields. Fast retrieval based on historical databases reduces the computational overhead of repeated matching, while fine-tuning of the BERT model improves the segmentation accuracy of complex fields. Regular expression rules and semantic logic verification for non-Chinese fields effectively solve the recognition challenge of code set fields. A dynamically updated thesaurus mechanism ensures the system's adaptability to newly added terms. The overall solution significantly improves the efficiency of data standardization while reducing the need for manual intervention, making it suitable for multilingual and multi-scenario data governance environments. Attached Figure Description

[0052] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:

[0053] Figure 1 This is a flowchart illustrating the steps of an embodiment of the present invention;

[0054] Figure 2 This is a structural diagram of an embodiment of the present invention;

[0055] Figure 3 This is a system structure diagram of an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the device structure according to an embodiment of the present invention. Detailed Implementation

[0057] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0058] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0059] Example 1

[0060] like Figure 1 As shown, this embodiment provides a method for intelligent alignment of table header fields based on semantic index segmentation, including:

[0061] Collect data from the header fields, decompose it into pairs of qualifiers and data elements using a large language model, and generate an initial lexicon.

[0062] The initial vocabulary is stored in a relational database to build a historical benchmarking module;

[0063] For Chinese header fields, retrieve matching standard terms from historical databases based on field name and table name similarity;

[0064] If historical matching fails, the semantic segmentation model is used to segment the field into limiting words and data elements, and candidate combinations are retrieved through the vector library to output Top-K recommendation results.

[0065] For non-Chinese fields, sample values ​​are extracted and regular expression rules and semantic logic are applied to map them into standard data elements;

[0066] The completed benchmarking data will be stored in the historical database to update the thesaurus. (See the corresponding structure diagram above.) Figure 2 .

[0067] Specifically, the steps include the following:

[0068] S1: Collect typical header data, use a large language model to decompose each header field data into qualifier and data element pairs, select semantically consistent data as initial qualifiers and data elements, and form a usable initial lexicon.

[0069] S2: Using the initial thesaurus generated in S1, the approved qualifying words and data element pairs are organized into a standard format, stored in a relational database, and indexes are created for commonly used search fields to optimize query efficiency. This database will serve as the basic data for the historical benchmarking module.

[0070] S3: For Chinese table header fields, quickly retrieve the most matching qualifiers and data elements from the historical benchmarking database based on the similarity between field name and table name, enabling rapid identification and benchmarking of duplicate fields.

[0071] S4: When matching cannot be completed using historical benchmarking data, the system will enter the "Full-Space Benchmarking Module". The core of this module is a downstream semantic segmentation model built based on existing benchmarking data. It fine-tunes the pre-trained BERT model to adapt it to the structured task of "field qualifier and data element segmentation". After segmentation, candidate options are retrieved from the vector library and replaced with synonyms. Finally, the K×K candidate combinations are ranked, and Top-K recommendation results are output. All data that has completed full-space benchmarking are stored in the historical benchmarking database.

[0072] S5: For non-Chinese fields, the system will proceed to the "Other Content Benchmarking Module". This module, for fields without Chinese names, first performs segmentation and vector matching to locate candidate data elements, then samples 10 field values ​​to calculate cosine similarity, and determines and maps them to code set data elements. All data that has completed other content benchmarking is stored in the historical benchmarking database.

[0073] As one implementation method in this embodiment, the process of generating the initial lexicon includes:

[0074] Extract data from the header fields and convert it to JSON format;

[0075] The input large language model is decomposed into pairs of qualifiers and data elements;

[0076] High-quality term pairs are filtered and categorized for storage through semantic validation.

[0077] Specifically, S1 includes:

[0078] S1.1: Systematically extract header field data from the target data source. Ensure the collected header data is representative and covers multiple scenarios. During the collection process, perform preliminary cleaning of the header data, removing duplicate, blank, or obviously invalid fields. Finally, compile a structured header field dataset, such as storing it in list or table format, for subsequent processing.

[0079] S1.2: Convert the collected header field dataset into a JSON structure. For each header field, supplement with necessary contextual information to help the model better understand the field's semantics. Check the consistency of the data format to ensure there are no encoding errors, missing values, or formatting issues. After preprocessing, generate a properly formatted header field dataset to ensure it can be directly input into a large language model for analysis.

[0080] S1.3: Input the preprocessed header field dataset into the large language model and configure the model to perform semantic analysis tasks. The model's task is to decompose each header field into pairs of qualifiers and data elements. The model configuration explicitly requires the output of structured decomposition results, returned as key-value pairs in JSON format. For complex or ambiguous header fields, the model should attempt to infer a reasonable decomposition method based on context. Finally, preliminary qualifiers and data elements corresponding to each header field are generated.

[0081] S1.4: Perform semantic validation on the decomposition results output by the large language model to ensure that each qualifier and data element accurately reflects the semantics of the header field. Decomposition results that are semantically inaccurate, unreasonable, or inconsistent with the context are automatically filtered out. High-quality qualifiers and data elements are retained to form a filtered list of decomposition results, ensuring their semantic correctness and usability.

[0082] S1.5: Organize the filtered qualifying terms and data elements into an initial thesaurus, constructing a structured terminology set. Classify and organize the qualifying terms and data elements according to data type, industry sector, or semantic function to improve the retrieval and usage efficiency of the thesaurus. Add meta-information to each term, including source headers, semantic descriptions, and applicable scenarios, to support subsequent thesaurus expansion and maintenance.

[0083] As an optional implementation method in this embodiment, the large language model mentioned above can be a depth-based large language model.

[0084] As one implementation method in this embodiment, the construction process of the historical benchmarking module includes:

[0085] The approved qualifiers and data elements are stored in a relational database by field name and table name;

[0086] Create full-text and vector indexes for field names, table names, qualifiers, and data elements.

[0087] Specifically, S2 includes:

[0088] S2.1: Extract approved qualifying words and data elements from the initial thesaurus generated in S1, ensuring that each entry contains field names, qualifying words, data elements, and their metadata. Standardize the format of these entries and organize them into a unified JSON format to prepare for storage in the historical benchmarking database.

[0089] S2.2: Store the organized qualifiers and data elements into a relational database according to attributes such as field name, table name, qualifier ID, and data element ID. Ensure that the database table structure supports efficient storage and querying, and includes all necessary fields and their metadata. Verify the integrity of the stored procedure to ensure no data loss or formatting errors.

[0090] S2.3: Create full-text and vector indexes for frequently used search fields in the historical benchmark database to improve query efficiency. Utilize vector indexes to support semantic similarity matching, optimizing the retrieval performance of the subsequent automatic matching module. Test the effectiveness of the indexes to ensure rapid response to diverse query needs.

[0091] S3: If the target field is not Chinese, execute the other content target module in S5; otherwise, execute the "historical target module": quickly retrieve the most matching qualifiers and data elements from the historical full-space target data and target library based on the similarity of field name and table name, so as to realize the rapid identification and target of duplicate fields.

[0092] As one implementation method in this embodiment, the working process of the semantic segmentation model includes:

[0093] Fine-tuning the BERT model to identify field qualifiers and data element boundaries;

[0094] The segmentation results are output using the BIO tagging system;

[0095] Combine regular expression rule base to verify edge cases.

[0096] Specifically, S4 includes:

[0097] S4.1: Extract some data from the constructed historical benchmark database to build a semantic index segmentation training dataset. Construct training data of [data field, segmentation index], where "data field" is the data field to be segmented, and "segmentation index" is the index position that can accurately segment the data field into "qualifiers" and "data elements". Use pre-trained large language models, including but not limited to BERT, to fine-tune the downstream task, enabling the BERT model to recognize the semantic segmentation of data fields.

[0098] S4.2: The fine-tuned BERT semantic segmentation model is used to perform sequence labeling on the input data items, automatically segmenting the original fields into two parts: "qualifiers" and "data elements". The model outputs labeled sequences using the BIO tagging system, which greatly improves the segmentation accuracy of complex and long fields.

[0099] S4.3: To address potential edge cases in BERT segmentation, secondary validation and correction are performed using a maintained regular expression rule library. For example, regular expression rules are used to fine-tune tags for common formats such as dates, currencies, and units of measurement.

[0100] As one implementation method in this embodiment, the process of retrieving candidate combinations from the vector library includes:

[0101] Encode the segmented qualifiers and data elements into vectors;

[0102] Search for similar candidates in the Faiss database and calculate the cosine similarity.

[0103] Expand the candidate set by synonym replacement and generate Top-K recommendations.

[0104] Specifically, the detailed steps include:

[0105] S4.4: The segmented qualifying words and data elements are mapped into vectors using the m3e encoder and stored in the Faiss vector library. For each input segment, a nearest neighbor search is performed to retrieve the sets of the most similar qualifying word vectors. and data element vector set Similarity is calculated using cosine similarity:

[0106] ;

[0107] in, For a limited set of word vectors One of the elements, For data element vector set One of the elements;

[0108] S4.5: For the retrieved data element candidates, further utilize rule matching (such as root and prefix / suffix rules) and synonym clustering in the vector space to replace or supplement the synonym list, thereby enhancing the coverage and robustness of the data element representation.

[0109] S4.6: Take the first part of the qualifier and the data element respectively. Each candidate is paired with another candidate to generate a total of [number] candidates. For each pair of combinations (composite items), calculate the overall similarity score with the original items, sort by score, and output the Top-ranked items with the highest similarity. The combination serves as the final benchmark recommendation, and the data fields, qualifiers, data elements, and metadata that have been benchmarked are all stored in the historical benchmark database.

[0110] As one implementation method in this embodiment, the mapping process for non-Chinese fields includes:

[0111] Extract sample values ​​from the fields and perform normalization processing;

[0112] Apply regular expression rules sequentially to match date, number, and email format;

[0113] Perform semantic logic rule verification on unmatched samples.

[0114] Specifically, S5 includes:

[0115] S5.1: For fields that do not contain Chinese characters in their names, randomly select... The non-empty records are processed, obvious outliers (such as entries that are too short or too long) are removed, and each sample is normalized for whitespace and case, resulting in the sample set to be identified. .

[0116] S5.2: Apply predefined regularization rules sequentially to the sample set:

[0117] Date (e.g., \d{4}-\d{2}-\d{2}, \d{2} / \d{2} / \d{4});

[0118] Email (such as \b[\w.%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}\b);

[0119] Mobile phone number (e.g., ^1[3-9]\d{9}$);

[0120] ID number (e.g., \d{17}[0-9Xx]);

[0121] For each sample record, record the data element type that was matched, the matching ratio, and the number of matched samples. If the matching ratio of a certain category If so, it is directly mapped to that data element.

[0122] S5.3: For samples not fully covered by regularization, apply semantic logic rules sequentially:

[0123] Name database comparison: Perform fuzzy matching between the sample and common Chinese name databases or English name databases;

[0124] Gender / Ethnicity Dictionary: Checks if the sample values ​​match "male", "female" or the names of various ethnic groups;

[0125] Region / Region Mapping: Compare samples with a list of region / region names;

[0126] Other enumerated fields are determined using a rule table or a small mapping table.

[0127] S5.4: For each candidate data element type Calculate its "regular expression matching score" and "semantic rule score", and then perform a weighted average. Calculate the overall score. For the overall score, These are the weighting coefficients. This is the function for calculating the regular expression matching score. Both are semantic rule score calculation functions, and both use vector cosine similarity matching for calculation. The function with the highest score is selected. Candidates This serves as the final mapping; otherwise, it is marked as "unrecognized." Furthermore, the completed data fields, qualifiers, data elements, and metadata are all stored in the historical mapping database.

[0128] More specific implementation processes include:

[0129] S1.1: From multiple common business systems, including customer management, order processing, financial accounting, and product configuration modules, typical header fields are systematically extracted through metadata scanning and structured export to ensure coverage of different business scenarios and naming styles. The extracted fields undergo preliminary cleaning to remove duplicate, blank, or invalid fields, and field names, table names, and sample values ​​are standardized to form a structured header field dataset that meets the requirements of semantic diversity and broad coverage.

[0130] S1.2: Convert the header field dataset to JSON format, supplementing contextual information, including field value examples and system source, to enhance the semantic understanding capabilities of the large language model. The system automatically performs semantic word segmentation and named entity recognition. It checks data format consistency to ensure there are no encoding errors or missing values, generating a formatted field dataset suitable for model input.

[0131] S1.3: Input the preprocessed field dataset into the large language model, configure the model to perform semantic analysis, and automatically decompose each header field into qualifying terms and data element pairs. The model infers the decomposition method for complex or ambiguous fields based on context, and outputs structured JSON format key-value pairs (e.g., {"qualifying term":"registrant", "data element":"name"}). Through semantic similarity calculation, the decomposition results that conform to the field semantics are filtered, unreasonable pairings are eliminated, and a candidate term set is formed.

[0132] S1.4: Organize the filtered qualifying words and data elements into an initial thesaurus, categorize them by data type or business scenario, and add metadata. Through automated semantic validation rules, standardize terminology granularity and naming conventions, and handle synonymous or polysemous cases. The thesaurus is stored in JSON or database format, generating the first standardized thesaurus.

[0133] S2.1: Extract approved qualifiers and data elements from the initial thesaurus generated in S1.4. For example, the header field "Cumulative Order Amount" is broken down in S1 into {"Qualifier": "Cumulative", "Data Element": "Order Amount"}, along with metadata (e.g., Source Table: Order Table, Semantic Description: Indicates Total Order Amount). The system organizes these term pairs into a unified JSON structure, for example: { "field_name": "Cumulative Order Amount", "table_name": "Order Table", "qualifier": "Cumulative", "data_element": "Order Amount", "metadata": { "source": "Order Management System", "description": "Total Order Amount"}}. Similarly, other fields such as "Registration Time" are processed {"Qualifier": "Registration", "Data Element": "Time"}, ensuring all term pairs have a consistent format for database storage.

[0134] S2.2: Import the organized terminology into a relational database. The table structure includes columns for field names, table names, qualifier IDs, data element IDs, and metadata. Taking "Cumulative Order Amount" as an example, the system generates a record with the following fields: Field Name: Cumulative Order Amount; Table Name: Order Table; Qualifier ID: Q001 (corresponding to "Cumulative"); Data Element ID: D001 (corresponding to "Order Amount"); Metadata: {"source": "Order Management System", "description": "Total Order Amount"}. Similarly, a record is generated for "Registration Time" (Qualifier ID: Q002, Data Element ID: D002). The system automatically verifies the stored procedure, checks the record integrity, ensures no data loss or format errors, and forms an initial historical benchmark database.

[0135] S2.3: Create full-text and vector indexes for field names, table names, qualifiers, and data elements in the database. Full-text indexes support fast keyword matching; for example, entering "order amount" will retrieve records for "cumulative order amount." Vector indexes are based on a semantic embedding model, embedding "order amount" as a vector, and support semantic similarity queries, such as matching the synonymous field "total order amount." The system tests index performance to ensure that queries for "cumulative order amount" return results in milliseconds, optimizing the retrieval efficiency of the subsequent automatic matching module.

[0136] S3.1: The system receives the new Chinese header field "Total Sales Amount" and extracts the table name information: Sales Record Table. Through automated preprocessing, the fields are converted to standard JSON format.

[0137] S3.2: Retrieve records from the historical benchmarking database that exactly match "Total Sales Amount". The system uses full-text indexing to precisely match the field name "Total Sales Amount" and the table name "Sales Record Table". Assuming the database contains the record stored in S2: {"field_name": "Total Sales Amount", "table_name": "Sales Record Table", "qualifier": "Total", "data element": "Sales Amount", "qualifier ID": "Q003", "data element ID": "D004"}, the system confirms an exact match. If no exact match is found for the field name or table name, the system skips subsequent steps, and the record is marked as unmatched in the log.

[0138] S3.3: For records with an exact match, the system directly uses the qualifier and data element results from the database. For example, after a record is matched for "total sales amount", the output is: qualifier: total (ID: Q003); data element: sales amount (ID: D004). If no matching record is found, the system does not generate a new term, skips the module, and logs the data to track unmatched fields.

[0139] If the field name is in English or an abbreviation, such as "reg_dt" or "pay_amt", the system skips the Chinese matching process and directly proceeds to the "Other Content Matching Module" (i.e., the subsequent S5 process). This judgment process is automatically completed by the field language feature detection model to achieve dynamic traffic separation between Chinese and non-Chinese paths, ensuring that fields of different languages ​​can enter the most suitable matching module.

[0140] S4.1: Extract partial data from the historical benchmark database constructed in S2, such as the record "Cumulative Order Amount": {"Qualifier": "Cumulative", "Data Elements": "Order Amount"}. The system constructs a training dataset in the format [data field, segmentation index], where "data field" is "Cumulative Order Amount" and "segmentation index" is [0, 2, 4] (representing "Cumulative" [0:2], "Order Amount" [2:4]). Other fields (such as "Registration Time": [0, 2, 4]) are processed in a similar manner to generate a training dataset containing 10,000 records. A pre-trained BERT model is fine-tuned and optimized for sequence labeling tasks, enabling the model to learn to accurately segment the Chinese header field into qualifyers and data elements.

[0141] S4.2: The fine-tuned BERT model uses the BIO tag system (B-determiner, I-determiner, B-data element, I-data element, O-other) for sequence annotation. The model is fine-tuned based on the training dataset constructed in S4.1 (containing 10,000 [data field, segmentation index] records, for example, "cumulative order amount" corresponds to the index [0, 2, 4]) to optimize the parameters to identify the semantic boundaries of Chinese table header fields. For "cumulative order amount", the model predicts the tags character by character, and the output sequence is as follows: lei: B-determiner (indicating the start of the determiner); ji: I-determiner (indicating the continuation of the determiner); ding: B-data element (indicating the start of the data element); dan: I-data element; jin: I-data element; e: I-data element. Through the multi-layer Transformer structure, the model combines context information (such as "order table" implying a sales scenario) to improve the semantic understanding of complex fields and ensure accurate segmentation. According to the BIO tag sequence, the system segments "cumulative order amount" into two parts: determiner: "cumulative" (composed of B-determiner and I-determiner, character positions [0:2]); data element: "order amount" (composed of B-data element and I-data element, character positions [2:6]). The segmentation result is output in JSON format.

[0142] S4.3: For potential edge cases in BERT segmentation, the system applies a regular expression rule library for secondary verification. For example, for "order amount 2023" which may be mis-segmented as "order" and "amount 2023", regular rules (such as "\d{4}$" to identify the year) correct it to "order amount" and "2023" (removing the date part). For "cumulative order amount", the regular rules confirm that there is no special format (such as date, currency), and the BERT segmentation result is retained: {"determiner": "cumulative", "data element": "order amount"}.

[0143] S4.4: Map the segmentation results "cumulative" and "order amount" to 768-dimensional vectors through the m3e encoder and store them in the Faiss vector library. The Faiss library has pre-stored the determiner vectors (such as "cumulative", "total") and data element vectors (such as "order amount", "sales amount") of the historical comparison database. For "cumulative", perform a nearest neighbor search to retrieve the Top-3 determiner vectors (such as "cumulative", "total", "total amount"); for "order amount", retrieve the Top-3 data element vectors (such as "order amount", "sales amount", "transaction amount"). The similarity is calculated using cosine. For example, the similarity between "cumulative" and "total" is 0.93.

[0144] S4.5: The system combines the top K candidates (K = 3) of determiners and data elements pairwise to generate K×K = 9 synthetic items, and outputs the Top-K recommended results through comprehensive similarity calculation. From the determiner set and data set The top 3 candidates from each group are selected, generating 9 combinations. For each combination, the system calculates a comprehensive similarity score using the following formula: , in, To calculate the overall similarity score, For candidate qualifying word vectors, For the original data item vector, For candidate data element vectors, and The cosine similarity is calculated for the qualifying words and data elements, respectively, using the following formula: , Similarly; Based on table names (e.g., "Order Table") and historical benchmarking frequency, assess the business rationale for the combination; for example, the "Order Table" context improves the score of "Order"-related combinations. Weights are set as follows: The system can be adjusted according to general domain scenarios. Nine combinations are sorted from highest to lowest score, and the Top-1 (K=1) is selected as the recommendation result {"qualifier": "cumulative", "data element": "order amount"}, with a similarity of 0.98. The benchmarking results (field name: cumulative order amount, qualifyifier: cumulative, data element: order amount, metadata: {"table_name": "order table", "sample_value": "123456.78"}) are stored in the historical benchmarking database, and the Faiss vector library is updated. Testing with 100 fields, the Top-1 accuracy reaches 93%.

[0145] S5.1: The system first preprocesses the "OrderID" field to generate a reliable sample set for subsequent analysis. Ten non-empty records are randomly selected from the "OrderID" field, such as "ORD12345", "2023-ORD-001", and "ORD98765", excluding obvious outliers (such as null values ​​or records containing only one character). The system then normalizes each sample, including removing extra spaces, standardizing case (e.g., converting "ord12345" to "ORD12345"), and standardizing character encoding to UTF-8 to ensure data consistency. The normalized sample set retains 10 valid records, such as "ORD12345", "2023-ORD-001", "ORD98765", and "ORD45678", which reflect the typical format of order numbers and may contain letters, numbers, and separators (such as "-"). The preprocessing process ensures that the sample set represents the true data distribution of the field, providing a reliable foundation for subsequent regularization rules and semantic analysis. The system records the original and normalized values ​​of each sample and generates a sample set to be identified for the next step of rule matching.

[0146] S5.2: The system sequentially applies a predefined regular expression rule library to the normalized sample set to identify the data element type of the "OrderID" field. The regular expression rule library is designed for general domains and includes various common formats, such as dates (e.g., `\d{4}-\d{2}-\d{2}` or `\d{2} / \d{2} / \d{4}`), mobile phone numbers (e.g., `^1[3-9]\d{9}$`), ID card numbers (e.g., `\d{17}[0-9Xx]`), and number formats (e.g., `[A-Za-z0-9\-]+`). For the "OrderID" sample set, the system matches each record one by one and finds that all samples (e.g., "ORD12345" and "2023-ORD-001") conform to the number format regular expression rule `[A-Za-z0-9\-]+`, which allows combinations of letters, numbers, and hyphens. The matching results showed that 10 out of 10 samples perfectly matched the number format, a matching rate of 100%. The system recorded the matching type (number) and matching rate for each sample, confirming that the "OrderID" field is highly likely to be a number type data element. Since the matching rate reached 100%, the system initially mapped the field to the standard data element "order number," but further verification using subsequent semantic logic rules is still needed to ensure accuracy.

[0147] S5.3: For fields not fully covered by regular expression rules or requiring further verification, the system applies semantic logic rules to analyze the sample set to exclude other possible semantic types and confirm the mapping results. Even though the regular expression matching rate for the "OrderID" field reaches 100%, the system still performs semantic logic rule checks to ensure robustness. Semantic rules include name database comparison (checking whether the sample matches common Chinese or English names), gender / ethnicity dictionary matching (checking whether it contains "male," "female," or ethnic names), region / area mapping (comparing with a list of region / area names), and rule table matching for other enumerated fields. For the "OrderID" sample set, the system first compares it against the common name database (including Chinese names such as "Zhang Wei" and English names such as "John Smith"), finding that samples like "ORD12345" do not match any name pattern. The gender / ethnicity dictionary check confirms that the sample does not contain "male," "female," or ethnic names. The region / area mapping also yields no matching results because the sample does not involve geographical names. The system further examined the mapping tables of enumerated fields (such as status fields like "Active" and "Pending"), but the alphanumeric combination of the "OrderID" sample did not conform to enumerated characteristics. Since the sample highly conformed to the number format and did not match other semantic rules, the system confirmed that "OrderID" was most likely an order number type data element, and no further extended analysis was required.

[0148] S5.4: The system calculates a comprehensive score for each candidate data element type to determine the final mapping result. For the "OrderID" field, the candidate data element type is "Order Number," based on regular expression matching and semantic rule analysis results. The system calculates the "regular expression matching score," defined as the proportion of matching samples, i.e.: ,in The number of samples to match the regular expression rule, This represents the total number of samples. Here, , ,therefore Since the regular expression matching rate reaches 100%, there is no need to calculate the semantic rule score (no other candidate types were found for the semantic rules). The comprehensive score formula is: The weight Weight ,and (Since there are no other semantic types). Therefore, the overall score is: Because the score exceeds the threshold of 0.8, the system maps "OrderID" to the standard data element "Order Number". If the overall score is below the threshold or multiple candidate types exist, the system will mark it as "Unrecognized". In this example, "Order Number" is the final mapping result, with an accompanying mapping description recording the correspondence between the original field and the standard terminology.

[0149] Based on this, the present invention provides an intelligent benchmarking method for table header fields based on semantic index segmentation, which significantly improves the efficiency, accuracy, and adaptability of field standardization in data governance compared to existing technologies. Through the collaborative work of a BERT-based fine-tuned semantic index segmentation model, semantic vector matching, and a historical benchmarking database, the system achieves automated decomposition and accurate matching of table header fields, greatly reducing manual intervention and maintenance costs. Simultaneously, the system effectively handles semantically ambiguous, non-Chinese, and code set fields, overcoming the limitations of traditional methods on complex fields and enhancing adaptability across multiple scenarios. The accuracy and recall rates for qualifying word matching both exceed 90% for the first three automatically recommended standard qualifying words; the accuracy and recall rates for data element matching both exceed 95% for the first three automatically recommended standard data elements, further validating the system's superior performance. Furthermore, modular design and dynamic thesaurus updates further optimize query performance and system scalability, promote data standardization and efficient circulation, provide strong support for data integration and analysis, and demonstrate significant creativity and practical value.

[0150] Example 2

[0151] In this embodiment, as Figure 4 As shown, a semantic index-based intelligent alignment device for table header fields is provided, comprising:

[0152] The processor is used to execute the steps of the above-described intelligent alignment method for header fields based on semantic index segmentation;

[0153] The memory is used to store the initial lexicon, historical benchmark database, and vector index.

[0154] In addition to the processor and memory, it also includes:

[0155] Input / output interfaces are used to receive data from the header fields and output the benchmarking results;

[0156] The communication module is used to interact with external data sources and governance platforms to obtain and update data.

[0157] In this embodiment, a computer terminal device is provided, including:

[0158] One or more processors;

[0159] A memory, coupled to the processor, for storing one or more programs;

[0160] When the one or more programs are executed by the one or more processors, the one or more processors implement the steps of the above-described intelligent alignment method for header fields based on semantic index segmentation.

[0161] In this embodiment, a computer-readable storage medium is also provided, on which a computer program is stored. When the computer program is executed by a processor, it implements the steps of the above-described intelligent alignment method for header fields based on semantic index segmentation.

[0162] In this embodiment, an electronic device is also provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps of the above-described intelligent tagging method for header fields based on semantic index segmentation.

[0163] In this embodiment, a computer program product is also provided, including a computer program that, when executed by a processor, implements the steps of the above-described intelligent alignment method for header fields based on semantic index segmentation.

[0164] The aforementioned program can run on a processor or be stored in memory (or a computer-readable medium). Computer-readable media includes both permanent and non-permanent, removable and non-removable media, and information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0165] These computer programs may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes can be implemented by different modules for different steps.

[0166] This embodiment provides such an apparatus or system. The system, referred to as a semantic index-based intelligent header field matching system, includes:

[0167] The lexicon building module is used to collect data from the header fields and decompose it into pairs of qualifying words and data elements using a large language model to generate an initial lexicon.

[0168] The historical benchmarking database module is used to store standard term pairs in the initial thesaurus and to create full-text indexes and vector indexes for field names, table names, qualifiers and data elements;

[0169] The historical matching module is used to retrieve matching standard terms from the historical database based on the similarity of field names and table names;

[0170] The semantic segmentation and vector matching module is used to fine-tune the BERT model's segmentation fields into limiting words and data elements, and to retrieve candidate combinations from the vector library to output Top-K recommendation results;

[0171] The non-Chinese field processing module is used to extract sample values ​​of non-Chinese fields and apply regular expression rules and semantic logic mapping to standard data elements;

[0172] The dynamic update module is used to store the completed benchmarking data into the historical database to update the thesaurus.

[0173] As one implementation method in this embodiment, the lexicon construction module includes:

[0174] The data extraction unit is used to extract header fields from the data source and perform preliminary cleaning.

[0175] Semantic decomposition unit, used to decompose fields into pairs of qualifiers and data elements using a large language model;

[0176] The verification storage unit is used to filter high-quality term pairs and classify and store them in the initial thesaurus.

[0177] As one implementation method in this embodiment, the historical benchmarking database module includes:

[0178] Standardized storage units are used to store qualifiers and data element pairs into a relational database by field name and table name;

[0179] The index optimization unit is used to create full-text and vector indexes for frequently used fields to accelerate retrieval.

[0180] As one implementation method in this embodiment, the semantic segmentation and vector matching module includes:

[0181] The BERT segmentation unit is used to fine-tune the BERT model and segment fields using the BIO tagging system.

[0182] The vector retrieval unit is used to encode the segmentation results into vectors and retrieve similar candidates from the Faiss library;

[0183] The recommendation generation unit is used to calculate the comprehensive similarity of candidate combinations and output the Top-K results.

[0184] As one implementation method in this embodiment, the non-Chinese field processing module includes:

[0185] The sample preprocessing unit is used to extract non-Chinese field sample values ​​and perform normalization.

[0186] The regular expression matching unit is used to apply predefined rules to identify date, number, and email format.

[0187] The semantic verification unit is used to eliminate ambiguity and confirm the final mapping through logical rules.

[0188] As one implementation method in this embodiment, the dynamic update module includes:

[0189] The data writing unit is used to store the benchmarking results into the database according to field names, qualifiers, and data elements;

[0190] The vector library update unit is used to encode new term pairs into vectors and expand the Faiss library.

[0191] The system or apparatus is used to implement the functions of the methods in the above embodiments. Each module in the system or apparatus corresponds to each step in the method, as has been described in the method and will not be repeated here.

[0192] like Figure 3 As shown, a system architecture for intelligent alignment of table header fields based on semantic index segmentation is provided, including:

[0193] Historical benchmarking database is used to store metadata corresponding to historical benchmarking data;

[0194] Historical benchmarking qualifier / data element library, used to store the qualifiers and data elements corresponding to historical benchmarking data;

[0195] The BERT semantic segmenter is used to segment the original data items;

[0196] The regular expression rule library is used to store regular expressions corresponding to different data types;

[0197] Content discriminator, used to identify non-Chinese data item fields;

[0198] The irregular content regular expression rule library is used to store regular expressions corresponding to different data types of non-Chinese fields;

[0199] The m3e encoder is an encoder used to encode text into vectors.

[0200] Faiss vector library, a vector database used to store vectors;

[0201] A standard qualifying thesaurus, a database used to store standard qualifying terms;

[0202] A standard data element lexicon is a database used to store standard data elements.

[0203] The above implementation method solves the problem of intelligent alignment of header fields based on semantic index segmentation in related technologies, thereby ensuring that the problems existing in the prior art are resolved.

[0204] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent matching of table header fields based on semantic index segmentation, characterized in that, The method comprises the following steps: collecting table header field data, decomposing into qualifier and data element pairs by a large language model, and generating an initial word library; storing the initial word library in a historical matching database; for Chinese table header fields, searching the historical matching database for matching standard terms based on field name and table name similarity; if historical matching fails, using a semantic segmentation model to segment the field into qualifiers and data elements, searching the vector library for candidate combinations, and outputting Top-K recommended results; for non-Chinese fields, extracting sample values and applying regular rules and semantic logic to map them to standard data elements; storing the completed matching data in the historical matching database to update the word library.

2. The method of claim 1, wherein, The process of generating the initial word library comprises: extracting table header field data and converting it to JSON format; inputting the large language model to decompose into qualifier and data element pairs; screening high-quality term pairs through semantic verification and storing them in categories.

3. The method of claim 1, wherein, The construction process of the historical matching database comprises: storing the approved qualifier and data element pairs in a relational database according to field name and table name; creating full-text and vector indexes for field name, table name, qualifier, and data element.

4. The method of claim 1, wherein, The working process of the semantic segmentation model comprises: fine-tuning the BERT model to identify the boundaries of qualifiers and data elements in the field; outputting the segmentation results using the BIO tag system; verifying edge cases in combination with the regular rule library.

5. The method of claim 1, wherein, The process of searching the vector library for candidate combinations comprises: encoding the segmented qualifiers and data elements into vectors; searching for similar candidates in the Faiss library and calculating the cosine similarity; expanding the candidate set through synonym replacement and generating Top-K recommendations.

6. The method of claim 1, wherein, The mapping process for non-Chinese fields comprises: extracting field sample values and normalizing them; applying regular rules to match dates, numbers, and email formats in sequence; performing semantic logic rule verification on unmatched samples.

7. A semantic index segmentation based table header field intelligent matching system, characterized in that, The system is used to implement the steps of the method of any one of claims 1-6, and comprises: a word library construction module for collecting table header field data and decomposing into qualifier and data element pairs by a large language model to generate an initial word library; a historical matching database module for storing standard term pairs in the initial word library and creating full-text and vector indexes for field name, table name, qualifier, and data element; a historical matching module for searching the historical matching database for matching standard terms based on field name and table name similarity; a semantic segmentation and vector matching module for segmenting the field into qualifiers and data elements by fine-tuning the BERT model and searching the vector library for candidate combinations to output Top-K recommended results; a non-Chinese field processing module for extracting non-Chinese field sample values and applying regular rules and semantic logic to map them to standard data elements; a dynamic updating module for storing the completed matching data in the historical matching database to update the word library.

8. A device for intelligent matching of table header fields based on semantic index segmentation, characterized in that The device comprises: a processor for executing the steps of the method of any one of claims 1-6; a memory for storing the initial word library, the historical matching database, and the vector index; an input-output interface for receiving table header field data and outputting matching results; a communication module for interacting with external data sources and governance platforms to obtain and update data.

9. A computer terminal device, characterized by The method comprises: one or more processors; a memory coupled to the processors for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement steps of the method of any one of claims 1-6.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, when the computer program is executed by a processor, steps of the method of any one of claims 1-6 are implemented.

Images