A method and system for mapping field names to standard signatures based on semantic matching

Abstract

This invention discloses a field name standard mapping method and system based on semantic matching, specifically relating to the field of data governance technology. The method involves obtaining a set of source fields to be mapped and a set of standard fields from a standard field library; standardizing field names and descriptions and generating textual representations of the fields; generating semantic vectors using a pre-trained language model and recalling a set of candidate mapping pairs; calculating string similarity, synonym expansion matching scores, data type consistency scores, and sample value pattern consistency scores for the candidate mapping pairs; generating a nearest neighbor confusion risk index based on semantic nearest neighbor density, entity category distribution deviation, synonym inflation, and contextual proof, and performing risk gating and correction, combined with difficult example discrimination scores; applying table-level consistency constraints within the same business table or entity domain to obtain the optimal table-level mapping scheme through joint consistency solving; generating confirmation confidence and outputting mapping results that are automatically confirmed or manually reviewed.

Description

Technical Field

[0001] This invention relates to the field of data governance technology, and more specifically, to a method and system for mapping field name standards based on semantic matching. Background Technology

[0002] Automatic data field mapping and standardized identification methods can establish source field correspondences and standard fields for multi-source systems. This typically involves encoding field names and descriptions into semantic vectors, retrieving similarity from a standard field library, and performing consistency checks on edit distance, synonym expansion, and data type. Candidate results are then ranked, and suggestions for automatic confirmation or manual review are provided. Currently, in areas such as data governance, standardized reporting, and cross-domain data sharing, this can significantly reduce manual comparisons and improve matching efficiency.

[0003] However, in actual business operations, there is still a type of mismatch that is easily overlooked but has a significant impact: when the candidate set contains fields such as name, legal representative name, responsible person name, and contact phone number, office phone number, and contact person phone number, these fields are highly similar in literal expression and descriptive wording, and often have the same data type. The semantic similarity model will treat them as neighbors and give them high scores. At the same time, string similarity and synonym expansion often treat words like "responsible person" and "contact person," which are not interchangeable in some scenarios, as gain signals, causing erroneous candidates to move further forward in the ranking. Since the above candidates are not obviously unreasonable from the rule perspective, for example, they all belong to text or number fields, it is difficult to reliably exclude them by relying on simple thresholds, type filtering, or a small number of blacklists and whitelists. Ultimately, this can easily lead to mislabeling, such as mapping a name to a legal representative name or mapping a contact person phone number to an office phone number. Such mislabeling often lies dormant downstream in a form that seems to be valid, and usually only becomes apparent during summary statistics, joint queries, or compliance audits. At that time, it is necessary to backtrack and correct the mapping and rerun multiple processing links, resulting in high rework costs and risk exposure.

[0004] To address the aforementioned issues, this invention introduces a discrimination criterion that is closer to business semantics during the candidate generation and decision ranking stages. This can identify and suppress hard negative examples that are similar but semantically different, thereby improving the reliability of automatic confirmation and preventing mislabeling from propagating to subsequent links. Summary of the Invention

[0005] To overcome the aforementioned deficiencies of the prior art, embodiments of the present invention provide a semantic matching-based field name standard label mapping method and system to solve the problems mentioned in the background art.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A method for mapping field names to standard signatures based on semantic matching includes the following steps:

[0008] Obtain the set of source fields to be mapped and the set of standard fields in the standard field library; normalize the source fields and standard fields and generate field text representations; obtain semantic vectors based on the field text representations through a pre-trained language model, and generate a set of candidate mapping pairs for each source field based on the semantic vectors;

[0009] For each candidate mapping pair, calculate string similarity, synonym expansion matching score, data type consistency score, and sample value pattern consistency score; for each source field, determine risk discrimination parameters and generate a nearest neighbor confusion risk index based on its candidate mapping pair set. The risk discrimination parameters include at least semantic nearest neighbor density parameter, entity category distribution deviation parameter, synonym inflation parameter, and contextual proof of contradiction parameter; perform risk gating correction on the candidate mapping pairs based on the nearest neighbor confusion risk index to obtain a corrected candidate mapping pair set.

[0010] The corrected candidate mapping set is subjected to hard case discrimination to obtain a hard case discrimination score; within the same business table or entity domain, based on table-level consistency constraints, semantic similarity, string similarity, synonym expansion matching score, data type consistency score, sample value pattern consistency score and hard case discrimination score are fused to perform joint consistency solution and output the optimal table-level mapping scheme;

[0011] Generate confirmation confidence based on the optimal mapping scheme at the table level, and output the automatically confirmed mapping result or the manually reviewed mapping result based on the confirmation confidence.

[0012] In a preferred embodiment, the normalization process includes performing at least one or more processing on field names and field descriptions: case unification, delimiter unification, camelCase splitting, redundant whitespace cleanup, special character cleanup, and domain abbreviation restoration.

[0013] In a preferred embodiment, the field text representation is generated by combining the field name and field description according to a preset concatenation rule, and the field text representation is generated using the field name when the field description is missing.

[0014] In a preferred embodiment, generating a candidate mapping pair set for each source field based on semantic vectors includes: establishing a vector retrieval index for the semantic vectors of standard fields, and performing nearest neighbor recall for the semantic vectors of each source field to obtain the corresponding candidate mapping pair set.

[0015] In a preferred embodiment, string similarity is calculated based on the field name of the source field and the field name of the standard field; before calculating string similarity, at least one of the following preprocessing steps is performed on the field names: case conversion, delimiter processing, and redundant character cleanup.

[0016] In a preferred embodiment, the synonym expansion matching score is obtained by performing synonym replacement or synonym expansion on the field text representation using a thesaurus, and then calculating the matching score based on the replaced or expanded field text representation and the field text representation of the standard field.

[0017] In a preferred embodiment, the data type consistency score is determined based on the data type information of the source field and the data type information of the standard field, and the sample value pattern consistency score is determined based on the sample value pattern information of the source field and the sample value pattern information of the standard field, wherein the sample value pattern information includes at least one of character category composition, length characteristics, and preset format hit status.

[0018] In a preferred embodiment, the risk discrimination parameter includes a semantic nearest neighbor density parameter, which is determined based on the degree of difference in semantic similarity among a preset number of candidate mapping pairs in the candidate mapping pair set.

[0019] In a preferred embodiment, the joint consistency solution includes: within the same business table or entity domain, determining the conflict of candidate mapping pairs of multiple source fields according to table-level consistency constraints, and applying conflict penalty processing to candidate mapping pairs with consistency conflicts, so as to obtain the optimal table-level mapping scheme.

[0020] In a preferred embodiment, the following modules are included:

[0021] The candidate generation module is used to obtain the set of source fields to be labeled and the set of standard fields in the standard field library; to normalize the source fields and standard fields and generate field text representations; to obtain semantic vectors based on the field text representations through a pre-trained language model, and to generate a set of candidate mapping pairs for each source field based on the semantic vectors.

[0022] The risk correction module calculates string similarity, synonym expansion matching score, data type consistency score, and sample value pattern consistency score for each candidate mapping pair. For each source field, it determines risk discrimination parameters and generates a nearest neighbor confusion risk index based on its candidate mapping pair set. The risk discrimination parameters include at least semantic nearest neighbor density parameter, entity category distribution deviation parameter, synonym inflation parameter, and contextual proof of contradiction parameter. Based on the nearest neighbor confusion risk index, the module performs risk gating correction on the candidate mapping pairs to obtain a corrected candidate mapping pair set.

[0023] The joint decision module is used to identify hard cases in the corrected candidate mapping set and obtain a hard case identification score. Within the same business table or entity domain, based on table-level consistency constraints, it integrates semantic similarity, string similarity, synonym expansion matching score, data type consistency score, sample value pattern consistency score and hard case identification score to perform joint consistency solution and output the optimal table-level mapping scheme.

[0024] The confidence confirmation module is used to generate a confirmation confidence level based on the optimal mapping scheme at the table level, and output the automatically confirmed mapping result or the manually reviewed mapping result based on the confirmation confidence level.

[0025] The technical effects and advantages of this invention are as follows:

[0026] This invention addresses the issue of hard negative example mismatches caused by semantic proximity and literal similarity in automatic field labeling by introducing nearest-neighbor confusion risk identification and a risk-driven table-level joint consistency mechanism during candidate generation and decision ranking. This effectively improves the accuracy of mapping results and business consistency. The main technical effects and advantages are as follows: First, by establishing an interpretable risk discrimination system, the confusion risk of each source field is dynamically measured based on parameters such as semantic proximity density, entity category distribution deviation, synonym inflation, and contextual proof. Adaptive correction is performed based on the risk index to identify and suppress candidate pairs with different business semantics but similar features, such as name and legal representative name, or contact phone number and office phone number. This reduces the probability of mislabeling from the source and avoids rework and compliance risks caused by the propagation of incorrect mapping downstream. Second, it overcomes the limitations of the original single-field independent mapping by introducing table-level consistency constraints and a joint decision mechanism to perform global optimization within the same business table or entity domain. By combining semantic similarity, string features, data types, sample value patterns, and difficult case discrimination, along with risk adaptive weight adjustment, the logical consistency of the mapping results between fields is ensured, effectively preventing overall semantic conflicts caused by local optima, and effectively improving the business rationality and interpretability of the bid rejection scheme.

[0027] Meanwhile, this invention establishes a confidence evaluation and closed-loop update process. It determines the credible confidence level based on local fusion scores, table-level decision stability, and risk level, and defines the scope of automatic confirmation and manual review. The results of manual review can be fed back through risk discrimination models and difficult case discrimination models, thereby calibrating the models to adapt to business changes and standard evolution, achieving a self-optimizing closed-loop implementation.

[0028] In summary, while maintaining the efficiency of existing semantic matching frameworks, this invention enables field mapping to have higher robustness and accuracy through risk awareness and global collaboration, providing reliable automation capabilities for data governance, report standardization, and cross-system data sharing. Attached Figure Description

[0029] To facilitate understanding by those skilled in the art, the present invention will be further described below with reference to the accompanying drawings;

[0030] Figure 1 This is a flowchart illustrating a field name standard offset mapping method based on semantic matching according to the present invention.

[0031] Figure 2 This is a schematic diagram of the structure of a field name standard label mapping system based on semantic matching according to the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1: This invention provides a field name standard offset mapping method based on semantic matching, such as... Figure 1 As shown, it includes the following steps:

[0034] Step 1: Field semantic modeling and candidate generation;

[0035] In this implementation, a set of source fields is first obtained from the source system to be matched, and a set of standard fields is obtained from a pre-established standard field library. For any source field, at least the field name, field description, and available sample values ​​are collected; for any standard field, at least the standard field name and standard field description are collected, and standard sample values ​​are read synchronously when the standard field library provides standard sample values. Since different systems differ in naming styles, symbol usage, and description habits, to ensure the consistency of input for subsequent semantic matching, the field names and field descriptions of the source and standard fields are uniformly standardized, converting them into field text content that can be directly used for modeling. The standardization process may include unifying capitalization, unifying the processing of underscores, hyphens, and other separators, splitting camelCase naming, cleaning up abnormal symbols, duplicate spaces, and meaningless characters, and combining this with domain abbreviation restoration and word segmentation to ensure that the field text content can stably express the field semantics. After normalization, the field text representation is generated according to preset concatenation rules: starting with the field name, followed by a Chinese colon ":" as a separator, and then connecting to the field description; if the field description is empty, only the field name is used; if the sample value is available, "Sample:" is appended after the description and a sample value fragment of the first 10 characters is concatenated; if the sample value cannot be used directly, "Pattern:" is appended and the sample value morphological fingerprint text is concatenated, such as a mixture of numbers and letters of 8-18 characters in length. The final generated field text representation is denoted as the source field. and standard fields .

[0036] After obtaining the field text representation, the field text representation is input into a pre-trained language model to obtain the field semantic vector representation. For any source field s, its field text representation is... Inputting the pre-trained language model yields the semantic vector of the source field. For any standard field t, represent its field text. Inputting a pre-trained language model yields standard field semantic vectors. To reduce online computation overhead, this implementation method uses the standard field semantic vector of the standard field set. Offline computation and storage are performed to form a standard field semantic vector library. To support efficient nearest neighbor retrieval, an IVF (Inverted File System) index is built using the FAISS library, with the number of cluster centers set to 1024, and the inner product is used as the similarity measure. During retrieval, for each source field vector... The system recalls the top N=10 standard fields with the highest similarity, retaining only candidates with a similarity greater than 0.6 to balance recall rate and computational cost. During actual dropout, computation is performed only on the source field set in real time. The similarity is calculated between the candidate standard field and the semantic vector library, thereby enabling rapid recall of the candidate standard field.

[0037] During candidate recall, semantic similarity is used as the basic recall criterion, and semantic similarity is calculated between any source field s and any standard field t. and in accordance with Sort the data from highest to lowest, and select the top N standard fields to form the semantic candidate set for that source field. Cosine similarity is preferred for semantic similarity, and its calculation method is as follows:

[0038] ;

[0039] in, This indicates the semantic similarity between the source field s and the standard field t; This is the semantic vector corresponding to the source field s. This is the semantic vector corresponding to the standard field t; Represents the dot product of vectors; and Representing vectors respectively with vector The 2-norm. and All fields are represented by pre-trained language models. and The output shows that the dot product and norm operations are performed as regular vector operations.

[0040] To ensure that candidate mapping pairs possess usable multi-source information in subsequent correction and joint decision-making stages, this step, while generating a semantic candidate set, calculates and binds string matching features and data features for each candidate mapping pair (s,t), forming a multi-dimensional feature-enhanced candidate mapping pair set. Specifically, a string similarity score is calculated based on the source field name and the standard field name, denoted as . The The result can be obtained from the normalized edit distance, thus reflecting the closeness between the two at the character level; when synonym expansion is enabled, the terms in the field text are replaced or expanded according to a pre-configured thesaurus to obtain the synonym expansion matching score. ,in This is used to characterize whether the matching of the candidate mapping pair depends on synonym triggering and the matching strength after triggering. Simultaneously, a data type consistency score is extracted based on the available information of the source field sample values. Consistency score with sample value pattern ,in This is used to reflect the degree of consistency between the data type that can be identified from the sample value of the source field and the expected type of the standard field. This is used to reflect the degree to which the sample value of the source field conforms to the expected pattern of the standard field at the format or pattern level; when the sample value is in a de-sensitized state, The pattern fingerprint can be calculated by matching the morphological fingerprint of sample values ​​with a preset format pattern. The morphological fingerprint includes information that can be directly extracted from the anonymized samples, such as length distribution, character category sequences, and pattern matching. The specific extraction method is as follows: For a set of anonymized sample values ​​for a field, firstly, the minimum, maximum, and mode of the value lengths are counted to form a length distribution feature. Secondly, each sample value is mapped to a character category sequence, and then the frequency of each category and common sequence patterns in all sequences are counted. Finally, the above statistical results are matched with a preset format pattern to obtain the pattern matching. This information, after being textualized, can be used as a supplementary part of the field's textual representation and input into the model.

[0041] Step Two: Identification and Correction of Nearest Neighbor Confusion Risk;

[0042] After completing the candidate mapping generation in step one, this implementation enters the candidate correction stage. Because candidate sets often contain semantically similar, character-similar, and data-data-consistent pairs such as names, legal entity names, responsible person names, contact numbers, office numbers, and contact person numbers, these pairs are difficult to distinguish using only semantic similarity or simple rules, leading to highly similar hard negative examples being ranked higher. Therefore, this step establishes a nearest-neighbor confusion risk index for each source field to quantify the likelihood of that source field causing nearest-neighbor confusion in its candidate set. Based on this index, gating correction is applied to the candidate mapping pairs, ensuring that subsequent re-ranking and joint consensus decisions are performed within a cleaner and more controllable candidate space.

[0043] Specifically, for any source field s, let its set of candidate standard fields generated in step one be . ,in Based on semantic similarity The ranking is based on the top N standard fields, and each candidate mapping pair (s,t) is also bound to string similarity. Synonym expansion matching score Data type consistency score and sample value pattern consistency score Based on the available parameters, this step calculates the nearest neighbor confusion risk index. ,in The nearest neighbor confusion risk index is derived from at least two types of parameters, including semantic nearest neighbor density, entity category distribution deviation, synonym inflation, and contextual counter-evidence. It reflects the degree of "crowding and difficulty in distinguishing highly similar candidates" within the candidate set and is used to drive risk gating correction. The nearest neighbor confusion risk index can be calculated using a normalized weighted summation, as shown in the following formula:

[0044] ;

[0045] in, This represents the nearest neighbor confusion risk index of the source field s; This is a monotonically increasing normalization function used to map risk values ​​to a preset range, preferably a Logistic function, with the following specific form:

[0046] ;

[0047] Wherein, the slope parameter k and the center point parameter It can be preset based on historical data distribution, for example, k=1.0. =0. This function ensures The interpretable risk value within the interval [0,1]; weighting coefficient , , , The search range was determined using a grid search combined with five-fold cross-validation, with a step size of 0.1. The objective was to maximize the AUC of the field distinguishing between high-risk and low-risk data on the validation set, and the final value was [value missing]. , , , .

[0048] Let be the semantic nearest neighbor density parameter, used to characterize the crowding degree of semantic similarity within the candidate set. Its specific calculation method is as follows: Let the candidate set... It contains N standard fields, ranked according to their semantic similarity to the source field s. Sort in descending order, denoted as .but It can be calculated based on the similarity difference between the top K candidates, for example:

[0049] ;

[0050] Where K is a preset value, such as K=3. When the number of candidates is less than K, the actual number will be used for calculation. The closer the value is to 1, the closer the similarity of the top K candidates is, and the higher the crowding level. Because... Only relying on the calculations already performed in step one .

[0051] This is the entity category distribution deviation parameter, used to characterize the overall deviation of the source field from its candidate standard fields at the entity category level; The synonym expansion parameter is used to characterize the degree to which a candidate set obtains a higher matching score by relying on synonym expansion. These are context-based parameters used to characterize the degree of conflict between the candidate set and the table structure or business domain context of the source field. All three parameters can be calculated at the candidate set level by averaging or weighted summarizing each candidate mapping pair; therefore, they are all readily available parameters.

[0052] To obtain This step first establishes the entity category distribution between the source field and the standard field.

[0053] Specifically, the entity category probability distribution vector P(e|x) is generated using a lightweight text classification model based on a bidirectional LSTM and attention mechanism. The input to this model is the field text representation. The output is a probability distribution of six entity categories: natural persons, legal entities, responsible parties, contact information, certificates / identifications, and geographical addresses. Training data comes from historically manually labeled field samples, each containing a field name, description, and corresponding entity category label. Training uses the cross-entropy loss function and the Adam optimizer with a learning rate of 0.001 for 50 iterations. The final model achieves a classification accuracy of over 92% on the validation set. If model inference fails in certain scenarios, a rule-based alternative is employed: keyword matching (e.g., field names containing "name" or "person" favor natural persons, while those containing "company" or "legal entity" favor legal entities), combined with sample value patterns (e.g., conforming to a phone number format favors contact information), is weighted, and then normalized to obtain the probability distribution vector. Based on this vector, for candidate mapping pairs (s,t), cosine distance is used to calculate the entity category deviation. ,in and These are the entity distribution vectors for the source field and the standard field, respectively.

[0054] Based on the entity category distribution, calculate the entity category deviation for the candidate mapping pair (s,t). The deviation reflects the degree of difference between the source field s and the standard field t at the entity category level. A larger deviation indicates that they may belong to different entity semantics, thus making them more likely to constitute hard negative examples with similar semantics but different business semantics. The deviation can be obtained using calculable measures such as cosine distance or Jensen-Shannon divergence; taking cosine distance as an example, the deviation can be calculated from the entity distribution vector of the source field. Compared with standard field entity distribution vector The calculation is obtained. Then, the candidate set is... Deviation of each candidate Aggregate to obtain For example, taking a weighted average, where the weights can be... Relevance is used to highlight the deviation impact of highly similar candidates.

[0055] To obtain This step utilizes the synonyms generated in step one to expand the matching score. This characterizes the degree of dependence of candidate mapping pairs on synonym expansion. When the high matching scores of candidate mapping pairs mainly come from synonym triggers, especially when the person in charge is not equivalent to the contact person in a specific business line, the probability of the candidate being boosted but actually being a hard negative example increases. Therefore, this implementation quantifies this phenomenon as synonym inflation. And aggregated at the candidate set level to obtain . It can be by and The differences or proportions between them are calculated to reflect the magnitude of the contribution of synonym expansion to the candidate ranking, and the above scores are all derived from the available parameters in step one.

[0056] Synonym inflation The specific calculation formula is as follows:

[0057] ;

[0058] in This is used to prevent the denominator from being zero. This value reflects the additional contribution of synonym expansion to the matching score; a larger value indicates that the candidate mapping pair is more likely to be falsely promoted due to overgeneralization of synonyms. Subsequently, for all candidate pairs in the candidate set... Take a weighted average, with the weights being the corresponding semantic similarity. That is, to obtain .

[0059] To obtain This step introduces contextual proof information from the source field. (Contextual conflict count) This is achieved through rule-based judgment. First, the business domain tags of the source table are extracted from the metadata. If the business domain of the candidate standard field does not match the tag of the source table, then... If other fields in the same table are already mapped to the category of legal entity, and the current candidate field is the category of natural person, then... If there are semantically paired fields in the same table, such as name and ID number, and the current candidate breaks this pairing, then... If there is no obvious conflict, then The business domain tag classification system is built on an enterprise-level data dictionary, containing 12 primary business domains and 36 secondary subdomains, which can be obtained in real time through a metadata service interface. This allows for the acquisition of each candidate pair. Then, take the arithmetic mean. .

[0060] In obtaining the nearest neighbor confusion risk index Then, this step is based on Risk-gated correction is performed on candidate mapping pairs. Specifically, candidate correction weights are calculated for candidate mapping pairs (s,t). ,in Follow Monotonic change makes when At higher levels, entity category deviation Synonym inflation and the amount of context conflict The penalty intensity for candidate scores is increased, thereby suppressing the position of hard negative examples in the candidate ranking; when At lower thresholds, the system maintains normal reliance on fundamental features such as semantic similarity, avoiding unnecessary over-suppression. To ensure a controllable candidate space, a risk gating threshold can also be set. Entity compatibility threshold ,For example , ;when Exceed Furthermore, the entity compatibility of candidate mapping pairs is lower than that of other mapping pairs. When the time is right, the candidate mapping pair is directly eliminated or marked as a low-priority candidate; the thresholds can all be determined by statistical analysis of historical verification data or given by system configuration, and are available parameters.

[0061] Step 3: Risk-driven table-level joint consensus decision-making;

[0062] Step two outputs the set of candidate mapping pairs after risk-gated correction and the nearest-neighbor confusion risk index. Next, this implementation method enters the final decision-making stage. The purpose of this stage is to further distinguish hard negative examples that are semantically similar but have different legal or business implications, and to avoid local optimum errors caused by independent field-by-field selection. To this end, this step, on the one hand, re-ranks the candidate mapping pairs based on hard example discrimination; on the other hand, it performs joint consistency solving on the mapping results of multiple source fields within the same business table or the same entity domain, thereby obtaining table-level consistent, interpretable, and more stable rejection results. It should be noted that the joint consistency decision objective constructed in this step is based on the output of step two. To drive the input, lacking It will be impossible to form a joint objective that adapts to risk, nor will it be possible to adjust the intensity of differentiated constraints on high-risk fields.

[0063] In terms of hard example discrimination and re-ranking, this implementation utilizes historical manually confirmed mapping results to construct training samples and train the hard example discrimination model. The hard example discrimination model is implemented using an XGBoost classifier, and its input features are 9 dimensions, including: , , , , , Risk indices for B, C, and source fields In the training samples, positive examples are historically verified correct mapping pairs, while difficult negative examples are candidate pairs with semantic similarity higher than 0.8 but ultimately rejected manually. The model is trained using a logarithmic loss function, a learning rate of 0.1, a maximum tree depth of 6, 100 iterations, and 10 early stopping iterations. The model output is a difficult example discrimination score between 0 and 1. A higher value indicates that the candidate pair is more likely to be a correct mapping. The model achieves an AUC of 0.94 on the test set, effectively identifying easily confused hard negative examples.

[0064] The training samples contain at least positive example mapping pairs and difficult negative example mapping pairs. Positive example mapping pairs are manually verified correspondences between source fields and standard fields. Difficult negative example mapping pairs are candidate pairs from the candidate sets of Step 1 or Step 2 that have high semantic similarity, string matching features, and data features, but were manually rejected, especially including near-negation scenarios such as names or legal entity names, responsible persons or contact persons, and contact phone numbers or office phone numbers. After training, during the inference phase, a difficult example discrimination score is output for any candidate mapping pair (s,t). The The input features are used to characterize the probability or confidence level that the candidate pair is a correct mapping, and are composed of the available parameters from steps one and two. To facilitate integration with subsequent joint consistency solutions, The value can be in the range [0,1]. The larger the value, the more likely the candidate pair is to be a correct mapping.

[0065] In terms of joint consistency decision-making, this implementation uses the set of source fields within the same business table or the same entity domain as the joint solution object. Let the set of source fields within a certain table be... For each source field Step two has already output its candidate set. and the risk index of confusion with neighboring countries ,in This represents the set of candidate mapping pairs after risk-gated correction. To simultaneously consider semantic similarity, string features, data features, hard case discrimination results, and consistency constraints at the table level, this implementation constructs a risk-driven table-level joint consistency decision objective. Where M represents a mapping scheme, the mapping scheme will map each source field Mapped to a specific standard field t in its candidate set. The table-level joint consistency decision objective can be constructed by summing local scores and adding conflict penalties, as shown in the following formula:

[0066] ;

[0067] in, This represents the joint objective value of mapping scheme M within the table-level range; a larger objective value indicates a better scheme. Indicates that mapping scheme M is the source field. Selected standard fields; Indicates source field The local fusion score between it and the selected standard field; Indicates the degree of conflict or inconsistency of the mapping scheme at the table level; The conflict penalty coefficient can be determined by statistical analysis of historical data or by parameter tuning through a validation set, and is an available parameter.

[0068] Among them, the local fusion score A risk-adaptive fusion approach is employed to reflect the idea that high-risk fields require stronger corrective constraints, while low-risk fields maintain normal semantic similarity. The local fusion score can be calculated using the following formula:

[0069] ;

[0070] in, The partial fusion score of source field s and standard field t; , , , and These are the semantic similarity, string similarity, synonym expansion matching score, data type consistency score, and sample value pattern consistency score obtained in step one and bound to the candidate mapping pairs, respectively. This is the hard case discrimination score output by the hard case discrimination model in this step; , and These are the computable quantities used in step two to reflect entity category deviation, synonym inflation, and contextual contradiction. This is the nearest neighbor confusion risk index output from step two. , , , , , , , , These weighting functions, which vary with the risk index, collectively implement a risk-adaptive fusion strategy: when R(s) is high, the dependence on basic similarity features is reduced, while the dependence on the correction penalty term and the hard example discrimination score is enhanced. As a specific implementation, the weighting function can be implemented using a two-segment linear function. Semantic similarity weights are used as the basis for this strategy. Entity deviation penalty weight For example:

[0071] ,

[0072] ;

[0073] Where θ is the risk threshold, such as θ = 0.6. and To adjust the slope, such as , This can be adjusted based on the validation set. Other weighting functions can be configured using this model to ensure that the sum of all weights remains relatively stable at each risk level.

[0074] By making the weights follow Changes, this implementation method is able to Increase when it is higher , and The corresponding penalty item weights are adjusted and increased. The contribution to the final score, thereby suppressing hard negative examples that are semantically similar but inconsistent with business semantics; When the values ​​are low, the fusion method should be based on semantic similarity and data features to avoid excessive suppression of normal fields.

[0075] The degree of conflict or inconsistency Used to reflect table-level consistency constraints. The calculation can be based on the consistency of the standard field categories, entity categories, or business domains to which each field in the mapping scheme is mapped. Its inputs come from the entity category distribution or entity category labels inferred in step two, and the table structure context information obtained in step one; all are obtainable parameters. For example, when multiple fields in the same table are simultaneously mapped to mutually exclusive standard field semantic categories, or when the same set of source fields should collectively point to the natural person domain in the entity domain but are partially mapped to the legal person domain, Increasing the value lowers the target value of the mapping scheme; conversely, when the mapping scheme forms a more common-sense and business-defined combination within the same table, The smaller the value, the more likely this solution is to be selected as the optimal solution. The calculation of the conflict degree can employ rule-based penalty counting or conflict probability estimation based on statistical models, and the required field entity category information, candidate mapping information, and table context information can all be obtained from existing system data. These are also available parameters.

[0076] After constructing the aforementioned joint consensus decision objective, this implementation method searches within the candidate space to make... Largest mapping scheme and will This serves as the final result within the scope of the table or entity domain. The search employs a greedy algorithm combined with a conflict backtracking strategy: first, all candidate mapping pairs are scored by local fusion. Sort the data in descending order; then, for each source field, select the candidate with the highest score. If this selection conflicts with the selected mapping under table-level consistency constraints, such as entity category conflict or business domain conflict, backtrack and try the second-best candidate. If the number of candidates exceeds 100, use the Beam Search algorithm with a beam width of 5 to approximate the global optimum within a finite time. This search method can complete the solution within 1 second on a real business table (≤50 fields).

[0077] Since the candidate set has converged in step two through risk gating correction, the joint solution can be completed within a controllable scale, significantly reducing the misleading effects of hard negative examples. To facilitate subsequent automatic verification and manual review, this implementation can also output a joint confidence score to characterize the stability of the optimal mapping scheme. The joint confidence score can be obtained from the optimal mapping scheme... and suboptimal mapping scheme The difference in the joint objective value is calculated, for example, by defining... ,in In the candidate space, except The solution with the largest objective value is the one with the largest gap. The larger the gap, the more stable the table-level decision is, and the more suitable it is for automatic confirmation.

[0078] Step 4: Confidence Confirmation and Closed-Loop Update;

[0079] The optimal table-level mapping scheme is obtained in step three. After considering the correlation with the joint confidence level, the results are confirmed and output, ensuring that the missing results can be reliably used in actual production environments and forming a reusable update strategy. The key point of this step is to integrate multidimensional evidence of candidate mapping pairs and the nearest neighbor confusion risk index. Joint consensus decision-making objectives at the table level The system is converted into an executable confirmation strategy, and the valid information obtained from manual review is fed back to update the risk gating and difficult case identification functions. This ensures that the mislabeling caused by neighbor confusion will not change even if factors such as changes in the naming style of the source system fields, changes in business definitions, or adjustments to the standard field library change.

[0080] Specifically, the confidence level of the confirmation output in step three. In this implementation, a corresponding confirmation confidence level is generated for each source field s. The Used to characterize the mapping of source fields s to The reliability of the confidence level is used as the basis for automatic verification or manual review. To ensure that the confidence level reflects the core mechanism of the invention, It considers not only the fusion score of the source fields at the local level, but also the decision stability and risk level at the table level. For example, The local fusion score from step three can be used as a basis. Neighbor confusion risk index The table-level joint objective gap Gap is calculated, where Gap is the difference between the optimal and suboptimal mapping schemes in the joint objective value, i.e. , In the candidate space, except The scheme that maximizes the combined objective value. Therefore, the confidence level can be mapped to a preset interval using a normalization function, as shown in the following formula:

[0081] ;

[0082] in, This indicates the confidence level of the source field s; It is a monotonically increasing normalization function used to map the result to the [0,1] interval, preferably the Logistic function; The local fusion score is calculated in step three; Gap is the table-level decision stability quantity obtainable in step three. The nearest neighbor confusion risk index is the output of step two; , , These are coefficient parameters used to balance the impact of local scores, global consistency, and risk on confidence levels. These coefficients can be determined through logistic regression fitting of historically manually verified mapping results. The specific steps are: collect a batch of labeled source fields, each with a calculated... Gap The values ​​and manually verified binary labels, where 1 indicates correct and 0 indicates doubt or error. For the linear part, a Logistic regression model is used to fit the labels, and the resulting regression coefficients are... , , The recommended value. As an initial configuration, it can be set to... , , .

[0083] Based on the confirmation confidence level In this implementation, the mapping results are divided into two categories: automatically confirmed output and manually reviewed output. For source field s that meets the automatic confirmation conditions, i.e. Not lower than the preset automatic confirmation threshold For example The mapping result is directly confirmed as And output; where The parameters can be statistically determined from historical data under the constraint of the target mislabeling rate, or given by system operation and maintenance configuration, and are therefore obtainable parameters. For source fields that do not meet the automatic confirmation conditions, their candidate sets, corresponding candidate evidence, and explanatory information are submitted for manual review. The explanatory information preferably includes traceable evidence related to nearest-neighbor confusion, such as the field's... The values, entity deviation correlation, synonym expansion correlation, and the triggering reasons corresponding to table-level consistency conflict penalties enable manual review to determine the actual business semantic differences between neighboring candidates such as names and legal person names at a lower cost, thereby reducing the probability of misjudgment due to insufficient information during the review process.

[0084] In the output layer, automatically confirmed mapping results are written into the mapping configuration, establishing a callable indexing relationship between source and standard fields for subsequent data exchange, field transformation, or data governance processes. Simultaneously, the version information and generation basis information corresponding to the mapping result are recorded as traceability and rollback information. Manually confirmed or adjusted mapping results are also written into the mapping configuration, with rejected highly similar candidates and confirmed correct candidates used as closed-loop learning data. The manually confirmed mapping pairs are then stored... Add to the positive example set; select candidate pairs that were rejected during manual review but had high semantic similarity, string features, and data features. Add a set of difficult negative examples for subsequent updates to the difficult example discrimination model and to improve the ability to identify nearest neighbor confusion; use candidate pairs that are raised due to synonym expansion in manual review as synonym risk samples to calibrate synonym-related parameters so that synonym expansion is no longer indiscriminately regarded as a positive recall signal.

[0085] Furthermore, this embodiment utilizes the aforementioned closed-loop data to adaptively update the nearest neighbor confusion risk index and gating mechanism in step two. Because... The weighting coefficients α, β, γ, δ, and the risk gating threshold are obtained by combining computable quantities such as semantic nearest neighbor density, entity category deviation, synonym inflation, and contextual proof. Entity compatibility threshold All methods can be statistically calibrated based on newly added manually reviewed samples, thereby maximizing the range of automatically verifiable fields under the same error rate constraint and maintaining a more stringent correction intensity for high-risk fields. Simultaneously, this implementation also calibrates the weighting function in the risk-driven joint consistency decision objective of step three, enabling local fusion scores to be more consistent across different business table structures and different domain standard field sets. right The adaptive sensitivity is kept within a reasonable range, thus balancing automation rate and accuracy. The data used in the above update process all come from the candidate set generated by the system operation, score parameters, and manual review results, all of which are available data. Therefore, the closed-loop update can be continuously executed and is reproducible.

[0086] Example 2: The design of the field name standard label mapping system based on semantic matching of the present invention is based on the method in Example 1, specifically as follows... Figure 2 The following modules are shown:

[0087] The candidate generation module is used to perform step one: field semantic modeling and candidate generation. It obtains the source field set from the source system to be mapped and the standard field set from the pre-established standard field library. It performs unified normalization processing on the field names and field descriptions of the source fields and standard fields to generate field text representations. When sample values ​​are available, it incorporates representative sample value fragments. When sample values ​​are not directly usable, it extracts the obtainable morphological information of the sample values ​​and writes it into the field text representation. Then, it inputs the field text representation into the pre-trained language model to obtain the field semantic vector representation. Based on semantic similarity, it recalls the semantic candidate set. At the same time, it binds string similarity, synonym expansion matching score, data type consistency score, and sample value pattern consistency score to each candidate mapping pair (s,t), forming the candidate mapping pair set and feature input for subsequent steps two and three.

[0088] The risk correction module is used to perform step two: nearest neighbor confusion risk identification and correction. It establishes a nearest neighbor confusion risk index for each source field to quantify the probability of the source field being confused with its nearest neighbors in its candidate set. It forms a risk judgment basis based on semantic nearest neighbor density parameters, entity category distribution deviation parameters, synonym inflation parameters, and contextual counter-evidence parameters. Then, it performs risk gating correction on candidate mapping pairs based on the nearest neighbor confusion risk index. When the risk is high, it increases the penalty intensity of entity category deviation, synonym inflation, and contextual conflict on the candidate score. It can also combine risk gating threshold and entity compatibility threshold to eliminate or downgrade candidate mapping pairs, thereby outputting the candidate mapping pair set and gating results after risk gating correction.

[0089] The joint decision-making module is used to execute step three: risk-driven table-level joint consistency decision-making. Within the candidate space after risk gating and correction, it re-ranks candidate mapping pairs based on hard case discrimination and outputs hard case discrimination scores. Simultaneously, it constructs a risk-driven table-level joint consistency decision objective and performs joint consistency solution within the same business table or the same entity domain. The local fusion score of the candidate mapping pair (s,t) is adaptively fused based on semantic similarity, string similarity, synonym expansion matching score, data type consistency score, sample value pattern consistency score, hard case discrimination score, entity category deviation, synonym inflation, and contextual counter-evidence correlation. Conflict penalty terms are superimposed at the table level to reflect the consistency constraints of standard field categories, entity categories, or business domains, thereby outputting the optimal table-level mapping scheme and joint confidence correlation.

[0090] The confidence confirmation module is used to execute step four: confidence confirmation and closed-loop update. It generates a confidence level based on the local fusion score in step three, the gap between the table-level joint objectives, and the nearest neighbor confusion risk index output in step two. It distinguishes between automatic confirmation output and manual review output: mapping results that meet the automatic confirmation conditions are directly confirmed and output, while those that do not meet the conditions are submitted for manual review and output traceable explanation information related to nearest neighbor confusion. At the same time, the automatically confirmed or manually reviewed / adjusted mapping results are written into the mapping configuration and the version information and generation basis information are recorded for traceability and rollback. It also accumulates closed-loop learning data such as rejected high similarity candidates and confirmed correct candidates, which are used to update the hard example discrimination ability and calibrate the weight function in the nearest neighbor confusion risk index, gating mechanism, and risk-driven table-level joint consistency decision objective, thereby forming a continuously iterative closed-loop update mechanism.

[0091] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0092] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0093] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for mapping field name standards based on semantic matching, characterized in that, Includes the following steps: Obtain the source field set to be marked and the standard field set in the standard field library; Normalize the source and standard fields and generate text representations of the fields; Semantic vectors are obtained through a pre-trained language model based on field text representation, and a set of candidate mapping pairs is generated for each source field based on the semantic vectors. For each candidate mapping pair, calculate string similarity, synonym expansion matching score, data type consistency score, and sample value pattern consistency score; For each source field, risk discrimination parameters are determined based on its candidate mapping set and a nearest neighbor confusion risk index is generated. The risk discrimination parameters include at least the semantic nearest neighbor density parameter, entity category distribution deviation parameter, synonym inflation parameter, and contextual proof of contradiction parameter. Based on the nearest neighbor confusion risk index, risk gating correction is performed on candidate mapping pairs to obtain a corrected set of candidate mapping pairs. The set of candidate mapping pairs after correction is subjected to hard case discrimination to obtain the hard case discrimination score; Within the same business table or entity domain, based on table-level consistency constraints, a joint consistency solution is performed by integrating semantic similarity, string similarity, synonym expansion matching score, data type consistency score, sample value pattern consistency score, and hard case discrimination score, and the optimal table-level mapping scheme is output. Generate confirmation confidence based on the optimal mapping scheme at the table level, and output the automatically confirmed mapping result or the manually reviewed mapping result based on the confirmation confidence.

2. The field name standard marker mapping method based on semantic matching according to claim 1, characterized in that: Normalization includes performing at least one or more processing steps on field names and field descriptions: case unification, delimiter unification, camelCase splitting, redundant whitespace cleanup, special character cleanup, and domain abbreviation restoration.

3. The field name standard marker mapping method based on semantic matching according to claim 1, characterized in that: The field text representation is generated by combining the field name and field description according to preset concatenation rules, and the field name is used to generate the field text representation when the field description is missing.

4. The field name standard marker mapping method based on semantic matching according to claim 1, characterized in that: Generating a candidate mapping pair set for each source field based on semantic vectors includes: establishing a vector retrieval index for the semantic vectors of standard fields, and performing nearest neighbor recall for the semantic vectors of each source field to obtain the corresponding candidate mapping pair set.

5. The field name standard marker mapping method based on semantic matching according to claim 1, characterized in that: String similarity is calculated based on the field name of the source field and the field name of the standard field; before calculating string similarity, at least one of the following preprocessing steps is performed on the field names: case conversion, delimiter processing, and redundant character cleanup.

6. The field name standard marker mapping method based on semantic matching according to claim 1, characterized in that: The synonym expansion matching score is calculated by matching the replaced or expanded field text representation with the standard field text representation after the synonym library has been used to replace or expand the field text representation.

7. The field name standard marker mapping method based on semantic matching according to claim 1, characterized in that: The data type consistency score is determined based on the data type information of the source field and the data type information of the standard field. The sample value pattern consistency score is determined based on the sample value pattern information of the source field and the sample value pattern information of the standard field. The sample value pattern information includes at least one of the following: character category composition, length characteristics, and preset format hit status.

8. The field name standard marker mapping method based on semantic matching according to claim 1, characterized in that: The risk discrimination parameters include the semantic nearest neighbor density parameter, which is determined based on the degree of difference in semantic similarity among a preset number of candidate mapping pairs in the candidate mapping pair set.

9. The field name standard marker mapping method based on semantic matching according to claim 1, characterized in that: Joint consistency resolution includes: within the same business table or entity domain, determining the conflict of candidate mapping pairs of multiple source fields based on table-level consistency constraints, and applying conflict penalty processing to candidate mapping pairs with consistency conflicts in order to obtain the optimal table-level mapping scheme.

10. A field name standard marker mapping system based on semantic matching, characterized in that, The mapping system is used to implement the method according to any one of claims 1-9, and includes the following modules: The candidate generation module is used to obtain the set of source fields to be targeted and the set of standard fields in the standard field library; to perform normalization processing on the source fields and standard fields and generate field text representations; Semantic vectors are obtained through a pre-trained language model based on field text representation, and a set of candidate mapping pairs is generated for each source field based on the semantic vectors. The risk correction module calculates string similarity, synonym expansion matching score, data type consistency score, and sample value pattern consistency score for each candidate mapping pair. For each source field, risk discrimination parameters are determined based on its candidate mapping set and a nearest neighbor confusion risk index is generated. The risk discrimination parameters include at least the semantic nearest neighbor density parameter, entity category distribution deviation parameter, synonym inflation parameter, and contextual proof of contradiction parameter. Based on the nearest neighbor confusion risk index, risk gating correction is performed on candidate mapping pairs to obtain a corrected set of candidate mapping pairs. The joint decision-making module is used to perform hard case discrimination on the corrected candidate mapping pair set and obtain the hard case discrimination score. Within the same business table or entity domain, based on table-level consistency constraints, a joint consistency solution is performed by integrating semantic similarity, string similarity, synonym expansion matching score, data type consistency score, sample value pattern consistency score, and hard case discrimination score, and the optimal table-level mapping scheme is output. The confidence confirmation module is used to generate a confirmation confidence level based on the optimal mapping scheme at the table level, and output the automatically confirmed mapping result or the manually reviewed mapping result based on the confirmation confidence level.

Images