Intelligent data benchmarking method and system based on cooperation of large model and small model
By employing an intelligent data benchmarking method that combines large and small models, the problems of low efficiency and insufficient accuracy in existing technologies have been solved. This approach enables efficient and accurate data benchmarking and autonomous standard discovery, thereby constructing a sustainable and optimized enterprise data governance system.
Patent Information
- Application Number
- CN202511677553.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-17
- Publication Date
- 2026-02-10
AI Technical Summary
Existing data benchmarking methods are inefficient and inaccurate, rely on human experience and are difficult to accumulate knowledge, cannot effectively handle complex naming and semantic ambiguity, and lack the ability to independently discover data standards.
We employ an intelligent data benchmarking method that combines large and small models. We construct a basic data standard library through feature engineering of all historical data, and combine semantic vector similarity calculation and a two-stage training mechanism to achieve automated field benchmarking. We also optimize the model through human-machine collaboration.
Significantly improve the speed and accuracy of data benchmarking, build a sustainable and self-optimizing intelligent benchmarking system, reduce corporate governance costs, and achieve efficient and accurate data standard discovery and management.
Smart Images
Figure CN121502379A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of data processing, and particularly relates to an intelligent data benchmarking method and system based on cooperation of a large model and a small model. BACKGROUND
[0002] In enterprise-level data governance practice, the existing data benchmarking methods have the following deficiencies:
[0003] (1) Pure manual benchmarking mode: This is the most original way, which is manually compared by data analysts or business experts according to experience. The disadvantages of this mode are:
[0004] Low efficiency: In the face of thousands or even millions of data fields in an enterprise, manual investigation is time-consuming and labor-intensive, and the project cycle is long.
[0005] Strong subjectivity and unstable accuracy: The judgment of experts is easily affected by personal knowledge background and understanding bias. For complex cases of "same name different meaning" or "same meaning different name", it is easy to misalign or miss.
[0006] High risk of knowledge loss: The benchmarking logic and experience exist in the individual brain of experts, and cannot form systematic knowledge assets. Once the personnel change, the core knowledge is at risk of loss.
[0007] (2) Semi-automatic mode based on rules or keyword matching: This mode preliminarily screens through a pre-set rule library or simple string matching. Its disadvantages are:
[0008] Poor flexibility: The rule library is difficult to maintain, and cannot exhaust all semantic changes. It has weak processing capability for non-standard naming (such as pinyin, abbreviation, and Chinese-English mixed).
[0009] Lack of semantic understanding: It cannot understand the context, and only relies on literal matching, which will lead to a large number of ambiguities and limited accuracy.
[0010] (3) Single AI model mode: In recent years, some schemes have tried to use a single semantic model for benchmarking. Its disadvantages are:
[0011] There are semantic ambiguities: Simply relying on semantic similarity cannot distinguish the subtle differences in business connotation. For example, the source field "amount" and the target standard "transaction amount" and "credit amount" are highly similar in semantics, and the model has difficulty in making accurate judgments.
[0012] Lack of factual basis verification: It cannot cross-verify combined with the characteristics of the data content itself, resulting in insufficient reliability of the matching results.
[0013] Passive governance: existing methods all assume the existence of a complete standard library. When the standard library is not perfect, they cannot actively discover and define new data standards, and the governance capability is limited.
[0014] In summary, in the existing benchmarking methods, the standard discovery method is to find the field Chinese name and the standard name consistent or the semantic similarity is high, ignoring the important role of the data format, code value interpretation and inventory benchmarking experience in benchmarking, so the accuracy is low.
[0015] Therefore, it is necessary to provide an intelligent data benchmarking method and system based on large model and small model cooperation, which can greatly improve the speed and accuracy of data benchmarking and reduce the cost of enterprise data governance. SUMMARY
[0016] The purpose of the present application is to provide an intelligent data benchmarking method and system based on large model and small model cooperation, which can greatly improve the speed and accuracy of data benchmarking and reduce the cost of enterprise data governance.
[0017] In order to solve the problems existing in the prior art, the present application provides an intelligent data benchmarking method based on large model and small model cooperation, comprising the following steps:
[0018] S1: data access and full historical data feature engineering, based on feature construction, building a basic data standard full search library, a data standard sub-search library and a code value search library;
[0019] S2: pre-screening of the field to be benchmarked, comprising the following steps:
[0020] S21: semantic normalization and vectorization of the field to be benchmarked;
[0021] S22: based on the basic data standard full search library, the data standard sub-search library and the code value search library, pre-screening the field to be benchmarked to obtain a candidate data standard set;
[0022] S3: calculating the semantic vector similarity of the field to be benchmarked and the candidate data standard set, ranking according to the similarity score from high to low to form a final data standard recommendation list:
[0023] S31: using a two-stage training mechanism, first pre-training a basic segmentation model based on general corpus, then using a standard name database for domain fine-tuning, completing the domain customized segmentation processing of the field to be benchmarked and the candidate data standard set;
[0024] S32: Parallelly generate two kinds of semantic representations of the to-be- benchmarked field and the candidate data standard set after the word segmentation processing, to obtain the global semantic vector of the to-be-benchmarked field at the original text level and the local semantic matrix of the subword level after the word segmentation, and to obtain the global semantic vector of the candidate data standard set at the original text level and the local semantic matrix of the subword level after the word segmentation;
[0025] And a mapping relationship table of subwords and original texts is established to retain context association information and to construct hierarchical semantic representations;
[0026] S33: The cosine similarity of the global semantic vector of the to-be-benchmarked field and the global semantic vector of all data standards of the candidate data standard set is calculated in turn, and the obtained cosine similarity is defined as a global score, and the TopN similar candidate data standards are obtained according to the cosine similarity score ranking;
[0027] S34: The cosine similarity of all subword vectors of the to-be-benchmarked field and all subword vectors of each data standard in the TopN similar candidate data standards is calculated to form an m×n similarity matrix, where m is the number of subword vectors of the to-be-benchmarked field, and n is the number of subword vectors of any candidate data standard;
[0028] The maximum value is taken for each row of the similarity matrix to obtain a highest matching score set {msxs 1j ,msxs 2j ,…,msxs mj} of each subword of the to-be-benchmarked field and each subword of the candidate data standard; and the local similarity score is calculated based on the highest matching score set, and the calculation formula is as follows:
[0029]
[0030] Wherein, score is the local similarity score, is the length of the subword , i is any subword vector of the to-be-benchmarked field, and m is the number of subword vectors of the to-be-benchmarked field;
[0031] S35: The similarity score is calculated based on the local semantic matching score and the global score, and the calculation formula is:
[0032] The first weight×local semantic matching score+second weight×global score.
[0033] Optionally, in the intelligent data benchmarking method based on the cooperation of large models and small models, the data access mode is:
[0034] S11: Connect the pre-set multiple data adapters to the source data systems inside and outside the enterprise, and automatically extract the historical full data set from various data adapters to form the historical full data set, which contains the metadata and sample data of the full data items.
[0035] Optionally, in the intelligent data matching method based on the cooperation of large models and small models, the full historical data feature engineering is performed in the following manner:
[0036] S12: Multi-dimensional feature calculation, deep feature engineering is performed on the accessed historical full data set, and a comprehensive portrait is constructed for each data item, which is formed by splicing metadata features, data content features and data format features;
[0037] S13: Intelligent extraction of data standards, the steps are as follows:
[0038] S131: Based on the density-based spatial clustering algorithm, a plurality of data item clusters are obtained by intelligent clustering analysis;
[0039] S132: The large language model uses Qwen2.5-72B large model as the core processing engine, constructs a field enhanced analysis framework by connecting an industry standard knowledge base, and the large language model designs a prompt project based on the basic naming logic of data standard name and the extraction requirements of technical attributes;
[0040] For each data item cluster, the large language model is used to analyze the text information of all members in the data item cluster, and the basic data standard is intelligently generated, which includes data standard name, business meaning and technical attribute;
[0041] S133: The name of each data standard in the historical full data set is deeply vectorized by using the BGE-m3 multi-language embedding model, and then the Annoy algorithm is used to construct a vector retrieval library as a full retrieval library of the basic data standard;
[0042] The full retrieval library of the basic data standard is divided according to the technical attributes, and a sub-vector retrieval library is constructed based on data standards with the same technical attributes, thereby cutting into a plurality of data standard sub-retrieval libraries;
[0043] For the enumerated data standard, the code value is vectorized based on the Annoy algorithm, and a vector retrieval library is constructed as a code value retrieval library.
[0044] Optionally, in the intelligent data matching method based on the cooperation of large models and small models,
[0045] The metadata features include structured field name, annotation, data type, data length and data precision;
[0046] Statistical analysis is performed on the sample data to obtain data content features, and the data content features include maximum value, minimum value, mean value, unique value proportion, null rate and enumeration value;
[0047] Adopting the regular expression library, specific format is identified and labeled to form data format features, including ID number, mobile phone number and date.
[0048] Optionally, in the intelligent data benchmarking method based on the cooperation of large and small models, the density-based spatial clustering algorithm is used to obtain data item clusters through intelligent clustering analysis in the following manner:
[0049] Apply the density-based spatial clustering algorithm to the historical full data set after constructing a comprehensive portrait to generate an initial cluster set; the density-based spatial clustering algorithm is Density-Based Spatial Clustering of Applications with Noise, abbreviated as DBSCAN algorithm.
[0050] Select the cluster with the largest cardinality from the initial cluster set, add the cluster with the largest cardinality to the final cluster set, and delete the cluster with the largest cardinality from the historical full data set to form a residual cluster set;
[0051] Based on the residual cluster set, use the DBSCAN algorithm to obtain a second initial cluster set, select the cluster with the largest cardinality from the second initial cluster set, add it to the final cluster set, and delete it from the historical full data set to form a second residual cluster set;
[0052] Iterative processing, constantly selecting the cluster with the largest cardinality to add it to the final cluster set and updating the residual cluster set, until the latest residual cluster set no longer generates any cluster and / or cannot form a new cluster after applying the DBSCAN algorithm, the clustering iteration is ended, and the final cluster set formed is the data item cluster.
[0053] Optionally, in the intelligent data benchmarking method based on the cooperation of large and small models, the candidate data standard set is obtained through pre-screening in the following manner:
[0054] Through the text semantic description of the source field of the field to be benchmarked, similarity retrieval is performed in the basic data standard full retrieval library, and the first three candidate data standards with the most similar name semantics are selected to form a first candidate data standard set, defined as D1.
[0055] Based on the data sample characteristics, data type characteristics and code value of the to-be-identified field, the technical attribute of the to-be-identified field is determined, a data standard sub-retrieval library consistent with the technical attribute of the to-be-identified field is queried, and the first three candidate data standards with the closest semantics are screened out from the data standard sub-retrieval library obtained by the query to form a second candidate data standard set, defined as D2; if the to-be-identified field has a code value attribute, the data standard sub-retrieval library with the closest semantic code value is selected from the code value retrieval library, and the first three candidate data standards with the closest semantics are screened out from the selected data standard sub-retrieval library to form a second candidate data standard set;
[0056] If the to-be-identified field has a same table and same field in the historical identification record, the corresponding data standard of the history becomes a third candidate data standard set, defined as D3.
[0057] The calculation formula of the candidate data standard set is: candidate data standard set = U (D1, D2, D3).
[0058] Optionally, in the intelligent data identification method based on the cooperation of the large model and the small model, before S3, the following steps are included: comparing the core features of the to-be-identified field and the candidate standards in the candidate data standard set, and eliminating the candidate data standard if any item does not meet the requirement, wherein the core features include data type consistency, data length consistency, data precision consistency and enumeration value intersection proportion.
[0059] Optionally, in the intelligent data identification method based on the cooperation of the large model and the small model, in S3, if there is a historical identification, the corresponding data standard of the history is returned as the only identification result.
[0060] Optionally, in the intelligent data identification method based on the cooperation of the large model and the small model, the following steps are further included:
[0061] S4: human-computer cooperation confirmation, and learning optimization based on artificial operation feedback.
[0062] The application also provides an intelligent data identification system based on the cooperation of a large model and a small model, which adopts the intelligent data identification method.
[0063] Compared with the prior art, the application has the following advantages:
[0064] (1) The application provides an automatic intelligent data identification method, which aims to greatly improve the speed and accuracy of data identification, reduce the cost of enterprise data governance, and build a sustainable self-optimizing intelligent identification system. The two-stage mode of "large model coarse screening + small model accurate judgment" improves the automation rate and accuracy of identification, and the efficiency can be improved by more than 90%.
[0065] (2) The present application also aims to solve the problem of incomplete or missing enterprise data standard library, breaking through the limitation of passive mapping, and intelligently discovering fact standards from massive and heterogeneous stock data in a "bottom-up" manner, which greatly enriches and perfects the enterprise data standard library and accelerates the data governance process.
[0066] (3) The present application has strong generalization and adaptability: the introduction of large language models enables the present application to effectively handle complex situations such as non-standard naming, Chinese-English mixing, and synonyms, with strong generalization ability.
[0067] (4) The present application has objectivity and explainability of decision-making: the decision of the small model is based on a series of explicit and quantitative features, avoiding the ambiguity of single semantic models, and providing explainable basis for matching results to some extent.
[0068] (5) The present application realizes the sedimentation and evolution of knowledge: through the feedback learning loop of human-computer collaboration, the implicit knowledge of experts is solidified into the model, and it can continuously learn and self-optimize, building an enterprise-level and evolving data benchmarking knowledge base. BRIEF DESCRIPTION OF DRAWINGS
[0069] Figure 1 Flowchart of the intelligent data benchmarking method provided by the embodiments of the present application; DETAILED DESCRIPTION
[0070] The specific embodiments of the present application will be described in more detail below with reference to the accompanying drawings. The advantages and features of the present application will be clearer according to the following description. It should be noted that the drawings are greatly simplified and use non-precise proportions, only to facilitate and clarify the purpose of assisting the description of the embodiments of the present application.
[0071] In the following, if the method described herein includes a series of steps, the order of these steps presented herein is not necessarily the only order in which these steps can be performed, and some of the described steps can be omitted and / or some other steps not described herein can be added to the method.
[0072] The existing data benchmarking work has the problems of low efficiency, insufficient accuracy, high dependence on artificial expert experience, and difficulty in knowledge sedimentation and reuse.
[0073] In order to solve the problems existing in the prior art, the present application provides an intelligent data benchmarking method based on the cooperation of large models and small models.
[0074] The data standard can be understood as follows: a unified rule defined in data governance to ensure the consistency, accuracy and reliability of data, including data format, value range, business meaning and other specifications. For example, for the data item "mobile phone number", the data standard can be defined as:
[0075] Data standard name: mobile phone number;
[0076] Business definition: personal real-name mobile phone number;
[0077] Technical format: data string, fixed 11-bit length;
[0078] Management department: xxx department;
[0079] …
[0080] The data standard can be understood as follows: comparing the actual data with the data standard, and assigning the data standard to the actual data. The actual data needs to be evaluated and corrected according to the assigned data standard to ensure that the data meets the specified specifications. For example, there is a data standard "customer name", and its technical format is Chinese and English string, not more than 50 bits long. When data standardizing, it is found that there is a field named "customer name" in a certain table, and its data format length is defined as 60. It is identified that this field should correspond to the data standard "customer name", and the name should be standardized to "customer name" and the field length should be standardized to 50 to meet the standard requirements.
[0081] As shown in Figure 1 The intelligent data standardization method includes the following steps:
[0082] S1: data access and full historical data feature engineering, based on feature construction, build a full search library of basic data standards, data standard sub-search library and code value search library;
[0083] The data access method is:
[0084] S11: connect the pre-set multiple data adapters to the source data systems inside and outside the enterprise, and automatically extract the historical full data set from various data adapters. The historical full data set contains the metadata and sample data of the full data item.
[0085] The way to do full historical data feature engineering is as follows:
[0086] S12: multi-dimensional feature calculation, deep feature engineering on the accessed historical full data set, build a comprehensive portrait for each data item, the portrait is formed by metadata features, data content features and data format features coding splicing, splicing and building a single field long feature, such as field name coding + annotation coding + data type coding + … + data content distribution coding + ….
[0087] Metadata characteristics include structured field names, comments, data types, data length, and data precision;
[0088] Statistical analysis of the sample data yields data content characteristics, including maximum value, minimum value, mean, percentage of unique values, null value rate, and enumerated values.
[0089] A regular expression library is used to identify and label specific formats to form data format features, including ID card numbers, mobile phone numbers, and dates.
[0090] S13: Intelligent extraction of data standards, the steps are as follows:
[0091] S131: Density-based spatial clustering algorithm. Intelligent clustering analysis yields multiple data item clusters. Using an unsupervised clustering algorithm, it calculates all data items with completed feature construction, automatically aggregating data items with similar features such as English names, data types, and data sample distribution in the fields to form multiple "data item clusters".
[0092] Specifically, the method for obtaining a single data item cluster is as follows:
[0093] Apply a density-based spatial clustering algorithm to the historical full dataset D after constructing a comprehensive profile, generating an initial cluster set C = {C1, C2, ..., Cn}, and calculate the cardinality |Cn| of each cluster. i |(Including the number of points); The density-based spatial clustering algorithm is called Density-Based Spatial Clustering of Applications with Noise, or DBSCAN for short. Its core idea is to discover data item clusters of arbitrary shapes by using the local density of data points.
[0094] Select the cluster with the largest cardinality from the initial cluster set C: C_max = argmax(|C i |), add the cluster with the largest cardinality C_max to the final cluster set F, and delete the cluster with the largest cardinality C_max from the historical full dataset D to form the remaining cluster set D'=D-C_max;
[0095] Based on the remaining cluster set D', the DBSCAN algorithm is used to obtain the second initial cluster set. The cluster with the largest cardinality is selected from the second initial cluster set and added to the final cluster set. The cluster with the largest cardinality is then deleted from the remaining cluster set D' to form the second remaining cluster set.
[0096] The process involves iteratively selecting the cluster with the largest cardinality to add to the final cluster set and updating the remaining cluster set. This continues until the latest remaining cluster set no longer generates any clusters after applying the DBSCAN algorithm (i.e., all remaining points are identified as noise points) and / or cannot form new clusters (i.e., the number of points in D' is less than MinPts). At this point, the clustering iteration ends, and the final cluster set formed is the data item cluster.
[0097] DBSCAN is a density-based clustering algorithm. Its core principle is that a "cluster" is the largest set of density-connected points in the data space. It identifies core points by defining a "neighborhood radius (eps)" and a "minimum number of points (minPts)": a point is a core point if its eps neighborhood contains at least minPts points. Points within the neighborhood of a core point are grouped into the same cluster, and this clustering expands to form clusters of arbitrary shapes. Points not belonging to any cluster are marked as noise. This algorithm does not require pre-specifying the number of clusters, is insensitive to noisy data, and can effectively discover cluster structures of arbitrary shapes.
[0098] S132: The large language model adopts the Qwen2.5-72B large model as the core processing engine. It constructs a domain-enhanced analysis framework by connecting to an external industry standard knowledge base (including financial industry standards, public data meta-files, industry technical specifications and other documents). The large language model is designed based on the basic naming logic of data standard names and the extraction requirements of technical attributes.
[0099] For each data item cluster, the large language model is used to analyze the text information of all members within the data item cluster and intelligently generate basic data standards. The basic data standards include data standard name, business meaning and technical attributes.
[0100] The input data provided to the large language model are:
[0101]
[0102] The large model combines an industry-standard knowledge base to extract a set of basic data standards, as shown in the following example:
[0103] Basic Standard Name: Customer Number
[0104] Business definition: refers to a coded identifier generated by a financial institution's information system according to unified rules, used to uniquely and non-repeatably identify a customer entity across the entire domain.
[0105] Technical attributes: Fixed 50 characters, only numbers [0-9].
[0106] S133: The BGE-m3 multilingual embedding model is used to perform deep vectorization representation of the name of each data standard in the historical full dataset, and then a vector retrieval library is built based on the Annoy algorithm as the basic data standard full retrieval library.
[0107] The basic data standard full retrieval library is divided according to technical attributes, and a sub-vector retrieval library is constructed based on data standards with the same technical attributes, thereby dividing it into multiple data standard sub-retrieval libraries;
[0108] For data standards specifically for enumeration classes, the code values are vectorized based on the Annoy algorithm, and a vector retrieval library is constructed as the code value retrieval library.
[0109] Ultimately, the model extracts a basic data standard full-scale retrieval library, a series of data standard sub-retrieval libraries with the same technical attributes, and a code value retrieval library from the full historical data.
[0110] S2: Pre-screening of the target fields based on a large language model, including the following steps:
[0111] S21: Perform semantic normalization and vectorization on the target fields;
[0112] For the name of the field to be benchmarked, the Large Language Model (LLM) is first used to process the name and text description information of the field to be benchmarked into a standardized business meaning; then the BGE-m3 multilingual embedding model is used to convert the standardized text description and code value description (if any) into a high-dimensional semantic vector of the field to be benchmarked.
[0113] S22: Based on the full retrieval library of basic data standards, the sub-retrieval library of data standards, and the code value retrieval library, the fields to be benchmarked are pre-screened to obtain a set of candidate data standards, as follows:
[0114] By performing a similarity search in the full retrieval library of basic data standards based on the textual semantic description of the source field to be benchmarked, the top 3 candidate data standards with the most similar names and semantics are selected to form the first candidate data standard set, which is defined as D1.
[0115] Based on the data sample characteristics, data type characteristics, and code values of the target field, the technical attributes of the target field are determined. A data standard sub-retrieval library with the same technical attributes as the target field is queried. The top three candidate data standards with the closest semantics are selected from the retrieved data standard sub-retrieval library to form the second candidate data standard set, defined as D2. If the target field has a code value attribute, the data standard sub-retrieval library with the closest code value semantics is first selected from the code value retrieval library. Then, the top three candidate data standards with the closest semantics are selected from the selected data standard sub-retrieval library to form the second candidate data standard set.
[0116] If the field to be benchmarked has a benchmarking result in the same table and field in the historical benchmarking records, then the corresponding historical data standard becomes the third candidate data standard set, defined as D3;
[0117] The formula for calculating the candidate data standard set is: Candidate data standard set = ∪(D1, D2, D3).
[0118] Compare the core features of the field to be benchmarked with the core features of the candidate criteria in the candidate data standard set. If any one of these features is not met, the candidate data will be removed. The core features include: data type consistency, data length consistency, data precision consistency, and the proportion of intersection of enumerated values.
[0119] S3: Calculate the semantic vector similarity between the field to be benchmarked and the candidate data standard set, rank them from high to low according to the similarity score, and form the final data standard recommendation list: if there is a historical benchmark, return the corresponding historical data standard as the unique benchmark result.
[0120] S31: A two-stage training mechanism is adopted. First, a basic word segmentation model is pre-trained based on a general corpus. Then, a standard name database is used for domain fine-tuning to complete the domain-customized word segmentation processing of the target field and the candidate data standard set.
[0121] This step establishes a professional terminology protection strategy to ensure that key information such as standard numbers and technical parameters are preserved through complete word segmentation. The pre-trained word segmentation model used is the BGE-m3 word segmentation model.
[0122] S32: Generate two semantic representations in parallel with the target field after word segmentation and the candidate data standard set. This yields the original text-level global semantic vector of the target field and the sub-word-level local semantic matrix after word segmentation. It also yields the original text-level global semantic vector and the sub-word-level local semantic matrix of the candidate data standard set.
[0123] A mapping table between subwords and the original text is established, retaining contextual information to construct a hierarchical semantic representation;
[0124] S33: Calculate the cosine similarity between the global semantic vector of the target field (i.e., the target field is directly input into the pre-trained vector representation model + average pooling to obtain the sentence vector) and the global semantic vector (i.e., sentence vector) of all data standards in the candidate data standard set. Define the obtained cosine similarity as the global score, and sort according to the cosine similarity score to obtain the Top N similar candidate data standards.
[0125] S34: Calculate the cosine similarity between all sub-word vectors of the target field and all sub-word vectors of each of the TopN similar candidate data standards, forming an m×n similarity matrix S, where m is the number of sub-word vectors of the target field and n is the number of sub-word vectors of any candidate data standard; take the maximum value of each row of the similarity matrix S to reflect the best matching situation of each sub-word in another field. This similarity matrix S captures all possible local semantic correspondences between texts.
[0126] Subsequently, a maximum similarity summation strategy is employed to enhance matching robustness. The maximum value is taken row-wise from the similarity matrix to obtain the set of highest matching scores {msxs} for each sub-word of the target field and each sub-word of the candidate data standard. 1j msxs 2j ,…,msxs mj To avoid common function words having too much influence on the results, the scores in this set will be weighted and summed according to the length of their corresponding subwords in the original text. Therefore, the local similarity score is calculated based on the set of highest matching scores, and the calculation formula is as follows:
[0127]
[0128] Wherein, score is the local similarity score. For the word The length of , i is any sub-word vector of the field to be benchmarked, and m is the number of sub-word vectors of the field to be benchmarked;
[0129] S35: Calculate the similarity score based on the local semantic matching score and the global score. The calculation formula is as follows:
[0130] First weight × local semantic matching score + second weight × global score.
[0131] In one embodiment, an example of a semantic similarity calculation method is provided:
[0132] Input text 1: Customer type;
[0133] Input text 2: Customer type code;
[0134] Word segmentation:
[0135] Input text 1 - after word segmentation: Customer type;
[0136] Input text 2 - after word segmentation: Customer type, code;
[0137] Vectorization:
[0138] For input text 2, we obtain vector representations of "customer type code", "customer type", and "code" respectively;
[0139] Local semantic similarity calculation:
[0140] Matrix multiplication (vertical axis represents the words after word segmentation of input text 1, horizontal axis represents the words after word segmentation of input text 2) first takes the maximum value by row. Since input text 1 has only one word after word segmentation, i.e.,
[100] , then performs a weighted sum based on the word length by column, and the final result is also 100:
[0141] Customer type code;
[0142] Customer type 100 60;
[0143] Global semantic similarity calculation:
[0144] The vector representations of "customer type" and "customer type code" are calculated based on cosine similarity, which is 92. That is, the two original input texts "customer type" and "customer type code" are respectively fed into the vector representation model + average pooling to obtain the sentence vectors of these two words (i.e. vectorized vector representations). Then, the two sentence vectors are scored based on cosine similarity, resulting in 92.
[0145] Global and local integration:
[0146] 0.6*100+0.4*92=96.8, which is the final similarity result.
[0147] Optionally, the intelligent data benchmarking method based on the collaboration of large and small models further includes the following steps:
[0148] S4: Human-machine collaborative confirmation, and learning and optimization based on human operation feedback.
[0149] Specifically: Results push and review: The system will push the recommended data standard matching list with confidence scores to data governance experts through a visual interface.
[0150] Manual confirmation or correction: Experts perform the final "confirmation" or "correction" operation on the recommendation list.
[0151] Closed-loop feedback and incremental model training: Every action of the expert is recorded as new labeled data, which is used to incrementally train the small model in step three, so that the model's decision-making ability is continuously optimized.
[0152] This invention also provides an intelligent data benchmarking system based on the collaboration of large and small models. The intelligent data benchmarking system is constructed using the intelligent data benchmarking method described above. The system includes: a data access module, a feature calculation module (containing a data standard extraction unit), an intelligent benchmarking module (containing a large model pre-screening unit and a small model verification unit), and a human-machine collaboration module. Each module works together to execute the method disclosed in this invention.
[0153] In summary, compared with the prior art, the present invention has the following advantages:
[0154] (1) This invention provides an automated intelligent data benchmarking method, aiming to significantly improve the speed and accuracy of data benchmarking, reduce enterprise data governance costs, and build a sustainable, self-optimizing intelligent benchmarking system. It achieves a dual improvement in efficiency and accuracy: the two-stage model of "large model coarse screening + small model fine judgment" takes into account both breadth and depth, realizing efficient, accurate, and adaptive benchmarking and standard discovery, with an efficiency improvement of over 90%.
[0155] (2) This invention also aims to solve the problem of incomplete or missing enterprise data standard library, break through the limitations of passive mapping, and can intelligently discover factual standards from massive and heterogeneous existing data in a "bottom-up" manner. This greatly enriches and improves the enterprise data standard library and accelerates the data governance process.
[0156] (3) This invention has strong generalization and adaptability: The introduction of the large language model enables this invention to effectively handle complex situations such as non-standard naming, mixed Chinese and English, and synonyms, and has strong generalization ability.
[0157] (4) The present invention has the objectivity and interpretability of decision-making: the decision of the small model is based on a series of clear and quantifiable features, avoiding the ambiguity of a single semantic model, and to a certain extent providing an interpretable basis for the matching results.
[0158] (5) This invention realizes the accumulation and evolution of knowledge: through the feedback learning loop of human-machine collaboration, the implicit knowledge of experts is solidified into the model, and it can continuously learn and self-optimize, thus constructing an enterprise-level, evolvable data benchmarking knowledge base.
[0159] The above are merely preferred embodiments of the present invention and do not constitute any limitation on the present invention. Any equivalent substitutions or modifications made by those skilled in the art to the technical solutions and content disclosed in the present invention without departing from the scope of the present invention shall be deemed to have remained within the protection scope of the present invention.
Claims
1. A smart data benchmarking method based on the collaboration of large and small models, characterized in that, Includes the following steps: S1: Data access and full-scale historical data feature engineering are performed. Based on the features, a full-scale retrieval library of basic data standards, a sub-retrieval library of data standards, and a code value retrieval library are constructed. S2: Pre-filter the fields to be compared, including the following steps: S21: Perform semantic normalization and vectorization on the target fields; S22: Based on the full retrieval library of basic data standards, the sub-retrieval library of data standards, and the code value retrieval library, the fields to be benchmarked are pre-screened to obtain a set of candidate data standards; S3: Calculate the semantic vector similarity between the field to be benchmarked and the candidate data standard set, rank them from high to low according to the similarity score, and form the final data standard recommendation list: S31: A two-stage training mechanism is adopted. First, a basic word segmentation model is pre-trained based on a general corpus. Then, a standard name database is used for domain fine-tuning to complete the domain-customized word segmentation processing of the target field and the candidate data standard set. S32: Generate two semantic representations in parallel with the target field after word segmentation and the candidate data standard set. This yields the original text-level global semantic vector of the target field and the sub-word-level local semantic matrix after word segmentation. It also yields the original text-level global semantic vector and the sub-word-level local semantic matrix of the candidate data standard set. A mapping table between subwords and the original text is established, retaining contextual information to construct a hierarchical semantic representation; S33: Calculate the cosine similarity between the global semantic vector of the field to be benchmarked and the global semantic vector of all data standards in the candidate data standard set in turn. Define the obtained cosine similarity as the global score and sort them according to the cosine similarity score to obtain the Top N similar candidate data standards. S34: Calculate the cosine similarity between all sub-word vectors of the target field and all sub-word vectors of each of the Top N similar candidate data standards, forming an m×n similarity matrix, where m is the number of sub-word vectors of the target field and n is the number of sub-word vectors of any candidate data standard; By taking the maximum value of each row in the similarity matrix, we obtain the set of highest matching scores for each sub-word of the target field and each sub-word of the candidate data standard. The local similarity score is calculated based on the set of highest matching scores, using the following formula: Where, score is the local similarity score. For the word The length of , i is any sub-word vector of the field to be benchmarked, and m is the number of sub-word vectors of the field to be benchmarked; S35: Calculate the similarity score based on the local semantic matching score and the global score. The calculation formula is as follows: First weight × local semantic matching score + second weight × global score.
2. The intelligent data benchmarking method based on the collaboration of large and small models as described in claim 1, characterized in that, The data access method is as follows: S11: Connects various pre-built data adapters to source data systems inside and outside the enterprise. Various data adapters automatically extract and form a historical full dataset, which contains metadata and sample data of all data items.
3. The intelligent data benchmarking method based on the collaboration of large and small models as described in claim 2, characterized in that, The following are the methods for performing feature engineering on all historical data: S12: Multi-dimensional feature calculation, deep feature engineering is performed on the accessed historical full dataset to build a comprehensive profile for each data item. The profile is formed by encoding and splicing metadata features, data content features and data format features. S13: Intelligent extraction of data standards, the steps are as follows: S131: Density-based spatial clustering algorithm, intelligent clustering analysis yields multiple data item clusters; S132: The large language model adopts the Qwen2.5-72B large model as the core processing engine, and builds a domain-enhanced analysis framework by connecting to an external industry standard knowledge base. The large language model is designed based on the basic naming logic of data standard names and the extraction requirements of technical attributes. For each data item cluster, the large language model is used to analyze the text information of all members within the data item cluster and intelligently generate basic data standards. The basic data standards include data standard name, business meaning and technical attributes. S133: The BGE-m3 multilingual embedding model is used to perform deep vectorization representation of the name of each data standard in the historical full dataset, and then a vector retrieval library is built based on the Annoy algorithm as the basic data standard full retrieval library. The basic data standard full retrieval library is divided according to technical attributes, and a sub-vector retrieval library is constructed based on data standards with the same technical attributes, thereby dividing it into multiple data standard sub-retrieval libraries; For data standards specifically for enumeration classes, the code values are vectorized based on the Annoy algorithm, and a vector retrieval library is constructed as the code value retrieval library.
4. The intelligent data benchmarking method based on the collaboration of large and small models as described in claim 3. Its features are, Metadata characteristics include structured field names, comments, data types, data length, and data precision; Statistical analysis of the sample data yields data content characteristics, including maximum value, minimum value, mean, percentage of unique values, null value rate, and enumerated values. Using a regular expression library, specific formats are identified and labeled to form data format features, including ID card number, mobile phone number, and date.
5. The intelligent data benchmarking method based on the collaboration of large and small models as described in claim 3, characterized in that, Density-based spatial clustering algorithms intelligently cluster data items as follows: Density-based spatial clustering of applications with noise is applied to the historical full dataset after the comprehensive profile is constructed to generate an initial cluster set. The density-based spatial clustering algorithm is called Density-Based Spatial Clustering of Applications with Noise, or DBSCAN algorithm for short. Select the cluster with the largest cardinality from the initial cluster set, add the cluster with the largest cardinality to the final cluster set, and delete the cluster with the largest cardinality from the historical full dataset to form the remaining cluster set; Based on the remaining cluster set, the DBSCAN algorithm is used to obtain the second initial cluster set. The cluster with the largest cardinality is selected from the second initial cluster set and added to the final cluster set. The cluster with the largest cardinality is deleted from the historical full dataset to form the second remaining cluster set. The process involves iteratively selecting the cluster with the largest cardinality to add to the final cluster set and updating the remaining cluster set. This continues until the latest remaining cluster set no longer generates any clusters and / or cannot form new clusters after applying the DBSCAN algorithm. The resulting final cluster set is the data item cluster.
6. The intelligent data benchmarking method based on the collaboration of large and small models as described in claim 1, characterized in that, The method for obtaining the candidate data standard set through pre-screening is as follows: By performing a similarity search in the full retrieval library of basic data standards based on the textual semantic description of the source field to be benchmarked, the top 3 candidate data standards with the most similar names and semantics are selected to form the first candidate data standard set, which is defined as D1. Based on the data sample characteristics, data type characteristics, and code values of the target field, the technical attributes of the target field are determined. A data standard sub-retrieval library with the same technical attributes as the target field is queried. The top three candidate data standards with the closest semantics are selected from the retrieved data standard sub-retrieval library to form the second candidate data standard set, defined as D2. If the target field has a code value attribute, the data standard sub-retrieval library with the closest code value semantics is first selected from the code value retrieval library. Then, the top three candidate data standards with the closest semantics are selected from the selected data standard sub-retrieval library to form the second candidate data standard set. If the field to be benchmarked has a benchmarking result in the same table and field in the historical benchmarking records, then the corresponding historical data standard becomes the third candidate data standard set, defined as D3; The formula for calculating the candidate data standard set is: .
7. The intelligent data benchmarking method based on the collaboration of large and small models as described in claim 1, characterized in that, Before S3, the following steps are included: compare the core features of the field to be benchmarked with the core features of the candidate standards in the candidate data standard set. If any one of them does not meet the requirements, the candidate data standard is removed. The core features include: data type consistency, data length consistency, data precision consistency, and the proportion of intersection of enumerated values.
8. The intelligent data benchmarking method based on the collaboration of large and small models as described in claim 1, characterized in that, In S3, if a historical benchmark exists, the data standard corresponding to that historical benchmark will be returned as the unique benchmark result.
9. The intelligent data benchmarking method based on the collaboration of large and small models as described in claim 1, characterized in that, It also includes the following steps: S4: Human-machine collaborative confirmation, and learning and optimization based on human operation feedback.
10. An intelligent data benchmarking system based on the collaboration of large and small models, characterized in that, An intelligent data benchmarking system is constructed using the intelligent data benchmarking method described in any one of claims 1-9.