Method for matching data fields and data standards of database and readable storage medium
By constructing a vector database of data standards and optimizing matching results using blood relationships, combined with pre-training and detection of large models, the problem of inefficiency of traditional methods is solved, and efficient and reliable matching of data fields and data standards is achieved to adapt to large-scale data processing.
Patent Information
- Application Number
- CN202410150427.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2024-12-03
- Estimated Expiration
- 2044-02-02
AI Technical Summary
The matching methods of traditional data fields with data standards are inefficient and difficult to adapt to the growth of data volume, and lack automated and reliable solutions.
By constructing a vector database of pre-acquisitioned data standards, vectorize the data fields to be matched, and use blood ties and pre-trained detection models for matching optimization and detection, to achieve automated and efficient matching of data fields and data standards.
It improves the accuracy and efficiency of data processing, realizes the matching of automated, efficient and reliable data fields and data standards, and meets the needs of large-scale data processing.
Smart Images

Figure CN118012890B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of database management, and in particular to a matching method for data fields and data standards of a database and a readable storage medium. Background Art
[0002] When processing large-scale data, it is necessary to integrate data from different sources and in different formats. One of the important links is the matching of data fields and data standards. Traditional data matching methods mainly rely on manual processing or simple machine learning algorithms, which are inefficient and difficult to adapt to the growth of data volume. Therefore, there is an urgent need for an automated, efficient and reliable method for matching data fields and data standards. Summary of the invention
[0003] The object of the present invention is to provide a method for matching data fields and data standards of a database and a readable storage medium.
[0004] In a first aspect, an embodiment of the present invention provides a method for matching data fields and data standards for a database, comprising:
[0005] Acquire a data field to be matched and a pre-built matching vector database, wherein the matching vector database is constructed by vectorizing a plurality of pre-collected data standards, wherein the data standards are used to define data relationship table attributes of the data field;
[0006] Vectorizing the to-be-matched data field to obtain a data field vector;
[0007] Perform matching in the matching vector database based on the data field vector to obtain a plurality of initial matching results;
[0008] Based on the predetermined blood relationship corresponding to the to-be-matched data field, the multiple initial matching results are optimized to obtain an intermediate matching result;
[0009] The pre-trained detection model is called to detect the intermediate matching result. If the detection passes, the intermediate matching result is output as the target matching result, and the target matching result includes the data field to be matched and the target data standard that has a matching relationship with the data field to be matched.
[0010] In a possible implementation manner, the matching vector database is constructed in the following manner, including:
[0011] obtaining the plurality of data standards;
[0012] Performing attribute integration on the multiple data standards, and inputting the multiple data standards after attribute integration into a pre-trained text embedding model for vectorization operation to obtain multiple data standard vectors corresponding to the multiple data standards;
[0013] The matching vector database is constructed based on the multiple data standard vectors.
[0014] In a possible implementation, the method further includes:
[0015] Determining whether there are references in the plurality of data standards;
[0016] If it exists, extract the document content corresponding to the reference;
[0017] Cutting the document content to obtain a document cutting result;
[0018] Vectorizing the document cutting result and using it as the data standard vector;
[0019] If not, the step of integrating the attributes of the multiple data standards is performed.
[0020] In a possible implementation, the matching is performed in the matching vector database based on the data field vector to obtain a plurality of initial matching results, including:
[0021] Calculating the similarity between the data field vector and each of the data standard vectors to obtain a similarity score for each of the data standard vectors, wherein the data field vectors include a data field name vector and a data field business attribute vector;
[0022] The plurality of initial matching results are obtained by sorting the data standard vectors according to their similarity scores.
[0023] In a possible implementation, the blood relationship includes a data flow relationship between data fields, and the optimizing the multiple initial matching results based on the predetermined blood relationship corresponding to the to-be-matched data fields to obtain the intermediate matching results includes:
[0024] By utilizing the data flow relationship between the data fields, the blood relationship information is obtained by analyzing the primary and foreign key association information and the past SQL syntax;
[0025] The matching range of the multiple initial matching results is expanded by combining the blood relationship graph algorithm and the blood relationship information to obtain the intermediate matching result.
[0026] In a possible implementation, the combining of the blood relationship graph algorithm and the blood relationship information to expand the matching range of the multiple initial matching results to obtain the intermediate matching results includes:
[0027] If the business attribute of the first data field included in the first initial matching result is missing more than the preset missing threshold, the attribute of the upstream field of the target data field is taken as a supplementary attribute to obtain the intermediate matching result;
[0028] If the second initial matching result and the third initial matching result have data lineage and association lineage, the second initial matching result and the third initial matching result are merged to obtain the intermediate matching result, and the first initial matching result, the second initial matching result and the third initial matching result are any initial matching result among the multiple initial matching results.
[0029] In a possible implementation manner, calling a pre-trained large detection model to detect the intermediate matching result includes:
[0030] Based on the intermediate matching result, the detection large model is asked questions according to a preset question template to obtain a matching evaluation answer output by the detection large model;
[0031] A detection result of the intermediate matching result is determined based on the matching evaluation answer.
[0032] In a second aspect, an embodiment of the present invention provides a device for matching data fields and data standards of a database, comprising:
[0033] An acquisition module, used to acquire a data field to be matched and a pre-built matching vector database, wherein the matching vector database is constructed by vectorizing a plurality of data standards collected in advance, wherein the data standards are used to define data relationship table attributes of the data field;
[0034] A matching module, used for vectorizing the to-be-matched data field to obtain a data field vector;
[0035] Matching the data field vector in the matching vector database to obtain a plurality of initial matching results; optimizing the plurality of initial matching results based on the predetermined blood relationship corresponding to the to-be-matched data field to obtain an intermediate matching result;
[0036] The detection module is used to call a pre-trained detection model to detect the intermediate matching result. If the detection passes, the intermediate matching result is output as a target matching result. The target matching result includes the data field to be matched and the target data standard that has a matching relationship with the data field to be matched.
[0037] In a third aspect, an embodiment of the present invention provides a computer device, comprising a processor and a non-volatile memory storing computer instructions, wherein when the computer instructions are executed by the processor, the computer device executes the method for matching data fields and data standards for a database as described in at least one possible implementation of the first aspect.
[0038] In a fourth aspect, an embodiment of the present invention provides a readable storage medium, wherein the readable storage medium includes a computer program, and when the computer program is running, the computer device where the readable storage medium is located is controlled to execute the method for matching data fields and data standards for a database as described in at least one possible implementation of the first aspect.
[0039] Compared with the prior art, the beneficial effects provided by the present invention include: using a matching method and a readable storage medium for data fields and data standards for a database disclosed in the present invention, and operating by obtaining the data field to be matched and a pre-constructed matching vector database. Then, the data field to be matched is vectorized to obtain a data field vector, and matched in the matching vector database to obtain multiple initial matching results. These initial matching results are optimized using the blood relationship corresponding to the predetermined data field to be matched to obtain an intermediate matching result. Then, the pre-trained detection large model is called to detect the intermediate matching result, and when the detection passes, the intermediate matching result is output as the target matching result. Finally, the output target matching result includes the data field to be matched and the target data standard that has a matching relationship with the data field to be matched. Such a design realizes automatic, efficient and reliable matching of data fields and data standards, and improves the accuracy and efficiency of data processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative work.
[0041] Figure 1 A schematic diagram of a step flow chart of a method for matching data fields and data standards provided in an embodiment of the present invention;
[0042] Figure 2 A schematic block diagram of the structure of a device for matching data fields and data standards in a database provided by an embodiment of the present invention;
[0043] Figure 3 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0044] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.
[0045] The specific implementation modes of the present invention are described in detail below in conjunction with the accompanying drawings.
[0046] In order to solve the technical problems in the aforementioned background technology, Figure 1 This is a flow chart of a method for matching data fields and data standards for a database provided in an embodiment of the present disclosure. The method for matching data fields and data standards for a database is introduced in detail below.
[0047] Step S201, obtaining a data field to be matched and a pre-built matching vector database, wherein the matching vector database is constructed by vectorizing a plurality of pre-collected data standards, wherein the data standards are used to define data relationship table attributes of the data field;
[0048] Step S202, vectorizing the to-be-matched data field to obtain a data field vector;
[0049] Step S203, performing matching in the matching vector database based on the data field vector to obtain a plurality of initial matching results;
[0050] Step S204, optimizing the multiple initial matching results based on the predetermined blood relationship corresponding to the to-be-matched data field to obtain an intermediate matching result;
[0051] Step S205, calling the pre-trained detection model to detect the intermediate matching result, and if the detection passes, outputting the intermediate matching result as the target matching result, wherein the target matching result includes the data field to be matched and the target data standard that has a matching relationship with the data field to be matched.
[0052] In an embodiment of the present invention, for example, an e-commerce platform wants to automatically standardize and match its product descriptions. They have obtained a large amount of product description data from suppliers and have established a matching vector database, which contains standardized vectors of various product categories and attributes. There is a product description to be matched on the platform, for example: "This phone has a 6-inch touch screen, 64GB storage space, and a 12MP camera." By extracting the key information in the description and converting it into a numerical vector representation, such as [6, 64, 12], a data field vector is obtained. The data field vector [6, 64, 12] is used to match in the matching vector database to find the most similar standardized vector to the vector. Assume that three initial matching results are found, corresponding to three different product categories: "smartphone", "storage device" and "digital camera". By analyzing the blood relationship of the data fields to be matched, for example, the key information in the product description belongs to the mobile phone category, it can be concluded that the final matching result should be "smartphone", so by optimizing the initial matching results, it is limited to only one intermediate matching result, "smartphone". The pre-trained detection model is used to verify the intermediate matching results to ensure their rationality and accuracy. If it passes the test, "smartphone" is considered a valid matching result and is output as the target matching result. The final target matching result is to associate the data field to be matched (the original product description) with the target data standard in the matching database (the standardized vector of the "smartphone" category) to establish a matching relationship between them. In this way, the platform can match the product description with the correct standard classification and realize automatic data field and data standard matching.
[0053] In the embodiment of the present invention, the matching vector database is constructed in the following manner.
[0054] (1) obtaining the multiple data standards;
[0055] (2) integrating the attributes of the multiple data standards, and inputting the multiple data standards after the attribute integration into a pre-trained text embedding model for vectorization operation to obtain multiple data standard vectors corresponding to the multiple data standards;
[0056] (3) Constructing the matching vector database based on the multiple data standard vectors.
[0057] In the embodiment of the present invention, illustratively, multiple data standards are obtained from different medical databases and literatures, and these standards may include data on symptoms, biochemical indicators, genetic information, etc.
[0058] Data standards obtained from different sources are integrated to create a more comprehensive dataset. For example, they may integrate symptom descriptions, laboratory test results, and gene mutation information from different studies. Using pre-trained text embedding models, such as Word2Vec or BERT, the integrated data standards are converted into vector representations. These vectors capture the semantic and contextual information between data standards. After the integrated data standards are input into the text embedding model, the vector representation corresponding to each data standard is obtained. For example, a data standard describing disease symptoms may be converted into a vector with dozens of dimensional features. All data standard vectors are combined into a database, namely the matching vector database. This database can be used to store and quickly retrieve vector representations related to various diseases. For example, the database may contain vectors for cancer, heart disease, and neurological diseases. Through the above steps, a matching vector database is successfully constructed, which can be used to compare and match with new cases in order to identify similar diseases or provide treatment recommendations. Such a database can help doctors speed up the diagnosis process and improve the accuracy of disease identification in clinical practice.
[0059] In the embodiments of the present invention, the following implementation modes are also provided.
[0060] (1) determining whether there are references in the multiple data standards;
[0061] (2) If it exists, extract the document content corresponding to the reference;
[0062] (3) cutting the document content to obtain a document cutting result;
[0063] (4) vectorizing the document segmentation result and using it as the data standard vector;
[0064] (5) If not present, execute the step of integrating the attributes of the multiple data standards.
[0065] In an embodiment of the present invention, illustratively, it is assumed that there is an intelligent search engine that can return relevant references and data standards based on keywords provided by the user. The user enters a set of data standards, such as certain treatment plans in the field of medical research. The system will first determine whether these data standards contain relevant references. If references are found in the previous step of judgment, the system will extract the specific content of these references from each data standard for subsequent processing. The system will use natural language processing technology to cut the content of the document and divide it into appropriate paragraphs, sentences or words to better understand and process the document information. By converting the document cutting results into numerical vector representations, the system can more conveniently perform data analysis and comparison. In this way, quantitative similarity comparisons or other correlation calculations can be performed between the references and the data standards. If no references are found in the first step of judgment, the system will perform another processing method. It will integrate the attributes of multiple data standards, perhaps by summarizing, screening or other methods, to generate a comprehensive data standard. These scenarios illustrate the application of the method proposed in the above content in an intelligent search engine. It enables the system to identify and extract relevant references, and effectively process and analyze these document contents and data standards through technologies such as vectorization and attribute integration. This method can improve the accuracy and efficiency of information retrieval and provide users with a better search experience.
[0066] In the embodiment of the present invention, the aforementioned step S203 may be implemented in the following manner.
[0067] (1) calculating the similarity between the data field vector and each of the data standard vectors to obtain a similarity score for each of the data standard vectors, wherein the data field vector includes a data field name vector and a data field business attribute vector;
[0068] (2) Sorting according to the similarity score of each of the data standard vectors to obtain the multiple initial matching results.
[0069] In an embodiment of the present invention, illustratively, it is assumed that there is a matching vector database, which contains multiple data standard vectors. Each data standard vector is composed of a data field name vector and a data field business attribute vector. Now, the above method is used to match a new data field vector. The data field vector is also composed of a data field name vector and a data field business attribute vector. Data field vector: [name: customer name, business attribute: age]; data standard vector 1: [name: name, business attribute: age]; data standard vector 2: [name: customer, business attribute: occupation]; data standard vector 3: [name: name, business attribute: address]; first calculate the similarity between the data field vector and each data standard vector. For example, for the data field vector and the data standard vector 1, a similarity calculation method (such as cosine similarity) can be used to obtain the similarity score between them. Similarity score: data field vector and data standard vector 1: 0.85; data field vector and data standard vector 2: 0.42; data field vector and data standard vector 3: 0.61; Next, sort according to the similarity score of each data standard vector to obtain multiple initial matching results. According to the above similarity scores, the sorted results are as follows: Sorted initial matching results: data standard vector 1 (similarity score: 0.85); data standard vector 3 (similarity score: 0.61); data standard vector 2 (similarity score: 0.42); Through these steps, matching based on data field vectors in the matching vector database is achieved, and multiple initial matching results are obtained. These results are sorted according to the similarity score between the data field vector and the data standard vector for further subsequent processing or screening.
[0070] It is worth noting that the step of using the vector database for similarity matching can also be implemented in the following manner.
[0071] Data preprocessing: Perform preprocessing operations such as cleaning, tokenization, and standardization on the input data to convert it into a form suitable for vector representation. This may include steps such as removing stop words, stemming, and part-of-speech tagging.
[0072] Vector representation: Converting text data into vector representation is one of the key steps for similarity matching. Text vector representation can be generated using statistical methods (such as TF-IDF) or neural network-based methods (such as Word2Vec, GloVe).
[0073] Vector indexing: Store vector-represented data in a vector database and build an index to speed up similarity queries. Commonly used vector databases include Milvus, Faiss, etc.
[0074] Similarity matching algorithm: You can choose different natural language similarity algorithms, such as cosine similarity, Euclidean distance, Manhattan distance, etc., to calculate the similarity between vectors. These algorithms can help determine which data in the database is most similar to the input data.
[0075] Machine learning prediction model: In addition to using similarity algorithms for matching, machine learning prediction models can also be used to further improve the accuracy of matching. For example, text classification models, clustering models, or neural network models can be used to predict the category or label of input data and use it as a reference for matching.
[0076] Data standardization matching: Combine the output results of other natural language similarity algorithms and machine learning prediction models and use them as the basis for standardized matching. By comparing the output results of different algorithms and models, the results can be combined and a voting algorithm or weighted algorithm can be used to generate the final recommendation solution.
[0077] Final recommendation plan: Based on the output results of various algorithms and models, the data with the highest score or the highest similarity can be selected as the final recommendation plan. Specific evaluation indicators and decision methods can be customized according to actual needs.
[0078] In the embodiment of the present invention, the blood relationship includes a data flow relationship between data fields, and the aforementioned step S204 can be implemented through the following example.
[0079] (1) utilizing the data flow relationship between the data fields, and obtaining the blood relationship information by analyzing the primary and foreign key association information and the past SQL syntax;
[0080] (2) Combining the blood relationship graph algorithm and the blood relationship information to expand the matching range of the multiple initial matching results to obtain the intermediate matching results.
[0081] In the embodiment of the present invention, for example, it is assumed that there is a bank database system, which includes a customer table, an account table and a transaction record table. In this scenario, by analyzing the foreign key relationship in the database and the SQL statements executed in the past, it can be determined that there is a primary-foreign key association relationship between the customer table and the account table, and the customer ID field in the transaction record table is associated with the primary key field of the customer table. By analyzing these relationships, the data flow relationship between the customer table, the account table and the transaction record table can be established, thereby obtaining the blood relationship information. Assume that on a large e-commerce platform, it is necessary to recommend related products based on the user's purchase record and browsing history. In this scenario, the blood relationship graph algorithm can be used to construct the data flow relationship between the user's purchase record and browsing history, and determine the blood relationship between them. Based on the blood relationship corresponding to the predetermined data field to be matched, multiple initial matching results can be optimized. For example, by analyzing the blood relationship information of the product ID field in the user's purchase record and the product ID field in the browsing history, the matching range can be expanded to find more intermediate matching results related to the user's interests. These scenarios are just simple examples of each of the above steps, and the specific application scenarios may be more complex and diverse. The key to this method is to optimize the matching results by analyzing the data flow relationship and blood relationship information, and to expand the matching range by combining the blood relationship graph algorithm to obtain more accurate intermediate matching results.
[0082] In the embodiment of the present invention, the step of combining the bloodline graph algorithm and the bloodline information to expand the matching range of the multiple initial matching results to obtain the intermediate matching results can be implemented through the following example.
[0083] (1) If the business attribute of the first data field included in the first initial matching result is missing more than a preset missing threshold, taking the attribute of the upstream field of the target data field as a supplementary attribute to obtain the intermediate matching result;
[0084] (2) If the second initial matching result and the third initial matching result have data lineage and association lineage, the second initial matching result and the third initial matching result are merged to obtain the intermediate matching result, and the first initial matching result, the second initial matching result and the third initial matching result are any initial matching result among the multiple initial matching results.
[0085] In an embodiment of the present invention, illustratively, it is assumed that there is a data matching system for comparing data sets from two different sources. The system uses a lineage graph algorithm and lineage information for matching. For example, there are two data sources A and B, each of which contains multiple data fields. First, according to the aforementioned method, the system uses a lineage graph algorithm and lineage information to expand the matching range of multiple initial matching results to obtain an intermediate matching result. This means that the system will further analyze the relationship between the data fields to find more possible matches. In the intermediate matching results, if the business attribute vacancies of the first data field contained in the first initial matching result exceed the preset vacancy threshold, the system will adopt a specific strategy to fill these vacancies. One strategy is to extract attributes from the lineage upstream field of the target data field as supplementary attributes. In this way, the system can obtain missing business attribute information by associating the lineage relationship of the data fields. For example, assume that the first initial matching result is to match the "Name" field in data source A with the "Name" field in data source B. If there are missing business attributes in the "Name" field, the system can find the lineage upstream fields related to the "Name" field in data source B, such as "FirstName" and "LastName", and use their attributes as supplementary attributes of the "Name" field. In the intermediate matching results, if the second initial matching result and the third initial matching result have data lineage and associated lineage, the system will merge them into a more detailed matching result. This means that the system found the correlation between the two initial matching results when analyzing the lineage information and believes that they should be considered as the same match. For example, suppose the second initial matching result matches the "Address" field in data source A with the "Location" field in data source B, and the third initial matching result matches the "ID" field in data source A with the "Identifier" field in data source B. Then, the system finds that there is a correlation between the "Address" field and the "ID" field in data source B in the data lineage and associated lineage analysis. Therefore, the system will merge the second and third initial matching results into an intermediate matching result to provide more comprehensive matching information. It should be noted that the first initial matching result, the second initial matching result and the third initial matching result are only any one group of multiple initial matching results, and which specific results will be merged depends on the analysis results of the bloodline graph algorithm and the bloodline information.
[0086] In the embodiment of the present invention, the aforementioned step S205 can be implemented through the following example execution.
[0087] (1) Based on the intermediate matching result, ask the detection model a question according to a preset question template to obtain a matching evaluation answer output by the detection model;
[0088] (2) Determine a detection result of the intermediate matching result based on the matching evaluation answer.
[0089] In an embodiment of the present invention, illustratively, it is assumed that an intelligent customer service system is being developed, in which users can enter questions and hope to get relevant answers. In this case, the intermediate matching result may be a set of candidate questions obtained by matching the questions entered by the user with the questions stored in the database. The assumed detection large model is a deep learning model, for which a set of body question templates are defined, which are intended to extract key information from the intermediate matching results and evaluate their matching degree with the questions. For example, if the user's question is: "Where can I find a nearby restaurant?" Then the question template may be: "Does this question mention the geographical location?", "Does this question involve catering services?" and so on. By asking these questions to the detection large model, a matching evaluation answer can be obtained. The assumed matching evaluation answer is a score indicating the matching degree of the intermediate matching result with the question. A threshold can be set to determine whether the intermediate matching result is judged as a valid detection result. If the score of the matching evaluation answer is higher than the threshold, the intermediate matching result is determined to be a valid detection result. For example, if the score of the matching evaluation answer is 0.8 for a given question, and the threshold is set to 0.5, the intermediate matching result can be considered valid. In summary, through this method, the pre-trained detection model can be used to detect the intermediate matching results, and the matching evaluation answers can be obtained through the question template, and finally the detection results of the intermediate matching results can be determined based on these answers. This method can be applied to various scenarios, such as intelligent customer service systems, information retrieval, etc.
[0090] The following provides an overall implementation process of an embodiment of the present invention.
[0091] In the field of data management, the formulation and application of data standards are crucial to ensure the consistency, accuracy and interoperability of data. Data standards define the names, definitions, formats and relationships of data fields in data relationship tables so that different systems can understand and share data. The data standard automatic matching technology also provided in the embodiment of the present invention is an intermediate layer functional module used by users on the management side of the data management platform. Its upstream input includes several modules: technical information and business information collection and entry module for data fields; data standards and related literature entry and management module; data lineage collection and display module. It obtains the necessary information for data standard-data field matching from the running results of the upstream modules, and then outputs the matching results through a series of matching and review algorithms. Its downstream is connected to the data standard display module, the automatic inspection module for the implementation of data standards, etc.
[0092] 1. Data standardization and knowledge base construction
[0093] The first is the data preparation stage, which requires integrating the attributes of all data standards within the matching range, vectorizing them through the text embedding model, and storing them in the vector database. If the data standard includes references, the reference content is segmented and vectorized.
[0094] 2. Data field similarity query and matching relationship tagging
[0095] The data field name and the relevant business attributes of the field are also vectorized. The vector of the data field is used to search in the vector database, and the relevant results are matched by cosine similarity, and scored and sorted according to the similarity.
[0096] 3. Combine data and blood relationship to expand the matching range
[0097] Data lineage records the data flow relationship between data fields. It can be obtained by analyzing the primary and foreign key association information of the data table and the syntax of the database's past SQL execution, and recorded in the "point-edge-point" format of [Table 1. Field 1]-(lineage type)-[Table 2. Field 2]. Examples of obtaining lineage include: if the product_id of the sales order table and the id of the product table have a foreign key association, the two columns can be considered to have "associated lineage"; the deal_id of the sales order table and the event_id of the mini program tracking table are used as join...on... conditions in the SQL execution history, and can also be recorded as having "associated lineage"; in the SQL execution history, the event_id of the mini program tracking table is written to the evt_id column of the coupon record using insertinto...select..., which can be recorded as "data lineage", etc.
[0098] After the first and second steps of data standard matching, a batch of confirmed matches have been obtained. Next, this technical solution will combine the data lineage information already obtained in the data platform. Combining lineage and graph algorithms, the following extensions are made to improve the coverage of automatic matching:
[0099] a. If a data field has many missing business-related attributes, the attributes of the upstream field are used as supplementary attributes of this field.
[0100] b. For fields with "data lineage" in the lineage graph (for example, by inserting field a into field b through an insert statement, it can be determined that the two fields contain the same content), the matching results of the two data fields can be merged, and the one with the highest score in the two matches can be selected as the common automatic matching result of the two fields.
[0101] c. For fields with "associated bloodlines" in the bloodline graph (for example, when using the join...on syntax to query, two fields are used as the conditions of the on expression), the matching results of the two data fields can be merged, and the one with the highest score in the two matches can be selected as the common automatic matching result of the two fields.
[0102] d. After completing the above steps, if there are still fields that have not obtained matching results, you can obtain the information of the first-level or second-level blood relationship fields of the field and splice them into an additional attribute: "Related fields: field 1 name, table name, field notes; field 2 name, table name, field notes...", add this attribute to the similarity matching problem, and rebuild the search vector
[0103] 4. Use LLM for automatic review
[0104] The above rules may produce uncertain matching results, such as low similarity calculation scores, multiple matches with high but similar similarity scores, or users specify tables and fields to focus on. By prompting the project to build the input of the large model, the reasoning ability of the large model is used to make judgments.
[0105] a. Example of using the model to make a selection: Ask the model a question: "You are a data expert. Given a field in a database and some field-related data standards found in a preliminary search, please select the most suitable standard for this field from these data standards.\nField information: [Field name] [Field business attribute]\nCandidate data standard list: 1. [Data standard name] [Data standard attribute] 2. [Data standard name] [Data standard attribute]...\nYour answer should only include the data standard number you selected and the basis for your selection. Please answer in Chinese."
[0106] b. Example of using the model to confirm that the match is correct: Ask the model a question: "You are a data expert. Given a field in a database and a data standard determined by similarity matching, please determine whether the field is suitable for associating with this standard. If not, please refer to some other similar results and recommend a data standard with a better match.\nField information: [field name] [field business attribute]\nRecommended data standard: [data standard name] [data standard attribute]\nCandidate data standard list: 1. [data standard name] [data standard attribute] 2. [data standard name] [data standard attribute]...\nIn your answer, you should use "yes" or "no" to indicate your judgment result. If no, give the serial number of the data standard you selected and the basis for your selection. Please answer in Chinese."
[0107] After that, the output of the model is parsed and the model's recommendation results are recorded. For the matching results confirmed by the model, special marks can be made when they are displayed to prompt the user: "This result has passed the authenticity check" and show the user the reason for the recommendation. When the user views the results, the user is also allowed to correct the matching results and record the correction results for use in the human feedback reinforcement learning (RLHF) process to improve the model's ability level in this task.
[0108] In summary, the embodiment of the present invention first vectorizes the data standards and references, and uses vector similarity to search and match the data standards, thereby ensuring the sufficiency and reliability of the matching information, while also realizing the function of automatic matching without the need for prior knowledge such as historical matching records. Next, the matched data standards are verified and expanded through data lineage relationships, providing more dimensions of information for data standard matching. Then, a large language model is used to check the correctness of the matching results in order to provide users with authentic and reliable data standard matching results, which makes up for the shortcomings of pure algorithms, actually reduces manual participation, and improves the credibility of matching results. Finally, the above steps are integrated into an automatic matching workflow, and each step complements each other to form a comprehensive, effective and efficient data field and data standard matching method.
[0109] Please refer to Figure 2 , Figure 2 A device 110 for matching data fields and data standards of a database provided in an embodiment of the present invention includes:
[0110] The acquisition module 1101 is used to acquire a data field to be matched and a pre-built matching vector database, wherein the matching vector database is constructed by vectorizing a plurality of data standards collected in advance, wherein the data standards are used to define data relationship table attributes of the data field;
[0111] A matching module 1102 is used to vectorize the to-be-matched data field to obtain a data field vector;
[0112] Matching the data field vector in the matching vector database to obtain a plurality of initial matching results; optimizing the plurality of initial matching results based on the predetermined blood relationship corresponding to the to-be-matched data field to obtain an intermediate matching result;
[0113] The detection module 1103 is used to call the pre-trained detection model to detect the intermediate matching result. If the detection passes, the intermediate matching result is output as the target matching result. The target matching result includes the data field to be matched and the target data standard that has a matching relationship with the data field to be matched.
[0114] It should be noted that the implementation principle of the aforementioned matching device 110 for data fields and data standards of a database can refer to the implementation principle of the aforementioned matching method for data fields and data standards, and will not be repeated here. It should be understood that the division of the various modules of the above device is only a division of logical functions. In actual implementation, they can be fully or partially integrated into a physical entity, or physically separated. And these modules can all be implemented in the form of software called by processing elements; they can also be fully implemented in the form of hardware; some modules can also be implemented in the form of software called by processing elements, and some modules can be implemented in the form of hardware. For example, the matching device 110 for data fields and data standards of a database can be a separately established processing element, or it can be integrated in a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called and executed by a processing element of the above device. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each module above can be completed by an integrated logic circuit of hardware in a processor element or an instruction in software form.
[0115] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application specific integrated circuits (ASIC), or one or more microprocessors (digital signal processors, DSP), or one or more field programmable gate arrays (FPGA), etc. For another example, when a module above is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).
[0116] The embodiment of the present invention provides a computer device 100, which includes a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device 100 executes the aforementioned matching device 110 for matching data fields and data standards of a database. Figure 3 As shown, Figure 3The computer device 100 provided in the embodiment of the present invention is a structural block diagram. The computer device 100 includes a matching device 110 for matching data fields and data standards of a database, a memory 111, a processor 112 and a communication unit 113.
[0117] To achieve data transmission or interaction, the memory 111, the processor 112 and the communication unit 113 are electrically connected to each other directly or indirectly. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The matching device 110 for data fields and data standards of a database includes at least one software function module that can be stored in the memory 111 in the form of software or firmware or solidified in the operating system (OS) of the computer device 100. The processor 112 is used to execute the matching device 110 for data fields and data standards of a database stored in the memory 111, such as the software function modules and computer programs included in the matching device 110 for data fields and data standards of a database.
[0118] An embodiment of the present invention provides a readable storage medium, which includes a computer program. When the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the aforementioned matching method for data fields and data standards of a database.
[0119] For illustrative purposes, the foregoing description is made with reference to specific embodiments. However, the above illustrative discussion is not intended to be exhaustive or to limit the present disclosure to the precise form disclosed. Numerous modifications and variations are possible in accordance with the above teachings. These embodiments are selected and described in order to best illustrate the principles of the present disclosure and its practical application, so that those skilled in the art can best utilize the present disclosure and utilize various embodiments with different modifications to suit the intended specific application.
Claims
1. A method for matching data fields and data standards for a database, characterized in that: include: Acquire a data field to be matched and a pre-built matching vector database, wherein the matching vector database is constructed by vectorizing a plurality of pre-collected data standards, wherein the data standards are used to define data relationship table attributes of the data field; Vectorizing the to-be-matched data field to obtain a data field vector; Perform matching in the matching vector database based on the data field vector to obtain a plurality of initial matching results; Based on the predetermined blood relationship corresponding to the to-be-matched data field, the multiple initial matching results are optimized to obtain an intermediate matching result; Calling a pre-trained detection model to detect the intermediate matching result, and if the detection passes, outputting the intermediate matching result as a target matching result, wherein the target matching result includes the data field to be matched and a target data standard that has a matching relationship with the data field to be matched; The matching vector database is constructed in the following manner, including: obtaining the plurality of data standards; Performing attribute integration on the multiple data standards, and inputting the multiple data standards after attribute integration into a pre-trained text embedding model for vectorization operation to obtain multiple data standard vectors corresponding to the multiple data standards; Based on the multiple data standard vectors, construct the matching vector database; The method further comprises: Determining whether there are references in the plurality of data standards; If it exists, extract the document content corresponding to the reference; Cutting the document content to obtain a document cutting result; Vectorizing the document cutting result and using it as the data standard vector; If not, executing the step of integrating the attributes of the multiple data standards; The matching is performed in the matching vector database based on the data field vector to obtain a plurality of initial matching results, including: Calculating the similarity between the data field vector and each of the data standard vectors to obtain a similarity score for each of the data standard vectors, wherein the data field vectors include a data field name vector and a data field business attribute vector; Sorting according to the similarity score of each of the data standard vectors to obtain the multiple initial matching results; The blood relationship includes a data flow relationship between data fields. The optimizing of the multiple initial matching results based on the predetermined blood relationship corresponding to the to-be-matched data fields to obtain the intermediate matching results includes: Utilizing the data flow relationship between the data fields, obtaining the blood relationship information by analyzing the primary and foreign key association information and the past SQL syntax; Expanding the matching range of the multiple initial matching results by combining the blood relationship graph algorithm and the blood relationship information to obtain the intermediate matching result; The combining of the blood relationship graph algorithm and the blood relationship information to expand the matching range of the multiple initial matching results to obtain the intermediate matching results includes: If the business attribute vacancies of the first data field included in the first initial matching result exceed the preset vacancies threshold, the attribute of the upstream field of the target data field is taken as a supplementary attribute to obtain the intermediate matching result, and the first data field is a field in the data field to be matched; If the second initial matching result and the third initial matching result have data lineage and association lineage, then the second initial matching result and the third initial matching result are merged to obtain the intermediate matching result, and the first initial matching result, the second initial matching result and the third initial matching result are any initial matching result among the multiple initial matching results; The calling of the pre-trained detection large model to detect the intermediate matching result includes: Based on the intermediate matching result, the detection large model is asked questions according to a preset question template to obtain a matching evaluation answer output by the detection large model; A detection result of the intermediate matching result is determined based on the matching evaluation answer.
2. A matching device for data fields and data standards, characterized in that: include: An acquisition module, used to acquire a data field to be matched and a pre-built matching vector database, wherein the matching vector database is constructed by vectorizing a plurality of data standards collected in advance, wherein the data standards are used to define data relationship table attributes of the data field; A matching module, used for vectorizing the to-be-matched data field to obtain a data field vector; Perform matching in the matching vector database based on the data field vector to obtain a plurality of initial matching results; Based on the predetermined blood relationship corresponding to the to-be-matched data field, the multiple initial matching results are optimized to obtain an intermediate matching result; A detection module, used for calling a pre-trained detection large model to detect the intermediate matching result, and outputting the intermediate matching result as a target matching result when the detection passes, wherein the target matching result includes the data field to be matched and a target data standard having a matching relationship with the data field to be matched; The matching vector database is constructed in the following manner, including: Acquire the multiple data standards; integrate the attributes of the multiple data standards, and input the multiple data standards after the attribute integration into a pre-trained text embedding model for vectorization operation to obtain multiple data standard vectors corresponding to the multiple data standards; construct the matching vector database based on the multiple data standard vectors; The acquisition module is also used for: Determine whether there is a reference document in the multiple data standards; if so, extract the document content corresponding to the reference document; segment the document content to obtain a document segmentation result; Vectorize the document cutting result and use it as the data standard vector; if it does not exist, perform the step of integrating the attributes of the multiple data standards; The matching is performed in the matching vector database based on the data field vector to obtain a plurality of initial matching results, including: Calculating the similarity between the data field vector and each of the data standard vectors to obtain a similarity score for each of the data standard vectors, wherein the data field vectors include a data field name vector and a data field business attribute vector; sorting the data standard vectors according to the similarity score to obtain the multiple initial matching results; The blood relationship includes a data flow relationship between data fields. The optimizing of the multiple initial matching results based on the predetermined blood relationship corresponding to the to-be-matched data fields to obtain the intermediate matching results includes: Utilizing the data flow relationship between the data fields, obtaining blood relationship information by analyzing primary and foreign key association information and past SQL syntax; expanding the matching range of the multiple initial matching results by combining the blood relationship graph algorithm and the blood relationship information to obtain the intermediate matching result; The combining of the blood relationship graph algorithm and the blood relationship information to expand the matching range of the multiple initial matching results to obtain the intermediate matching results includes: If the business attribute of the first data field included in the first initial matching result is missing more than the preset missing threshold, the attribute of the upstream field of the target data field is taken as the supplementary attribute to obtain the intermediate matching result; if the second initial matching result and the third initial matching result have data lineage and associated lineage, the second initial matching result and the third initial matching result are merged to obtain the intermediate matching result, the first initial matching result, the second initial matching result and the third initial matching result are any initial matching results among the multiple initial matching results, and the first data field is a field in the data field to be matched; The detection module is specifically used for: Based on the intermediate matching result, questions are asked to the detection large model according to a preset question template to obtain a matching evaluation answer output by the detection large model; and the detection result of the intermediate matching result is determined based on the matching evaluation answer.
3. A computer device, characterized in that: The computer device comprises a processor and a non-volatile memory storing computer instructions. When the computer instructions are executed by the processor, the computer device performs the method of claim 1.
4. A readable storage medium, characterized in that: The readable storage medium includes a computer program, and when the computer program is executed, the computer device where the readable storage medium is located is controlled to execute the method according to claim 1.
