Data table evaluation method and device, computer equipment and storage medium

Abstract

The embodiment of the invention provides a data table evaluation method and device, computer equipment and a storage medium, and relates to the technical field of data management. The method comprises the following steps: acquiring a first data table, a vector database and at least one standard field information of an enterprise; effective metadata in the first data table is subjected to HTML (HyperText Markup Language) tagging processing, a reference text field is obtained, and the reference text field comprises a table name, a field name and classification; converting the reference text field into a first vector; and based on the first vector, the vector database and the at least one standard field information, determining an evaluation result of the first data table, the evaluation result being used for indicating whether the first data table meets the evaluation requirements of the enterprise. By adopting the technical scheme provided by the embodiment of the invention, the evaluation accuracy of the data table can be improved.

Description

Technical Field

[0001] This application relates to the field of data management technology, and in particular to a method, apparatus, computer device, and storage medium for evaluating data tables. Background Technology

[0002] Large enterprises have a wide variety of data types and quantities.

[0003] In related technologies, rule-based matching and manual evaluation and inspection of data tables are relied upon. However, due to the diversity and complexity of the data, the accuracy of simple rule-based matching and manual inspection in these technologies is relatively low. Summary of the Invention

[0004] This application provides a method, apparatus, computer device, and storage medium for evaluating data tables, which can improve the evaluation accuracy of data tables. The technical solution provided by this application is as follows:

[0005] According to one aspect of the embodiments of this application, a method for evaluating a data table is provided, the method comprising:

[0006] Obtain the enterprise's first data table, vector database, and at least one standard field information;

[0007] The valid metadata in the first data table is tagged with HTML (HyperText Markup Language) to obtain reference text fields, which include table name, field name and category;

[0008] Convert the reference text field into a first vector;

[0009] Based on the first vector, the vector database, and the field information of the at least one standard, the evaluation result of the first data table is determined, and the evaluation result is used to indicate whether the first data table meets the evaluation requirements of the enterprise.

[0010] In some embodiments, determining the evaluation result of the first data table based on the first vector, the vector database, and the field information of the at least one standard includes:

[0011] The first vector is matched with at least one vector in the vector database to obtain a classification matching result for the first vector. The classification matching result is used to indicate at least one first category label corresponding to the first vector.

[0012] Obtain at least one candidate logical model and the table name, field information and category label of the at least one candidate logical model;

[0013] Based on the table name, field information, and at least one first category label corresponding to the first vector of the first data table, as well as the table name, field information, and category label of the at least one candidate logical model, similarity matching is performed on the first data table and the at least one candidate logical model respectively to obtain the logical model matching result corresponding to the first vector. The logical model matching result is used to indicate at least one first logical model corresponding to the first vector.

[0014] Based on the field information in the first data table and the field information in the at least one first logical model, the first data table and the at least one first logical model are respectively matched by field;

[0015] If a logical model matching the first data table exists in at least one first logical model, the first data table is determined to meet the enterprise's evaluation requirements.

[0016] In some embodiments, the step of performing similarity matching between the first vector and at least one vector in the vector database to obtain a classification matching result for the first vector includes:

[0017] The similarity between the first vector and the at least one vector is calculated respectively to obtain the first similarity value corresponding to the at least one vector;

[0018] Based on the first similarity value corresponding to each of the at least one vector, a plurality of second vectors are determined from the at least one vector;

[0019] Based on the first similarity values ​​corresponding to the plurality of second vectors respectively, the second vectors are aggregated by mean to obtain the at least one first classification label.

[0020] In some embodiments, the step of performing similarity matching on the first data table and the at least one candidate logical model based on the table name, field information, at least one first category label corresponding to the first vector, and the table name, field information, and category label of the at least one candidate logical model to obtain the logical model matching result corresponding to the first vector includes:

[0021] The similarity between the table name of the first data table and the table name of the at least one candidate logical model is calculated to obtain the second similarity value corresponding to the at least one candidate logical model.

[0022] The similarity between the field information of the first data table and the field information of the at least one candidate logical model is calculated to obtain the third similarity value corresponding to the at least one candidate logical model respectively.

[0023] The similarity between at least one first category label of the first data table and the category label of the at least one candidate logical model is calculated to obtain the fourth similarity value corresponding to the at least one candidate logical model.

[0024] The second similarity value, the third similarity value, and the fourth similarity value of each candidate logical model are summed to obtain the total similarity value of each candidate logical model.

[0025] Based on the total similarity value of each of the candidate logical models, the at least one first logical model is determined from the at least one candidate logical model.

[0026] In some embodiments, the field information includes field name, field type, and foreign key result, wherein the foreign key result includes two results: foreign key and not foreign key; the vector database corresponds to an index library, and the index library stores standard field information;

[0027] The step of matching fields in the first data table and the at least one first logical model based on field information in the first data table includes:

[0028] The first score of the first data table is obtained by weighted summing of the field names, field types, and foreign key results.

[0029] For each first logical model, the field name, field type, and foreign key result of each first logical model are weighted and summed to obtain the first score of each first logical model;

[0030] Based on the first score of the first data table and the first score corresponding to each of the first logical models, the first data table is matched with the at least one first logical model for the fields respectively.

[0031] In some embodiments, converting the reference text field into a first vector includes:

[0032] A pre-trained semantic vectorization model is used to convert the reference text fields into the first vector;

[0033] The first vector is a high-dimensional vector.

[0034] According to one aspect of the embodiments of this application, an evaluation apparatus for a data table is provided, the apparatus comprising:

[0035] The acquisition module is used to acquire the enterprise's first data table, vector database, and at least one standard field information;

[0036] The tagging module is used to perform HTML tagging processing on the valid metadata in the first data table to obtain reference text fields, which include table name, field name and category.

[0037] A vector conversion module is used to convert the reference text field into a first vector;

[0038] The result determination module is used to determine the evaluation result of the first data table based on the first vector, the vector database, and the field information of the at least one standard. The evaluation result is used to indicate whether the first data table meets the evaluation requirements of the enterprise.

[0039] In some embodiments, the result determination module includes:

[0040] The classification matching submodule is used to perform similarity matching between the first vector and at least one vector in the vector database to obtain the classification matching result of the first vector. The classification matching result is used to indicate at least one first category label corresponding to the first vector.

[0041] The acquisition submodule is used to acquire at least one candidate logical model and the table name, field information and category label of the at least one candidate logical model;

[0042] The model matching submodule is further configured to perform similarity matching on the first data table and the at least one candidate logical model based on the table name, field information, at least one first category label corresponding to the first vector, and the table name, field information, and category label of the at least one candidate logical model, to obtain the logical model matching result corresponding to the first vector. The logical model matching result is used to indicate at least one first logical model corresponding to the first vector.

[0043] The field matching submodule is used to perform field matching between the first data table and the at least one first logical model based on the field information in the first data table and the field information in the at least one first logical model respectively.

[0044] An evaluation submodule is used to determine whether the first data table meets the enterprise's evaluation requirements if a logical model matching the first data table exists in at least one first logical model.

[0045] In some embodiments, the classification matching submodule is configured to:

[0046] The similarity between the first vector and the at least one vector is calculated respectively to obtain the first similarity value corresponding to the at least one vector;

[0047] Based on the first similarity value corresponding to each of the at least one vector, a plurality of second vectors are determined from the at least one vector;

[0048] Based on the first similarity values ​​corresponding to the plurality of second vectors respectively, the second vectors are aggregated by mean to obtain the at least one first classification label.

[0049] In some embodiments, the model matching submodule is configured to:

[0050] The similarity between the table name of the first data table and the table name of the at least one candidate logical model is calculated to obtain the second similarity value corresponding to the at least one candidate logical model.

[0051] The similarity between the field information of the first data table and the field information of the at least one candidate logical model is calculated to obtain the third similarity value corresponding to the at least one candidate logical model respectively.

[0052] The similarity between at least one first category label of the first data table and the category label of the at least one candidate logical model is calculated to obtain the fourth similarity value corresponding to the at least one candidate logical model.

[0053] The second similarity value, the third similarity value, and the fourth similarity value of each candidate logical model are summed to obtain the total similarity value of each candidate logical model.

[0054] Based on the total similarity value of each of the candidate logical models, the at least one first logical model is determined from the at least one candidate logical model.

[0055] In some embodiments, the field information includes field name, field type, and foreign key result, wherein the foreign key result includes two results: foreign key and not foreign key; the vector database corresponds to an index library, and the index library stores standard field information;

[0056] The field matching submodule is used for:

[0057] The first score of the first data table is obtained by weighted summing of the field names, field types, and foreign key results.

[0058] For each first logical model, the field name, field type, and foreign key result of each first logical model are weighted and summed to obtain the first score of each first logical model;

[0059] Based on the first score of the first data table and the first score corresponding to each of the first logical models, the first data table is matched with the at least one first logical model for the fields respectively.

[0060] In some embodiments, the vector transformation module is configured to:

[0061] A pre-trained semantic vectorization model is used to convert the reference text fields into the first vector;

[0062] The first vector is a high-dimensional vector.

[0063] According to one aspect of the embodiments of this application, a computer device is provided, the computer device including a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the above-described data table evaluation method.

[0064] According to one aspect of the embodiments of this application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the above-described method for evaluating the data table.

[0065] According to one aspect of the embodiments of this application, a computer program product is provided, which is loaded and executed by a processor to implement the above-described method for evaluating the data table.

[0066] The technical solutions provided in this application embodiment may have the following beneficial effects:

[0067] By preprocessing the data table using an HTML tag-based text processing method, reference text fields and a first vector are obtained, avoiding excessively low scores during semantic matching and improving the accuracy of semantic matching. This effectively improves the accuracy of semantic vectorized retrieval, thereby enhancing the evaluation accuracy of the data table.

[0068] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0070] Figure 1 This is a flowchart of a data table evaluation method provided in one embodiment of this application;

[0071] Figure 2 This is a flowchart of an evaluation method for a data table provided in another embodiment of this application;

[0072] Figure 3 This is a flowchart of an evaluation method for a data table provided in another embodiment of this application;

[0073] Figure 4 This is a schematic diagram of the result of a field matching task provided in one embodiment of this application;

[0074] Figure 5 This is a block diagram of an evaluation apparatus for a data sheet provided in one embodiment of this application;

[0075] Figure 6 This is a block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0076] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods consistent with some aspects of this application as detailed in the appended claims.

[0077] The method provided in this application can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. This computer device can be a terminal such as a PC (Personal Computer), tablet computer, smartphone, wearable device, or intelligent robot; or it can be a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0078] The technical solution of this application will be described and illustrated below through several embodiments.

[0079] Please refer to Figure 1, which shows a flowchart of an evaluation method for a data table provided in an embodiment of the present application. In this embodiment, the method is mainly exemplified by being applied to the computer device introduced above. The method may include at least one of the following steps (110 to 140).

[0080] Step 110, obtain the first data table of the enterprise, the vector database, and at least one standard field information.

[0081] In some embodiments, in large enterprises, there are often problems such as various information systems standing side by side and it being difficult to统一 the data standards of each system. To achieve the integration and penetration of data, enterprises usually build analytical big data platforms such as data lakes, data warehouses, and data mid-platforms, extract data from each information system, complete data integration and standardization in the big data platform, and support the development and application of data. However, as the data source, the data standards generated and stored within each information system have not been unified with the construction of the big data platform. The big data platform always faces a large amount of data cleaning and processing work, and the costs of data construction and application remain high, and the efficiency needs to be improved.

[0082] In some embodiments, the first data table may be obtained after the enterprise preprocesses the original data, such as cleaning and filtering.

[0083] In some embodiments, the standard field information may be the field information of a logic model预先 set by relevant technical personnel. The logic model may also be referred to as a logic data table, a standard data table, etc.

[0084] Step 120, perform HTML tagging on the valid metadata in the first data table to obtain a reference text field, where the reference text field includes a table name, a field name, and a classification.

[0085] In some embodiments, through HTML tagging, a structured text in the form of "Organization Information Table<logic model name>Organization Structure Information< / logic model name><field name>Employee Number< / field name><field name>Gender< / field name>" can be obtained, that is, the reference text segment.

[0086] It should be noted that in the translation of the Chinese text in item , the word "统一" is directly translated as "统一" here because it seems to be a specific term in the context and may not have a more appropriate English equivalent at present. If there is a more accurate English expression for this specific concept in the relevant field, it can be adjusted accordingly.In some embodiments, HTML tagging is used to smooth the similarity calculation scores for vectorized retrieval, making the similarity matching process smoother and the score distribution more uniform. Since both the input text and the text in the index library are added with templated HTML tags, it is equivalent to adding a semantic constant term to both the numerator and denominator parts during the semantic similarity calculation process, similar to the smoothing calculation in mathematics. For example, for the input item "Organization Information Table<field name>Employee ID< / field name><field name>Position< / field name>" and the retrieval item "Institutional Information Table<logical model name>Organization Structure Information< / logical model name><field name>Employee Number< / field name><field name>Gender< / field name>", the common tag text between the two will generate a basic score when calculating semantic similarity, thus preventing text with low semantic relevance from having too low scores and text with high semantic relevance from having too high scores, ensuring the stability of the semantic similarity matching process and reducing the error problem in similarity calculation.

[0087] In some embodiments, structured HTML tags can also function as text segmentation and context supplementation. Rule-based classification tags semantically cut disordered text segments and encapsulate each text entity, avoiding context semantic confusion caused by text adhesion before and after during the semantic similarity matching process, and improving the accuracy of semantic similarity matching.

[0088] Step 130, convert the reference text field into a first vector.

[0089] In some embodiments, a pre-trained semantic vectorization model is used to convert the reference text field into a first vector; wherein, the first vector is a high-dimensional vector. In some embodiments, a semantic vectorization model is pre-trained, and then the trained semantic vectorization model is used to convert the reference text field into a first vector.

[0090] Step 140, determine the evaluation result of the first data table based on the first vector, the vector database, and at least one standard field information, and the evaluation result is used to indicate whether the first data table meets the enterprise's evaluation requirements.

[0091] In some embodiments, the standard field information may refer to the field information of the logical model. If there is a logical model matching the first data table, it indicates that the first data table meets the enterprise's evaluation requirements; if there is no logical model matching the first data table, it indicates that the first data table does not meet the enterprise's evaluation requirements.

[0092] In summary, the technical solution provided in this application preprocesses the data table using an HTML tag-based text processing method, thereby obtaining reference text fields and a first vector. This avoids excessively low scores during semantic matching, improves the accuracy of semantic matching, and effectively enhances the accuracy of semantic vectorized retrieval, thereby improving the evaluation accuracy of the data table.

[0093] In some possible implementations, such as Figure 2 As shown, based on the first vector, the vector database, and at least one standard field information, the evaluation result of the first data table (i.e., the above) is determined. Figure 1 Step 140 in the embodiment may further include the following steps (141-145):

[0094] Step 141: Perform similarity matching between the first vector and at least one vector in the vector database to obtain the classification matching result of the first vector. The classification matching result is used to indicate at least one first category label corresponding to the first vector.

[0095] In some embodiments, the first vector is searched for similarity matching in a vector database to obtain at least one vector with a high similarity to the first vector. The classification labels of these vectors are the first classification labels.

[0096] In some embodiments, the data architecture management process mainly includes three tasks: category matching, logical model matching, and data item matching. Category matching requires obtaining relevant category labels based on the text information of the input table; data model matching requires obtaining a standard data model related to the input table based on the text information of the input table; and data item matching requires matching a standard field name for each field name in the input table according to the logical model paradigm.

[0097] In some embodiments, step 141 may further include the following steps:

[0098] 1. Calculate the similarity between the first vector and at least one vector respectively, and obtain the first similarity value corresponding to at least one vector respectively;

[0099] 2. Based on the first similarity value corresponding to at least one vector, determine multiple second vectors from at least one vector;

[0100] 3. Based on the first similarity values ​​corresponding to multiple second vectors, perform mean aggregation on the second vectors to obtain at least one first classification label.

[0101] In some embodiments, a first vector is used to perform similarity matching retrieval in a vector database to obtain several coarse-ranked results and multiple second vectors. Then, the average values ​​of the multiple second vectors corresponding to the first vectors are aggregated to obtain at least one first classification label with the highest similarity.

[0102] Step 142: Obtain at least one candidate logical model and the table name, field information and category label of at least one candidate logical model.

[0103] Step 143: Based on the table name, field information, at least one first category label corresponding to the first vector of the first data table, and the table name, field information, and category label of at least one candidate logical model, perform similarity matching on the first data table and at least one candidate logical model respectively to obtain the logical model matching result corresponding to the first vector. The logical model matching result is used to indicate at least one first logical model corresponding to the first vector.

[0104] In some embodiments, step 143 may further include the following steps:

[0105] 1. Calculate the similarity between the table name of the first data table and the table name of at least one candidate logical model to obtain the second similarity value corresponding to each of the at least one candidate logical models.

[0106] 2. Calculate the similarity between the field information of the first data table and the field information of at least one candidate logical model to obtain the third similarity value corresponding to each of the at least one candidate logical model;

[0107] 3. Calculate the similarity between at least one first category label of the first data table and the category label of at least one candidate logical model to obtain the fourth similarity value corresponding to each of the at least one candidate logical models.

[0108] 4. Sum the second, third, and fourth similarity values ​​of each candidate logical model to obtain the total similarity value of each candidate logical model;

[0109] 5. Based on the total similarity value of each candidate logical model, determine at least one first logical model from at least one candidate logical model.

[0110] In some embodiments, the classification matching results obtained in step 141 are used to perform coarse ranking of the vectors in the vector database. Then, the total similarity value between the first data table and each candidate logical model is calculated. The total similarity value is the sum of the second, third, and fourth similarity values. The second similarity value refers to the similarity between the table names, the third similarity value refers to the similarity between the field information of the data tables, and the fourth similarity value refers to the similarity between the classification labels.

[0111] The n logical models with the highest total similarity scores are taken as the logical model matching results obtained in step 143. n can be any positive integer such as 1, 2, 3, 4, 5, etc.

[0112] Step 144: Based on the field information in the first data table and the field information in at least one first logical model, perform field matching between the first data table and at least one first logical model respectively.

[0113] In some embodiments, a field can also be referred to as a data item, and field matching can also be referred to as data item matching. In some embodiments, the logical model matching result obtained in step 143 is used as the coarse ranking result of field matching. Further fine ranking is performed based on the coarse ranking result.

[0114] In some embodiments, field information includes field name, field type, and foreign key result, where the foreign key result includes two types: foreign key and not foreign key; the vector database has a corresponding index library, which stores standard field information.

[0115] In some embodiments, step 144 may further include the following steps:

[0116] 1. Take a weighted sum of the field names, field types, and foreign key results of the first data table to obtain the first score of the first data table;

[0117] 2. For each first logical model, the field names, field types, and foreign key results of each first logical model are weighted and summed to obtain the first score of each first logical model;

[0118] 3. Based on the first score of the first data table and the first score corresponding to each first logical model, perform field matching between the first data table and at least one first logical model.

[0119] In some embodiments, fine-grained ranking of logical model matching may include the following steps:

[0120] (1) Calculate the score for each data item in the first data table and the standard data item in the logical model using a weighted scoring method. The weight is composed of the field name, data type, and whether it is a primary key or foreign key (i.e., foreign key result). The specific weight allocation satisfies the following rule: x + y + z = 1. Where x represents the weight of the field name, y represents the weight of the data type, and z represents the weight of whether it is a primary key or foreign key.

[0121] (2) Perform threshold judgment. If the score of the data item in the newly created table is greater than the preset threshold, proceed to the next calculation.

[0122] (3) Perform deduplication operation. For data items in the first data table whose calculated score is greater than or equal to the threshold, if the data item in the first data table is mapped one-to-one with the corresponding data item in the best standard table, then the data item pair is successfully matched; if there are multiple standard table data items corresponding to a data item in the first data table, then the standard table data item is matched with the data item in the first data table with the highest score.

[0123] (4) Perform iterative matching. For the data items in the first data table that failed to match in step (3), perform the operations of steps (1) to (3) with all the unmatched standard table data items until there are no more data items in the first data table or standard table data items that need to be matched.

[0124] Step 145: If at least one first logical model exists that matches the first data table, determine that the first data table meets the enterprise's evaluation requirements.

[0125] In the above implementation, by calculating the highest-scoring standard table field across the entire set for each field in the first data table and adding a method to handle many-to-one pairings, it is ensured that each standard table field is paired only with the most similar newly created table field, thus improving the pairing accuracy. Adding threshold filtering further improves the matching rate of the first data table fields.

[0126] In the above implementation, the accuracy of the top 5 category labels in the classification matching task reaches 0.89, the accuracy of the top 5 logical models matched in the logical model matching task reaches 0.91, and the accuracy of the field matching task reaches 0.81. Moreover, the recognition efficiency is high (e.g., it takes 300 seconds to process 116 test access tables), which reduces the workload of the evaluators and improves the efficiency of data evaluation.

[0127] In some embodiments, rough ranking refers to a preliminary screening process that generates a smaller candidate set by quickly evaluating a large number of candidates. This process uses simple and efficient algorithms to analyze user preferences and behavior data for more accurate recommendations in subsequent refined ranking steps. Refined ranking refers to the further ranking process in a recommendation system after preliminary screening, aiming to accurately rank the candidate set. This process uses complex algorithms and multi-dimensional features, including user preferences, behavior data, and context information, to generate optimal recommendation results, thereby enhancing the user experience and the accuracy of the recommendation system. In the above implementation, multiple potential results are quickly pre-screened and recalled through the rough ranking process, and the results are refined using refined ranking. At the same time, the rough ranking largely excludes irrelevant items, thus accelerating the refined ranking process. This processing method can effectively improve the processing speed.

[0128] In some embodiments, taking the example that the full-scale test data comes from the business system of an enterprise, the full-scale test data is about 98,000 pieces, mainly including the metadata of the business data table. As Figure 3 shown, the evaluation of the data table can include the following steps:

[0129] Data processing stage:

[0130] Step 310, process the data table information.

[0131] Clean the original data (i.e., the above full-scale test data) and retain the valid metadata, and perform HTML tagging on the text information of the cleaned data to obtain a new text field in the following form:

[0132] " Organization Information Table<Field Name> Employee ID< / Field Name><Field Name> Position< / Field Name>" and retrieval item " Institution Information Table<Logical Model Name> Organizational Structure Information< / Logical Model Name><Field Name> Employee Number< / Field Name><Field Name> Gender< / Field Name>"

[0133] Step 320, semantic vectorization conversion.

[0134] In this embodiment, BGE-base-zh-v1.5 is selected as the semantic vectorization model. The pre-trained semantic vectorization model is used to convert the new field into a high-dimensional vector, which is used as the new field of the data.

[0135] The processed data is stored in a vector database. In this embodiment, ElasticSearch 7.9.0 is used as the underlying vector database, and the full-scale data is classified and stored. The classification matching task and the logical model matching task share one index library, and the data item matching task has one index library.

[0136] Classification matching task:

[0137] Step 330: Classification matching and coarse ranking.

[0138] Using the vector transformed in step one, a cosine similarity-based matching search was performed in the corresponding index of Elasticsearch to obtain 20 coarsely ranked results.

[0139] Step 340: Categorization and matching for fine-grained ranking.

[0140] The similarity between the category tags in the metadata of the coarse ranking results and the access table name is calculated to obtain the 5 category tags with the highest similarity.

[0141] Logical model matching task:

[0142] Step 350: Logical model matching and coarse ranking.

[0143] First, use the transformed high-dimensional vector to perform a classification matching task once, and get 20 coarse ranking results and the metadata corresponding to each result.

[0144] Step 360: Rule-based fine sorting.

[0145] Based on the results of the previous step, the similarity score between the access table and the logical model in each coarse ranking result is calculated, including the table name score and field information score. The Top-5 logical model results are obtained based on the similarity scores.

[0146] Field matching task:

[0147] Step 370, coarse matching of fields.

[0148] Perform a logical model matching task to obtain the top-20 results.

[0149] Step 380: Iterative matching and ranking based on rules.

[0150] 1) Calculate the weighted score. This embodiment tested three different weight allocation schemes: (0.7, 0.2, 0.1), (0.5, 0.3, 0.2), and (0.8, 0.1, 0.1). The three weights are field weight, type weight, and whether it is a primary or foreign key. The field weight is mainly composed of the similarity between the field in the access table and the field in the standard table of the index. From the perspective of actual business, it should be assigned a relatively large weight. The type weight mainly judges whether the type of the field in the access table is consistent with the type of the field in the standard table. It is auxiliary information. It is worth noting that the type of the field in the access table may be incorrect in actual business scenarios, so the weight should not be too high. The primary or foreign key weight is used to check whether the field in the access table has the attribute of a primary or foreign key. Similar to the field type, it is also auxiliary information and is also prone to access table bias. The weight should not be too high. Test results show that setting the field weight to 0.7, the type to 0.2, and the primary / foreign key status to 0.1 yields the best test accuracy. Field weights higher or lower than 0.7 lead to a decrease in accuracy, and the weights for type and primary / foreign key status should not be too high. In some embodiments, the field matching results can be as follows: Figure 4 As shown.

[0151] 2) Perform threshold judgment. In this embodiment, the threshold for data item score is set to 0.8. If the score of the newly created table field is greater than this threshold, proceed to the next calculation.

[0152] 3) Perform deduplication;

[0153] 4) Perform iterative matching.

[0154] Please refer to Figure 5 This diagram illustrates a block diagram of an evaluation apparatus for a data table according to an embodiment of this application. The apparatus has the functionality to implement the aforementioned data table evaluation method example; this functionality can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be mounted on a computer device. The apparatus 500 may include:

[0155] The acquisition module 510 is used to acquire the enterprise's first data table, vector database, and at least one standard field information;

[0156] The tagging module 520 is used to perform HTML tagging processing on the valid metadata in the first data table to obtain reference text fields, the reference text fields including table name, field name and category;

[0157] Vector conversion module 530 is used to convert the reference text field into a first vector;

[0158] The result determination module 540 is used to determine the evaluation result of the first data table based on the first vector, the vector database, and the field information of the at least one standard. The evaluation result is used to indicate whether the first data table meets the evaluation requirements of the enterprise.

[0159] In some embodiments, the result determining module 540 includes:

[0160] The classification matching submodule is used to perform similarity matching between the first vector and at least one vector in the vector database to obtain the classification matching result of the first vector. The classification matching result is used to indicate at least one first category label corresponding to the first vector.

[0161] The acquisition submodule is used to acquire at least one candidate logical model and the table name, field information and category label of the at least one candidate logical model;

[0162] The model matching submodule is further configured to perform similarity matching on the first data table and the at least one candidate logical model based on the table name, field information, at least one first category label corresponding to the first vector, and the table name, field information, and category label of the at least one candidate logical model, to obtain the logical model matching result corresponding to the first vector. The logical model matching result is used to indicate at least one first logical model corresponding to the first vector.

[0163] The field matching submodule is used to perform field matching between the first data table and the at least one first logical model based on the field information in the first data table and the field information in the at least one first logical model respectively.

[0164] An evaluation submodule is used to determine whether the first data table meets the enterprise's evaluation requirements if a logical model matching the first data table exists in at least one first logical model.

[0165] In some embodiments, the classification matching submodule is configured to:

[0166] The similarity between the first vector and the at least one vector is calculated respectively to obtain the first similarity value corresponding to the at least one vector;

[0167] Based on the first similarity value corresponding to each of the at least one vector, a plurality of second vectors are determined from the at least one vector;

[0168] Based on the first similarity values ​​corresponding to the plurality of second vectors respectively, the second vectors are aggregated by mean to obtain the at least one first classification label.

[0169] In some embodiments, the model matching submodule is configured to:

[0170] The similarity between the table name of the first data table and the table name of the at least one candidate logical model is calculated to obtain the second similarity value corresponding to the at least one candidate logical model.

[0171] The similarity between the field information of the first data table and the field information of the at least one candidate logical model is calculated to obtain the third similarity value corresponding to the at least one candidate logical model respectively.

[0172] The similarity between at least one first category label of the first data table and the category label of the at least one candidate logical model is calculated to obtain the fourth similarity value corresponding to the at least one candidate logical model.

[0173] The second similarity value, the third similarity value, and the fourth similarity value of each candidate logical model are summed to obtain the total similarity value of each candidate logical model.

[0174] Based on the total similarity value of each of the candidate logical models, the at least one first logical model is determined from the at least one candidate logical model.

[0175] In some embodiments, the field information includes field name, field type, and foreign key result, wherein the foreign key result includes two results: foreign key and not foreign key; the vector database corresponds to an index library, and the index library stores standard field information;

[0176] The field matching submodule is used for:

[0177] The first score of the first data table is obtained by weighted summing of the field names, field types, and foreign key results.

[0178] For each first logical model, the field name, field type, and foreign key result of each first logical model are weighted and summed to obtain the first score of each first logical model;

[0179] Based on the first score of the first data table and the first score corresponding to each of the first logical models, the first data table is matched with the at least one first logical model for the fields respectively.

[0180] In some embodiments, the vector conversion module 530 is configured to:

[0181] A pre-trained semantic vectorization model is used to convert the reference text fields into the first vector;

[0182] The first vector is a high-dimensional vector.

[0183] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0184] Please refer to Figure 6 This diagram illustrates a structural block diagram of a computer device according to an embodiment of this application. The computer device is used to implement the data table evaluation method provided in the above embodiments. Specifically:

[0185] The computer device 600 includes a CPU (Central Processing Unit) 601, a system memory 604 including RAM (Random Access Memory) 602 and ROM (Read-Only Memory) 603, and a system bus 605 connecting the system memory 604 and the central processing unit 601. The computer device 600 also includes a basic I / O (Input / Output) system 606 that facilitates information transfer between various components within the computer, and a mass storage device 607 for storing the operating system 613, application programs 614, and other program modules 615.

[0186] The basic input / output system 606 includes a display 608 for displaying information and an input device 609 for user input, such as a mouse or keyboard. Both the display 608 and the input device 609 are connected to the central processing unit 601 via an input / output controller 610 connected to the system bus 605. The basic input / output system 606 may also include the input / output controller 610 for receiving and processing input from multiple other devices such as a keyboard, mouse, or electronic stylus. Similarly, the input / output controller 610 also provides output to a display screen, printer, or other types of output devices.

[0187] The mass storage device 607 is connected to the central processing unit 601 via a mass storage controller (not shown) connected to the system bus 605. The mass storage device 607 and its associated computer-readable media provide non-volatile storage for the computer device 600. That is, the mass storage device 607 may include computer-readable media (not shown) such as a hard disk or a CD-ROM (Compact Disc Read-Only Memory) drive.

[0188] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include RAM, ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash memory or other solid-state storage, CD-ROM, DVD (Digital Video Disc) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage, or other magnetic storage devices. Of course, those skilled in the art will recognize that the computer storage media are not limited to the above-mentioned types. The system memory 604 and the mass storage device 607 described above can be collectively referred to as memory.

[0189] According to various embodiments of this application, the computer device 600 can also be connected to a remote computer on a network, such as the Internet. That is, the computer device 600 can be connected to a network 612 via a network interface unit 611 connected to the system bus 605, or the network interface unit 611 can be used to connect to other types of networks or remote computer systems (not shown).

[0190] In an exemplary embodiment, a computer-readable storage medium is also provided, wherein a computer program is stored therein, which, when executed by a processor, implements the above-described method for evaluating the data table.

[0191] In an exemplary embodiment, a computer program product is also provided, which is loaded and executed by a processor to implement the evaluation method of the data table described above.

[0192] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0193] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for evaluating a data table, characterized in that, The method includes: Obtain the enterprise's first data table, vector database, and at least one standard field information; The valid metadata in the first data table is processed by Hypertext Markup Language (HTML) tagging to obtain reference text fields, which include table name, field name and category. Convert the reference text field into a first vector; Based on the first vector, the vector database, and the field information of the at least one standard, the evaluation result of the first data table is determined, and the evaluation result is used to indicate whether the first data table meets the evaluation requirements of the enterprise.

2. The method according to claim 1, characterized in that, The evaluation result of the first data table is determined based on the first vector, the vector database, and the field information of at least one standard, including: The first vector is matched with at least one vector in the vector database to obtain a classification matching result for the first vector. The classification matching result is used to indicate at least one first category label corresponding to the first vector. Obtain at least one candidate logical model and the table name, field information and category label of the at least one candidate logical model; Based on the table name, field information, and at least one first category label corresponding to the first vector of the first data table, as well as the table name, field information, and category label of the at least one candidate logical model, similarity matching is performed on the first data table and the at least one candidate logical model respectively to obtain the logical model matching result corresponding to the first vector. The logical model matching result is used to indicate at least one first logical model corresponding to the first vector. Based on the field information in the first data table and the field information in the at least one first logical model, the first data table and the at least one first logical model are respectively matched by field; If a logical model matching the first data table exists in at least one first logical model, the first data table is determined to meet the enterprise's evaluation requirements.

3. The method according to claim 2, characterized in that, The step of performing similarity matching between the first vector and at least one vector in the vector database to obtain the classification matching result of the first vector includes: The similarity between the first vector and the at least one vector is calculated respectively to obtain the first similarity value corresponding to the at least one vector; Based on the first similarity value corresponding to each of the at least one vector, a plurality of second vectors are determined from the at least one vector; Based on the first similarity values ​​corresponding to the plurality of second vectors respectively, the second vectors are aggregated by mean to obtain the at least one first classification label.

4. The method according to claim 2, characterized in that, The step of performing similarity matching on the first data table and the at least one candidate logical model based on the table name, field information, at least one first category label corresponding to the first vector, and the table name, field information, and category label of the at least one candidate logical model, to obtain the logical model matching result corresponding to the first vector, includes: The similarity between the table name of the first data table and the table name of the at least one candidate logical model is calculated to obtain the second similarity value corresponding to the at least one candidate logical model. The similarity between the field information of the first data table and the field information of the at least one candidate logical model is calculated to obtain the third similarity value corresponding to the at least one candidate logical model respectively. The similarity between at least one first category label of the first data table and the category label of the at least one candidate logical model is calculated to obtain the fourth similarity value corresponding to the at least one candidate logical model. The second similarity value, the third similarity value, and the fourth similarity value of each candidate logical model are summed to obtain the total similarity value of each candidate logical model. Based on the total similarity value of each of the candidate logical models, the at least one first logical model is determined from the at least one candidate logical model.

5. The method according to claim 2, characterized in that, The field information includes field name, field type, and foreign key result, whereby the foreign key result includes two types: foreign key and not foreign key; the vector database corresponds to an index library, which stores standard field information. The step of matching fields in the first data table and the at least one first logical model based on field information in the first data table includes: The first score of the first data table is obtained by weighted summing of the field names, field types, and foreign key results. For each first logical model, the field name, field type, and foreign key result of each first logical model are weighted and summed to obtain the first score of each first logical model; Based on the first score of the first data table and the first score corresponding to each of the first logical models, the first data table is matched with the at least one first logical model for the fields respectively.

6. The method according to any one of claims 1 to 5, characterized in that, The step of converting the reference text field into a first vector includes: A pre-trained semantic vectorization model is used to convert the reference text fields into the first vector; The first vector is a high-dimensional vector.

7. An evaluation device for a data table, characterized in that, The device includes: The acquisition module is used to acquire the enterprise's first data table, vector database, and at least one standard field information; The tagging module is used to perform Hypertext Markup Language (HTML) tagging on the valid metadata in the first data table to obtain reference text fields, which include table name, field name, and category. A vector conversion module is used to convert the reference text field into a first vector; The result determination module is used to determine the evaluation result of the first data table based on the first vector, the vector database, and the field information of the at least one standard. The evaluation result is used to indicate whether the first data table meets the evaluation requirements of the enterprise.

8. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the evaluation method of the data table according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which is loaded and executed by a processor to implement the evaluation method of the data table according to any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product is loaded and executed by a processor to implement the evaluation method of the data table as described in any one of claims 1 to 6.

Images