A bloodline data mining identification method and device

Abstract

The embodiment of the application provides a blood relationship data mining and identifying method and device, which can be used in the field of artificial intelligence technology, and the method comprises the following steps: obtaining metadata information of a database, wherein the metadata information comprises metadata and metadata attributes; performing similarity calculation on to-be-mined metadata and candidate blood relationship data according to the metadata attributes and the metadata, so as to obtain final similarity; and determining blood relationship data of the to-be-mined metadata from the candidate blood relationship data according to the final similarity. The method can automatically mine underlying blood relationship data from the database, obtain complete data blood relationship, and meet the mining requirement of underlying data blood relationship.

Description

Technical Field

[0001] This invention relates to the field of computer technology, particularly to the field of artificial intelligence technology, and especially to a method and apparatus for kinship data mining and identification. Background Technology

[0002] Data lineage refers to the relationships formed between data during their creation, processing, flow, and eventual disappearance, similar to kinship in human society. Lineage analysis is frequently used in data governance as a means to ensure data fusion and enable traceability of data processing. Related technologies primarily obtain field-level relationships by parsing join conditions in SQL queries, with some solutions using aggregate functions to obtain field-level fusion paths. However, in the task of mining underlying data lineage, due to limitations in database development standards and other development habits, functions relying solely on SQL for data querying and insertion are extremely rare. Furthermore, join conditions in SQL statements cannot cover all potential data lineages, failing to obtain complete data lineage relationships and thus not meeting the needs of mining underlying data lineage. Summary of the Invention

[0003] One object of this invention is to provide a kinship data mining and identification method that can automatically mine underlying kinship data from a database to obtain complete data kinship relationships, thus meeting the needs of underlying data kinship relationship mining. Another object of this invention is to provide a kinship data mining and identification device. A further object of this invention is to provide a computer-readable medium. A still other object of this invention is to provide a computer device.

[0004] To achieve the above objectives, this invention discloses a method for kinship data mining and identification, comprising:

[0005] Retrieve the database's metadata information, which includes metadata and metadata attributes;

[0006] Based on metadata attributes and metadata, the similarity between the metadata to be mined and the candidate lineage data is calculated to obtain the final similarity.

[0007] Based on the final similarity, the lineage data of the metadata to be mined is determined from the candidate lineage data.

[0008] Preferably, before calculating the similarity between the metadata to be mined and the candidate lineage data based on metadata attributes and metadata to obtain the final similarity, the following steps are also included:

[0009] The sample size is obtained by using a histogram sampling algorithm based on the number of rows in the metadata.

[0010] Based on the sample size, the metadata in the metadata information is sampled to obtain the sampled metadata.

[0011] Preferably, based on metadata attributes and metadata, a similarity calculation is performed between the metadata to be mined and the candidate lineage data to obtain the final similarity, including:

[0012] Based on the metadata attributes, the similarity between the attributes of the metadata to be mined and the candidate lineage data is calculated to obtain the attribute similarity. The attribute similarity includes column name similarity, column annotation similarity, and table name similarity.

[0013] Generate naming similarity based on column name similarity, column comment similarity, and table name similarity;

[0014] Based on the metadata, the similarity between the metadata to be mined and the candidate lineage data is calculated to obtain the data similarity.

[0015] Based on naming similarity and data similarity, the final similarity between the metadata to be mined and the candidate lineage data is obtained.

[0016] Preferably, based on metadata attributes, the similarity between the attributes of the metadata to be mined and the candidate lineage data is calculated to obtain attribute similarity, including:

[0017] If the pre-generated set of general phrases includes metadata attributes, the similarity between the metadata attributes of the metadata to be mined and the metadata attributes of the candidate lineage data is calculated using the Jaccard similarity coefficient algorithm to obtain the attribute similarity.

[0018] If the pre-generated general phrase set does not include metadata attributes, the metadata attributes of the metadata to be mined and the candidate lineage data are transformed into word vectors using a word vector model to obtain the set of vectors to be mined and the set of candidate lineage vectors.

[0019] The word shift distance algorithm is used to calculate the similarity between the set of vectors to be mined and the set of candidate lineage vectors, and the attribute similarity is obtained.

[0020] Preferably, naming similarity is generated based on column name similarity, column comment similarity, and table name similarity, including:

[0021] By using a normalization function and preset attribute weights, the similarity of column names, column comments, and table names is calculated to obtain the naming similarity.

[0022] Preferably, based on the metadata, the similarity between the metadata to be mined and the candidate lineage data is calculated to obtain the data similarity, including:

[0023] Based on the metadata, the amount of information contained in the metadata to be mined is calculated to obtain the first correction parameter;

[0024] Based on the metadata, data selection is performed on the data of the metadata to be mined to obtain the second correction parameter;

[0025] Based on the metadata, the correlation degree between the metadata to be mined and the candidate lineage data is calculated to obtain the data correlation degree.

[0026] The data similarity is calculated using a normalization function based on the first correction parameter, the second correction parameter, and the data correlation.

[0027] Preferably, based on the final similarity, the lineage data for which metadata to be mined is determined from the candidate lineage data, including:

[0028] Based on the final similarity, select the target similarity;

[0029] Candidate lineage data corresponding to target similarity are identified as lineage data for metadata to be mined.

[0030] Preferably, the target similarity is selected based on the final similarity, including:

[0031] Rank the final similarity scores;

[0032] Based on the final similarity after sorting, the target similarity of the specified sequence is selected.

[0033] This invention also discloses a bloodline data mining and identification device, comprising:

[0034] The acquisition unit is used to acquire metadata information from the database. The metadata information includes metadata and metadata attributes.

[0035] The similarity calculation unit is used to calculate the similarity between the metadata to be mined and the candidate lineage data based on the metadata attributes and metadata, and obtain the final similarity.

[0036] The determination unit is used to identify the lineage data of the metadata to be mined from the candidate lineage data based on the final similarity.

[0037] The present invention also discloses a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the method described above.

[0038] The present invention also discloses a computer device, including a memory and a processor, wherein the memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions, wherein the processor executes the program to implement the method described above.

[0039] The present invention also discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the method described above.

[0040] This invention acquires metadata information from a database, including metadata and metadata attributes; based on the metadata attributes and metadata, it calculates the similarity between the metadata to be mined and candidate lineage data to obtain the final similarity; based on the final similarity, it determines the lineage data of the metadata to be mined from the candidate lineage data, which can automatically mine the underlying lineage data from the database to obtain the complete data lineage relationship and meet the needs of underlying data lineage relationship mining. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating a bloodline data mining and identification method provided in an embodiment of the present invention;

[0043] Figure 2 A flowchart illustrating another bloodline data mining and identification method provided in this embodiment of the invention;

[0044] Figure 3 This is a schematic diagram of the structure of a bloodline data mining and identification device provided in an embodiment of the present invention;

[0045] Figure 4 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0046] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0047] It should be noted that the bloodline data mining and identification method and apparatus disclosed in this application can be used in the field of artificial intelligence technology, or in any field other than artificial intelligence technology. The application field of the bloodline data mining and identification method and apparatus disclosed in this application is not limited.

[0048] To facilitate understanding of the technical solution provided in this application, the relevant content of the technical solution will be explained below. In a broad sense, data lineage encompasses the path of data aggregation in a big data architecture, manifested as hierarchical relationships at the table level or even column level. Generally, big data R&D systems are clearly defined, and these hierarchical relationships are easily registered and marked during the R&D process. However, data lineage relationships within the scope of source data (or data attached to source data) rely on the product business R&D department's design for functional implementation, lacking the involvement of data management departments. Besides the existence of conventional database logical foreign keys, there are many redundant data designs for the convenience of functional development or performance improvement. Under the current consensus development specifications for business databases, foreign key referencing technology has been largely banned, and it is difficult to guarantee the integrity of documentation under rapid product iteration. Therefore, data lineage relationships within the scope of source data (or data attached to source data) gradually become information buried in the memories of R&D personnel, making it difficult to serve as data assets to support product development. This type of data lineage is referred to as the underlying data lineage in this paper.

[0049] The value of data lineage in enterprise data governance is reflected in:

[0050] First, in the data processing process, from the data source to the final data generation, each step can potentially lead to data quality issues. Data lineage allows for data quality detection and processing at each stage. A usable data lineage also facilitates establishing mutual trust among various roles such as operations, product, and data development engineers regarding data logic, data requirements, and final data results, reducing communication costs in data service development. Second, operational analysts may have diverse data retrieval needs. In a self-service analysis workflow, data lineage can be presented to analysts as an important data asset, facilitating data tracing and origination. Furthermore, it can serve as a project asset, providing crucial reference for system design and helping to determine the rationality of data redundancy and reuse designs.

[0051] In summary, data lineage is a highly valuable data asset and a crucial component of data infrastructure. This invention can automatically extract underlying data lineage relationships from business databases, thereby solving the problem of difficult underlying data lineage mining.

[0052] The following uses a kinship data mining and identification device as an example to illustrate the implementation process of the kinship data mining and identification method provided in this embodiment of the invention. It is understood that the execution subject of the kinship data mining and identification method provided in this embodiment of the invention includes, but is not limited to, a kinship data mining and identification device.

[0053] Figure 1 A flowchart of a bloodline data mining and identification method provided in an embodiment of the present invention is shown below. Figure 1 As shown, the method includes:

[0054] Step 101: Obtain the database metadata information, which includes metadata and metadata attributes.

[0055] Step 102: Based on the metadata attributes and metadata, calculate the similarity between the metadata to be mined and the candidate lineage data to obtain the final similarity.

[0056] Step 103: Based on the final similarity, determine the lineage data of the metadata to be mined from the candidate lineage data.

[0057] It is worth noting that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. The user information in the embodiments of this application was obtained through legal and compliant means, and the acquisition, storage, use, and processing of user information have been authorized and agreed upon by the client.

[0058] In the technical solution provided by the embodiments of the present invention, metadata information of the database is obtained, including metadata and metadata attributes; based on the metadata attributes and metadata, the similarity between the metadata to be mined and the candidate lineage data is calculated to obtain the final similarity; based on the final similarity, the lineage data of the metadata to be mined is determined from the candidate lineage data, which can automatically mine the underlying lineage data from the database to obtain the complete data lineage relationship and meet the needs of underlying data lineage relationship mining.

[0059] Figure 2 A flowchart of another bloodline data mining and identification method provided in an embodiment of the present invention is shown below. Figure 2 As shown, the method includes:

[0060] Step 201: Obtain the sample size based on the number of data rows in the metadata using the histogram sampling algorithm.

[0061] In this embodiment of the invention, each step is performed by a bloodline data mining and identification device, and the metadata is the underlying data. The bloodline data mining and identification method provided in this embodiment of the invention can be applied to the mining of bloodline relationships in the underlying data.

[0062] Specifically, through The sampling parameters and the number of data rows of metadata are calculated to obtain the sample size. The sampling parameters include histogram size k, maximum relative error in bin size f, and error probability gamma γ. The sampling parameters can be set according to actual needs, and this embodiment of the invention does not limit them. n is the number of data rows of metadata, and r is the sample size.

[0063] Step 202: According to the sample size, sample the metadata in the metadata information to obtain the sampled metadata.

[0064] Specifically, with a sample size of r, the metadata in the metadata information is randomly sampled, and r rows of metadata are extracted.

[0065] In this embodiment of the invention, the number of data rows in the data tables of the database can be pre-counted. If the number of data rows exceeds a set threshold, it indicates a large amount of data in the database. Steps 201 to 202 are then executed to sample the metadata. Since the subsequent mining process involves a full table traversal, this can lead to significant performance pressure. To conserve database computing resources and ensure the accuracy of the mining, a portion of the metadata can be sampled for subsequent calculations. The row count threshold is set according to actual needs; this embodiment of the invention does not limit its setting, as long as the data volume is sufficient to ensure the accuracy of the mining.

[0066] It is worth noting that if metadata sampling is performed according to the sample size, then for the calculation formula in the subsequent mining process, let n = r.

[0067] Step 203: Obtain the metadata information of the database. The metadata information includes metadata and metadata attributes.

[0068] In this embodiment of the invention, metadata information is obtained from the database, with metadata columns as the unit. Metadata attributes include, but are not limited to, column name, column comment, and table name, wherein the table name is the name of the data table to which the metadata column belongs.

[0069] Furthermore, the metadata attributes are preprocessed through word segmentation to obtain preprocessed metadata attributes, which are then used to continue subsequent steps. Table names and column names, due to technical limitations, generally only support English and are presented as combinations of English words (or abbreviations) according to their actual meaning. Comments are generally Chinese phrases. Specifically, metadata attributes are separated and filtered based on a specific strategy and configured stop word library. The specific strategy can be set according to actual needs, and this embodiment of the invention does not limit it. As an optional solution, table names and column names are separated using a specified delimiter; column comments are separated using Chinese word segmentation technology.

[0070] Step 204: Based on the metadata attributes, calculate the similarity between the attributes of the metadata to be mined and the candidate lineage data to obtain the attribute similarity. The attribute similarity includes column name similarity, column annotation similarity, and table name similarity.

[0071] In this embodiment of the invention, within organizations with high levels of project management and R&D management maturity, there are unified and common terms for frequently used technologies and business objects. For example, for payment, the unified English phrase is "pay," rather than synonyms such as "payment" or "spend." Therefore, a set of common phrases can generally be compiled, and the words and abbreviations used in the attributes are all included in this set, resulting in a high degree of overall attribute standardization. For databases with standardized attribute management and all attributes derived from the set of common phrases, a similarity algorithm for the set is used to calculate similarity. For databases where at least one attribute does not exist in the set of common phrases, an algorithm based on open-source pre-trained word vectors and employing natural language processing is used to calculate short text similarity.

[0072] Specifically, step 204 includes:

[0073] Step 2041: Determine whether the pre-generated general phrase set includes metadata attributes. If yes, proceed to step 2042; otherwise, proceed to step 2043.

[0074] In this embodiment of the invention, if each metadata attribute exists in the common phrase set, it indicates that the metadata attribute naming is standardized, and step 2042 is executed; if at least one metadata attribute does not exist in the common phrase set, it indicates that the metadata attribute naming is not standardized, and step 2043 is executed.

[0075] Step 2042: Calculate the similarity of the metadata attributes of the metadata to be mined and the candidate lineage data using the Jaccard similarity coefficient algorithm to obtain the attribute similarity, and then proceed to step 205.

[0076] Specifically, the number of elements in the intersection and the number of elements in the union of the metadata attributes of the metadata to be mined and the candidate lineage data are counted; the number of elements in the intersection is divided by the number of elements in the union to obtain the attribute similarity.

[0077] It is worth noting that this calculation method requires preprocessing: for English table names and column names, word segmentation can be performed based on common delimiters in the naming (such as the table name "user_info", where the delimiter is the underscore "_") to obtain a phrase set; for Chinese column comments without delimiters, word segmentation can be performed using commonly used word segmenters in the industry, such as jieba, to obtain a phrase set.

[0078] In this embodiment of the invention, if the metadata attributes include column name, column comment, and table name, then the column name similarity, column comment similarity, and table name similarity are calculated accordingly.

[0079] Step 2043: Use a word vector model to convert the metadata attributes of the metadata to be mined and the candidate lineage data into word vectors respectively, to obtain the set of vectors to be mined and the set of candidate lineage vectors.

[0080] In the embodiments of the present invention, when there is no unified general vocabulary for attribute management, the namer determines naming phrases and combinations based on their own understanding of services and own English proficiency. In this case, the same service or technical object may have different names in different tables. For example, for the application service, the English phrases for application may be selected as "APPLICATION", "CONTRACT", etc., and synonym words such as "handle", "request" may also appear in Chinese annotations. In this case, it is inappropriate to crudely regard synonyms as completely different phrases. It is necessary to consider the semantic approximation degree and word dependency to calculate the similarity, which requires the assistance of relevant natural language processing (NLP) technologies. Words are converted into structured vectors according to their dependency and meaning, and the Euclidean distance between the vectors corresponding to two words can represent the dissimilarity between words. Such conversion algorithms are already mature. The present invention does not train word vectors additionally, and directly adopts an existing word vector model to perform word vector conversion.

[0081] Specifically, the metadata attributes of the metadata to be mined are input into a word vector model for word vector conversion, and a vector set to be mined is output; the metadata attributes of candidate blood relationship data are input into the word vector model for word vector conversion, and a candidate blood relationship vector set is output.

[0082] Step 2044: Through the Word Mover's Distance (WMD for short) algorithm, perform similarity calculation on the vector set to be mined and the candidate blood relationship vector set to obtain attribute similarity.

[0083] In the embodiments of the present invention, the metadata attributes include column names, column annotations and table names, then the corresponding vector set to be mined includes the column name vector set to be mined, the column annotation vector set to be mined and the table name vector set to be mined, the corresponding candidate blood relationship vector set includes the candidate blood relationship column name vector set, the candidate blood relationship column annotation vector set and the candidate blood relationship table name vector set, and the corresponding attribute similarity includes column name similarity, column annotation similarity and table name similarity.

[0084] Specifically, the column name vector set to be mined and the candidate blood relationship column name vector set are input into the WMD algorithm for similarity calculation to obtain column name similarity; the column annotation vector set to be mined and the candidate blood relationship column annotation vector set are input into the WMD algorithm for similarity calculation to obtain column annotation similarity; the table name vector set to be mined and the candidate blood relationship table name vector set are input into the WMD algorithm for similarity calculation to obtain table name similarity.

[0085] Step 205: Generate naming similarity according to the column name similarity, column annotation similarity and table name similarity.

[0086] In this embodiment of the invention, the similarity of column names, column comments, and table names is calculated using a normalization function and preset attribute weights to obtain the naming similarity. The attribute weights include column name weights, column comment weights, and table name weights, which are set according to actual needs; this embodiment of the invention does not impose any limitations on them.

[0087] Specifically, taking a column name weight of 0.4, a column comment weight of 0.4, and a table name weight of 0.2 as an example, the similarity of column names, column comments, and table names is calculated using n_sim = 0.4 × norm(coln_sim) + 0.4 × norm(comment_sim) + 0.2 × norm(tablen_sim) to obtain the naming similarity. Here, norm refers to the use of the minimum-maximum normalization function, and the normalized interval is [0,1]; n_sim is the naming similarity; coln_sim is the column name similarity; comment_sim is the column comment similarity; and tablen_sim is the table name similarity.

[0088] Step 206: Based on the metadata, calculate the similarity between the metadata to be mined and the candidate lineage data to obtain the data similarity.

[0089] In this embodiment of the invention, step 206 specifically includes:

[0090] Step 2061: Based on the metadata, calculate the amount of information to be contained in the metadata to be mined, and obtain the first correction parameter.

[0091] In this embodiment of the invention, the information capacity measures the richness of the data style stored in the column. This is calculated only for the single column containing the metadata to be mined, and its main purpose is to correct the degree of data correlation to obtain the final data similarity. When a column stores data in a very simple format, such as a single decimal digit, another unrelated column may have a high degree of data correlation with that column because it also stores a single decimal digit (since both columns have a value range of 0-9). To correct this situation, this invention considers that the simpler the data format, i.e., the smaller the information capacity, the lower the degree of data correlation representing the degree of data similarity.

[0092] Specifically, through The information capacity of the metadata to be mined is calculated to obtain the first correction parameter. Here, IC is the first correction parameter, i.e., the information capacity, cnt. cat This refers to the total number of occurrences of the corresponding character class, c cat This refers to the number of values ​​that the corresponding character can take, where n is the number of data rows. The range of values ​​for cat includes uppercase letters, lowercase letters, numbers, and Chinese characters.

[0093] For example, column col_a stores a four-digit ID, where the first two digits are all uppercase letters and the last two digits are all decimal numbers. According to the formula above, the information capacity of column col_a is lg(26)×2 / 4+lg(10)×2 / 4, approximately 2.78. When calculating the information capacity, the c of uppercase letters and lowercase letters... cat The value is 26, the default value for Chinese characters is 100000, and the values ​​for other punctuation and special characters are determined according to the actual data. cat The value of .

[0094] Step 2062: Based on the metadata, select data from the metadata to be mined to obtain the second correction parameter.

[0095] In this embodiment of the invention, data selectivity measures the degree of data duplication in a column of data, and is also used to correct the degree of data association to obtain data similarity. Consider an extreme case where two columns store only one value, for example, 1 million data entries, but all of them are 1. In this case, the degree of data association is calculated to be 1 (maximum value) according to the following formula. However, this does not necessarily mean that the two columns are related. In this case, when calculating data similarity, data selectivity should be used to reduce the weight of the degree of data association.

[0096] Specifically, through The data to be mined for metadata is selected to obtain the second correction parameter. Here, CD is the second correction parameter, count(col) is the total number of data items stored in this column, and count(distinct_col) is the number of duplicate values ​​in this column that do not need to be retrieved.

[0097] For example, if the column stores 3 'a's and 4 'b's, then according to the formula above, the second correction parameter CD is 5 / 7.

[0098] Step 2063: Based on the metadata, calculate the correlation degree between the metadata to be mined and the candidate lineage data to obtain the data correlation degree.

[0099] In this embodiment of the invention, the degree of data correlation between two columns of data is calculated by measuring the size of the intersection of their values. The larger the intersection, the greater the likelihood that the data in one column comes from the other. That is, when calculating the degree of correlation between the metadata to be mined and the candidate lineage data, the greater the degree of correlation, the greater the likelihood that the metadata column to be mined comes from the candidate lineage data column.

[0100] Specifically, through The correlation degree between the metadata to be mined and the candidate lineage data is calculated to obtain the data correlation degree. Here, RT represents the data correlation degree; card refers to the cardinality of the set, i.e., the number of elements in the set; A refers to the data set stored in the metadata column to be mined, and B refers to the data set stored in the candidate lineage data column.

[0101] Step 2064: Calculate the data similarity using a normalization function based on the first correction parameter, the second correction parameter, and the data correlation degree.

[0102] Specifically, the data similarity is calculated by using d_sim = norm(IC×CD)×RT, the first correction parameter, the second correction parameter, and the data correlation degree. Here, d_sim is the data similarity, IC is the first correction parameter, CD is the second correction parameter, RT is the data correlation degree, and norm refers to the normalization function (calculated for each data row of the metadata column to be mined), with the normalized interval being [0,1].

[0103] Step 207: Based on naming similarity and data similarity, obtain the final similarity between the metadata to be mined and the candidate lineage data.

[0104] In this embodiment of the invention, the data weight β is set according to actual needs, and this embodiment of the invention does not limit it.

[0105] Specifically, taking a data weight β of 0.1 as an example, through... The naming similarity and data similarity are calculated to obtain the final similarity between the metadata to be mined and the candidate lineage data. Here, β is the data weight, sim is the final similarity, d_sim is the data similarity, and n_sim is the naming similarity.

[0106] Step 208: Select the target similarity based on the final similarity.

[0107] As an optional approach, the final similarities are sorted; based on the sorted final similarities, the target similarity for a specified sequence is selected. Either the highest-ranked final similarity is chosen as the target similarity, or the top m final similarities in descending order are selected, where m is an integer and its value can be determined according to actual needs.

[0108] Step 209: Determine the candidate lineage data corresponding to the target similarity as the lineage data of the metadata to be mined.

[0109] In this embodiment of the invention, the candidate lineage data corresponding to the target similarity is at least one data column that is most similar to the metadata to be mined, that is, the lineage data of the metadata to be mined, and there is a lineage relationship between the metadata to be mined and the lineage data.

[0110] This invention is primarily based on a weighted index system. Therefore, by adjusting the weights of different similarity measurement components, it can adapt to different R&D specifications and habits. For example, when database data definition language (DDL) statements are well-managed and there are strict specifications for naming database objects, the weight value of naming similarity can be appropriately increased, making the algorithm more inclined to consider data columns with similar names as related. Similar parameter adjustments give this invention high customization potential and make it adaptable to a wider range of scenarios.

[0111] It is worth noting that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. The user information in the embodiments of this application was obtained through legal and compliant means, and the acquisition, storage, use, and processing of user information have been authorized and agreed upon by the client.

[0112] In the technical solution of the bloodline data mining and identification method provided in this embodiment of the invention, metadata information of the database is obtained, including metadata and metadata attributes; based on the metadata attributes and metadata, the similarity between the metadata to be mined and the candidate bloodline data is calculated to obtain the final similarity; based on the final similarity, the bloodline data of the metadata to be mined is determined from the candidate bloodline data, which can automatically mine the underlying bloodline data from the database to obtain the complete data bloodline relationship and meet the needs of underlying data bloodline relationship mining.

[0113] Figure 3 This is a schematic diagram of a bloodline data mining and identification device provided in an embodiment of the present invention. This device is used to execute the aforementioned bloodline data mining and identification method, such as... Figure 3 As shown, the device includes: an acquisition unit 11, a similarity calculation unit 12, and a determination unit 13.

[0114] The acquisition unit 11 is used to acquire the metadata information of the database, which includes metadata and metadata attributes.

[0115] The similarity calculation unit 12 is used to calculate the similarity between the metadata to be mined and the candidate lineage data based on the metadata attributes and metadata, and obtain the final similarity.

[0116] The determination unit 13 is used to determine the lineage data of the metadata to be mined from the candidate lineage data based on the final similarity.

[0117] In this embodiment of the invention, the device further includes a generation unit 14 and a sampling unit 15.

[0118] The generation unit 14 is used to obtain the sample size based on the number of data rows in the metadata using a histogram sampling algorithm.

[0119] The sampling unit 15 is used to sample the metadata in the metadata information according to the sample size to obtain the sampled metadata.

[0120] In this embodiment of the invention, the similarity calculation unit 12 is specifically used to calculate the similarity of the attributes of the metadata to be mined and the candidate lineage data according to the metadata attributes, and obtain the attribute similarity. The attribute similarity includes column name similarity, column annotation similarity and table name similarity. Based on the column name similarity, column annotation similarity and table name similarity, a naming similarity is generated. Based on the metadata, the data similarity of the metadata to be mined and the candidate lineage data is calculated to obtain the data similarity. Based on the naming similarity and data similarity, the final similarity between the metadata to be mined and the candidate lineage data is obtained.

[0121] In this embodiment of the invention, the similarity calculation unit 12 is specifically used to calculate the similarity between the metadata attributes of the metadata to be mined and the metadata attributes of the candidate lineage data by using the Jaccard similarity coefficient algorithm if the pre-generated general phrase set includes metadata attributes; if the pre-generated general phrase set does not include metadata attributes, it performs word vector transformation on the metadata attributes of the metadata to be mined and the metadata attributes of the candidate lineage data by using a word vector model to obtain a set of vectors to be mined and a set of candidate lineage vectors; and calculates the similarity between the set of vectors to be mined and the set of candidate lineage vectors by using the word shift distance algorithm to obtain the attribute similarity.

[0122] In this embodiment of the invention, the similarity calculation unit 12 is specifically used to calculate the similarity of column names, column annotations, and table names by using a normalization function and preset attribute weights to obtain the naming similarity.

[0123] In this embodiment of the invention, the similarity calculation unit 12 is specifically used to calculate the amount of information contained in the data of the metadata to be mined based on the metadata, to obtain a first correction parameter; to select data from the data of the metadata to be mined based on the metadata, to obtain a second correction parameter; to calculate the degree of association between the data of the metadata to be mined and the candidate lineage data based on the metadata, to obtain a data association degree; and to calculate the data similarity based on the first correction parameter, the second correction parameter and the data association degree through a normalization function.

[0124] In this embodiment of the invention, the determining unit 13 is specifically used to select the target similarity based on the final similarity; and to determine the candidate lineage data corresponding to the target similarity as the lineage data of the metadata to be mined.

[0125] In this embodiment of the invention, the determining unit 13 is specifically used to sort the final similarity scores; and select the target similarity score of the specified sequence based on the sorted final similarity scores.

[0126] In the solution of this invention embodiment, metadata information of the database is obtained, including metadata and metadata attributes; based on the metadata attributes and metadata, similarity calculation is performed on the metadata to be mined and candidate lineage data to obtain the final similarity; based on the final similarity, the lineage data of the metadata to be mined is determined from the candidate lineage data, which can automatically mine the underlying lineage data from the database to obtain the complete data lineage relationship and meet the needs of underlying data lineage relationship mining.

[0127] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer device, specifically, a computer device can be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0128] This invention provides a computer device including a memory and a processor. The memory is used to store information including program instructions, and the processor is used to control the execution of the program instructions. When the program instructions are loaded and executed by the processor, they implement the steps of the above-described kinship data mining and identification method. For a detailed description, please refer to the above-described kinship data mining and identification method.

[0129] The following is for reference. Figure 4 It shows a schematic diagram of the structure of a computer device 600 suitable for implementing the embodiments of this application.

[0130] like Figure 4 As shown, the computer device 600 includes a central processing unit (CPU) 601, which can perform various appropriate tasks and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the computer device 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0131] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal feedback (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed in storage section 608 as needed.

[0132] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program including program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611.

[0133] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0134] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0135] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0136] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0137] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0138] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0139] The acquisition, storage, use, and processing of data in this application all comply with the relevant provisions of national laws and regulations.

[0140] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0142] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0143] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for identifying lineage through data mining, characterized in that, The method includes: Obtain the metadata information of the database, which includes metadata and metadata attributes; Based on the metadata attributes and metadata, a similarity calculation is performed between the metadata to be mined and the candidate lineage data to obtain a final similarity. This includes: calculating the similarity of attributes between the metadata to be mined and the candidate lineage data based on the metadata attributes to obtain attribute similarity, whereby attribute similarity includes column name similarity, column annotation similarity, and table name similarity; generating a naming similarity based on the column name similarity, column annotation similarity, and table name similarity; calculating the data similarity between the metadata to be mined and the candidate lineage data based on the metadata to be mined to obtain data similarity; and obtaining the final similarity between the metadata to be mined and the candidate lineage data based on the naming similarity and data similarity. Based on the final similarity, the lineage data of the metadata to be mined is determined from the candidate lineage data; The step of calculating the similarity between the metadata to be mined and the candidate lineage data based on the metadata to obtain the data similarity includes: calculating the information capacity of the metadata to be mined based on the metadata to obtain a first correction parameter: according to the formula Determine the first correction parameter, IC is the first correction parameter, cnt cat This refers to the total number of occurrences of the corresponding character class, c cat This refers to the number of possible values ​​for the corresponding character, where n is the number of data rows. Based on the metadata, data selection is performed on the metadata to be mined to obtain the second correction parameter: according to the formula... Determine the second correction parameter, CD is the second correction parameter. The total number of data items stored in column 'col'. The number of duplicate values ​​that do not need to be retrieved after deduplication of the data stored in this column; based on the metadata, the correlation degree between the metadata to be mined and the candidate lineage data is calculated to obtain the data correlation degree; through a normalization function, the data similarity is calculated based on the first correction parameter, the second correction parameter and the data correlation degree.

2. The bloodline data mining and identification method according to claim 1, characterized in that, Before calculating the similarity between the metadata to be mined and the candidate lineage data based on the metadata attributes and metadata to obtain the final similarity, the method further includes: The sample size is obtained by using a histogram sampling algorithm based on the number of data rows in the metadata; According to the stated sample size, the metadata in the metadata information is sampled to obtain the sampled metadata.

3. The bloodline data mining and identification method according to claim 1, characterized in that, The step of calculating the similarity between the attributes of the metadata to be mined and the candidate lineage data based on the metadata attributes to obtain the attribute similarity includes: If the pre-generated set of general phrases includes the metadata attributes, the similarity between the metadata attributes of the metadata to be mined and the candidate lineage data is calculated using the Jaccard similarity coefficient algorithm to obtain the attribute similarity. If the pre-generated general phrase set does not include the metadata attributes, the metadata attributes of the metadata to be mined and the candidate lineage data are converted into word vectors using a word vector model to obtain the vector set to be mined and the vector set of candidate lineage data. The similarity between the set of vectors to be mined and the set of candidate lineage vectors is calculated using the word shift distance algorithm to obtain the attribute similarity.

4. The bloodline data mining and identification method according to claim 1, characterized in that, Based on the column name similarity, column comment similarity, and table name similarity, a naming similarity is generated, including: The similarity of column names, column annotations, and table names is calculated using a normalization function and preset attribute weights to obtain the naming similarity.

5. The bloodline data mining and identification method according to claim 1, characterized in that, The step of determining the lineage data of the metadata to be mined from the candidate lineage data based on the final similarity includes: Based on the final similarity, the target similarity is selected; The candidate lineage data corresponding to the target similarity are determined as the lineage data of the metadata to be mined.

6. The bloodline data mining and identification method according to claim 5, characterized in that, The step of selecting the target similarity based on the final similarity includes: The final similarity scores are then ranked. Based on the final similarity after sorting, the target similarity of the specified sequence is selected.

7. A bloodline data mining and identification device, characterized in that, The device includes: The acquisition unit is used to acquire metadata information from the database, the metadata information including metadata and metadata attributes; The similarity calculation unit is used to calculate the similarity between the metadata to be mined and the candidate lineage data based on the metadata attributes and metadata, and obtain the final similarity. A determining unit is used to determine the lineage data of the metadata to be mined from the candidate lineage data based on the final similarity. Specifically, the similarity calculation unit is used for: calculating the similarity of attributes between the metadata to be mined and the candidate lineage data based on the metadata attributes to obtain attribute similarity, wherein the attribute similarity includes column name similarity, column annotation similarity, and table name similarity; generating naming similarity based on the column name similarity, column annotation similarity, and table name similarity; calculating the data similarity between the metadata to be mined and the candidate lineage data based on the metadata to be mined to obtain data similarity; and obtaining the final similarity between the metadata to be mined and the candidate lineage data based on the naming similarity and data similarity. The similarity calculation unit is specifically used for: calculating the information capacity of the metadata to be mined based on the metadata, and obtaining the first correction parameter: according to the formula Determine the first correction parameter, IC is the first correction parameter, cnt cat This refers to the total number of occurrences of the corresponding character class, c cat This refers to the number of possible values ​​for the corresponding character, where n is the number of data rows. Based on the metadata, data selection is performed on the metadata to be mined to obtain the second correction parameter: according to the formula... Determine the second correction parameter, CD is the second correction parameter. The total number of data items stored in column 'col'. The number of duplicate values ​​that do not need to be retrieved after deduplication of the data stored in this column; based on the metadata, the correlation degree between the metadata to be mined and the candidate lineage data is calculated to obtain the data correlation degree; through a normalization function, the data similarity is calculated based on the first correction parameter, the second correction parameter and the data correlation degree.

8. A computer-readable medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the lineage data mining and identification method as described in any one of claims 1 to 6.

9. A computer device comprising a memory and a processor, the memory for storing information including program instructions, and the processor for controlling the execution of the program instructions, characterized in that, When the program instructions are loaded and executed by the processor, the bloodline data mining and identification method according to any one of claims 1 to 6 is implemented.

Images