A method, device, computer equipment and medium for determining data blood relationship

By analyzing the similarity of data table names and attribute information, and combining this with a trained model to determine the strength of the association between data tables, the problem of inaccurate determination of data lineage was solved, and more accurate identification and classification of data dependencies were achieved.

CN113760918BActive Publication Date: 2024-11-19SHANGHAI AVIATION IND GRP CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111066589.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-09-13
Publication Date
2024-11-19
Estimated Expiration
2041-09-13

AI Technical Summary

Technical Problem

In existing technologies, the lineage relationships between data are not accurately determined, resulting in complex and difficult-to-trace relationships between data tables.

Method used

By analyzing the similarity of names and attribute information of data tables, and combining a trained keyword extraction model and a semantic similarity calculation model, the degree of strong and weak association between data tables is determined, thereby accurately determining the lineage relationship between data.

Benefits of technology

It improves the accuracy of determining data lineage, enabling accurate determination of lineage even when data table names differ significantly, and simplifies the sorting and classification analysis of data dependencies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113760918B_ABST
    Figure CN113760918B_ABST
Patent Text Reader

Abstract

The present application provides a method, apparatus, computer equipment and medium for determining data lineage, the method comprising: determining first data information of a first data table where the first data to be detected is located and second data information of a second data table where the second data to be detected is located according to a first data to be detected and a second data to be detected in a data lineage determination requirement; determining a strong association degree between the first data table and the second data table according to a first name similarity between a data table name of the first data information and a data table name of the second data information; determining a weak association degree between the first data table and the second data table according to a data table name of the first data information and a data table name of the second data information, as well as attribute information corresponding to the first data table and the second data table respectively; determining a lineage relationship between the first data to be detected and the second data to be detected according to the strong association degree and the weak association degree.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular, to a method, apparatus, computer equipment and medium for determining data lineage. Background Art

[0002] Data lineage is the link relationship between data, which can be used to describe the processes and stages of data generation, including addition, processing, combination, conversion, etc. With the rapid development of society, in this era of high-speed data information circulation, whether in the manufacturing industry or in daily life, a large amount of data is generated every moment, and in today's society, these data have become valuable wealth of social production. After analysis by scientific researchers, these data will continue to create higher value.

[0003] Generally, data is stored in a database. Different data is stored in corresponding data tables according to the corresponding classification or source. That is, multiple data tables are stored in the database. As time goes by, the dependency relationship of each data table in the database will change frequently, causing the relationship between data tables to become complicated and difficult to trace. Summary of the invention

[0004] In view of this, the purpose of the present application is to provide a method, apparatus, computer device and medium for determining data lineage relationships, so as to solve the problem of inaccurate determination of lineage relationships between data in the prior art.

[0005] In a first aspect, an embodiment of the present application provides a method for determining data lineage, including:

[0006] Determine the first data to be detected and the second data to be detected in the demand according to the data lineage, determine the first data information of the first data table where the first data to be detected is located and the second data information of the second data table where the second data to be detected is located;

[0007] determining a strong association degree between the first data table and the second data table according to a first name similarity between a data table name of the first data information and a data table name of the second data information;

[0008] Determining a weak association degree between the first data table and the second data table according to a second name similarity between a data table name of the first data information and a data table name of the second data information, and according to an attribute similarity between attribute information of the first data table where the first data information is located and attribute information of the second data table where the second data information is located;

[0009] The blood relationship between the first data to be detected and the second data to be detected is determined according to the strong correlation degree and the weak correlation degree.

[0010] Optionally, determining the degree of strong association between the first data table and the second data table according to a first name similarity between a data table name of the first data information and a data table name of the second data information includes:

[0011] Calculating first name similarity between data table names corresponding to the first data information and the second data information, respectively, based on identical keywords appearing in the data table name of the first data information and the data table name of the second data information;

[0012] The first name similarity between the data table names corresponding to the first data information and the second data information respectively is determined as the strong association degree between the first data table and the second data table.

[0013] Optionally, the second name similarity between the data table name of the first data information and the data table name of the second data information is calculated by the following steps:

[0014] According to the semantics corresponding to the data table name of the first data information and the data table name of the second data information, a second name similarity between the data table names corresponding to the first data information and the second data information is calculated.

[0015] Optionally, the attribute similarity between the attribute information of the first data table where the first data information is located and the attribute information of the second data table where the second data information is located is calculated by the following steps:

[0016] Calculating the first attribute similarity between the data fields corresponding to the first data table and the second data table respectively according to the distance between the character strings of the data fields of the first data table and the data fields of the second data table;

[0017] Calculate the second attribute similarity between the data fields corresponding to the first data table and the second data table respectively according to the intersection and union of the field values ​​of the first data table and the field values ​​of the second data table;

[0018] According to the intersection of the field value range of the first data table and the field value range of the second data table, the third attribute similarity between the field value ranges corresponding to the first data table and the second data table is calculated.

[0019] Optionally, determining the weak association degree between the first data table and the second data table according to a second name similarity between a data table name of the first data information and a data table name of the second data information, and according to an attribute similarity between attribute information of the first data table where the first data information is located and attribute information of the second data table where the second data information is located, includes:

[0020] The weak association degree between the first data table and the second data table is determined based on a weighted sum of the second name similarity, the first attribute similarity, the second attribute similarity, and the third attribute similarity.

[0021] Optionally, determining the blood relationship between the first to-be-detected data and the second to-be-detected data according to the strong correlation degree and the weak correlation degree includes:

[0022] The blood relationship between the first data to be detected and the second data to be detected is determined based on the sum of the product of the first weighted value and the strong association degree and the product of the second weighted value and the weak association degree; the first weighted value is greater than the second weighted value.

[0023] In a second aspect, an embodiment of the present application provides a device for determining data lineage relationship, including:

[0024] A first determination module, configured to determine the first data to be detected and the second data to be detected in the data lineage determination requirement, and determine the first data information of the first data table where the first data to be detected is located and the second data information of the second data table where the second data to be detected is located;

[0025] A second determination module, configured to determine a strong association degree between the first data table and the second data table according to a first name similarity between a data table name of the first data information and a data table name of the second data information;

[0026] a third determination module, configured to determine a weak association degree between the first data table and the second data table according to a second name similarity between a data table name of the first data information and a data table name of the second data information, and according to an attribute similarity between attribute information of the first data table where the first data information is located and attribute information of the second data table where the second data information is located;

[0027] The fourth determination module is used to determine the blood relationship between the first data to be detected and the second data to be detected according to the strong correlation degree and the weak correlation degree.

[0028] Optionally, the second determining module includes:

[0029] A first calculation unit, configured to calculate a first name similarity between data table names corresponding to the first data information and the second data information respectively, based on the same keywords appearing in the data table name of the first data information and the data table name of the second data information;

[0030] The first determining unit is configured to determine a first name similarity between data table names corresponding to the first data information and the second data information respectively as a strong association degree between the first data table and the second data table.

[0031] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0032] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0033] The method for determining data lineage relationship provided in an embodiment of the present application firstly determines, according to the first data to be detected and the second data to be detected in the data lineage determination requirement, first data information of the first data table where the first data to be detected is located and second data information of the second data table where the second data to be detected is located; then, according to the first name similarity between the data table name of the first data information and the data table name of the second data information, the strong association degree between the first data table and the second data table is determined; secondly, according to the second name similarity between the data table name of the first data information and the data table name of the second data information, and according to the attribute similarity between the attribute information of the first data table where the first data information is located and the attribute information of the second data table where the second data information is located, the weak association degree between the first data table and the second data table is determined; finally, according to the strong association degree and the weak association degree, the lineage relationship of the first data to be detected and the second data to be detected is determined.

[0034] In some embodiments, the data table name of some data tables may be just a number or a random string. Therefore, the similarity between the data table names alone cannot accurately determine the blood relationship between different data. Therefore, when determining the blood relationship between different data, in addition to using the data table name of the data table to which the data belongs, the present application also uses the data stored in the two data tables to determine the blood relationship. That is, when the data table names of different data tables are quite different, the blood relationship between different data tables can also be accurately determined based on the similarity of the data stored in different data tables.

[0035] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0037] Figure 1 A flowchart of a method for determining data lineage relationship provided in an embodiment of the present application;

[0038] Figure 2 A flowchart of a method for determining a strong association relationship provided in an embodiment of the present application;

[0039] Figure 3 A schematic diagram of the structure of a device for determining data kinship provided in an embodiment of the present application;

[0040] Figure 4 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0042] Since there is a certain data kinship between the data tables stored in the database, in order to facilitate the sorting of kinship, a data kinship diagram can be drawn for the data tables in the database to reflect the kinship between the data. However, the kinship between the data is complicated, which will lead to the drawn data kinship diagram being relatively complicated. Furthermore, the data kinship diagram displayed graphically cannot allow users to quickly determine the kinship between the data. Moreover, to determine the kinship between the data through the data kinship diagram, the user needs to observe the result himself, which can easily lead to inaccurate kinship between the determined data due to the user's careless observation. Alternatively, the kinship between different data can be determined by the similarity between the data table names corresponding to different data. The data in the data table is diverse, and it is also inaccurate to determine the kinship between the data only by the data table name.

[0043] Based on the above defects, the present application embodiment provides a method for determining data lineage relationship, such as Figure 1 As shown, the following steps are included:

[0044] S101, determining first data to be detected and second data to be detected in a demand according to data lineage, and determining first data information of a first data table where the first data to be detected is located and second data information of a second data table where the second data to be detected is located;

[0045] S102, determining a strong association degree between the first data table and the second data table according to a first name similarity between a data table name of the first data information and a data table name of the second data information;

[0046] S103, determining a weak association degree between the first data table and the second data table according to a second name similarity between a data table name of the first data information and a data table name of the second data information, and according to an attribute similarity between attribute information of the first data information and attribute information of the second data information;

[0047] S104: Determine the blood relationship between the first data to be detected and the second data to be detected according to the strong correlation degree and the weak correlation degree.

[0048] The method for determining data lineage relationship in the embodiment of the present application can be run on a server.

[0049] In the above step S101, the data lineage determination requirement is used to characterize the requirement for determining the lineage relationship between the first data to be detected and the second data to be detected. The data lineage determination requirement can be sent by the user to the server through the terminal device, and the terminal device is a device with data transmission function close to the user side, such as a mobile terminal, a television, a computer, a handheld computer, etc. The data lineage determination requirement carries the first data to be detected and the second data to be detected that need to determine the lineage relationship. The first data to be detected or the second data to be detected can be data in a data table stored in the database, or a data table stored in the database, or multiple data tables stored in the database, wherein the first data to be detected and the second data to be detected are different data. If the first data to be detected or the second data to be detected can be data in a data table stored in the database, then according to the additional information of the data, the data table corresponding to the data can be understood. The data information of the data table is the data carried by the data table, and the data information of the data table includes the data table name and attribute information, and the attribute information includes data fields, field values ​​and field value ranges. The data table name is an identifier that can be used to characterize the data table. The data field of the data table refers to the string corresponding to the header in each data table. The field value is the value in the table corresponding to the data field. The field value range is the value range corresponding to the data field when taking a value. In the present application, if the data corresponding to the first data to be detected and the second data to be detected are clearly given in the data lineage determination requirement, the first data information of the first data table where the first data to be detected is located and the second data information of the second data table where the second data to be detected is located can be determined based on the data corresponding to the first data to be detected and the second data to be detected. If the data corresponding to the first data to be detected is given in the data lineage determination requirement, and the requirement points out that the lineage relationship between the first data to be detected and other data in the database needs to be determined, then the second data to be detected is the other data in the database except the first data to be detected.

[0050] In a specific implementation, the data information includes the data table name and attribute information of the data table. Therefore, the first data information can be obtained from the first data table where the first data to be detected is located through a regular expression, and the second data information can be obtained from the second data table where the second data to be detected is located through a regular expression.

[0051] In the above step S102, the strong association degree is determined by the similarity between the data table names corresponding to different data, and the strong association degree can be determined by character comparison between different data table names. Therefore, according to the same number of characters between the data table names of the first data information and the data table names of the second data information, the first name similarity between the first data table and the second data table is determined, and then the first name similarity is determined as the strong association degree between the first data table and the second data table. The strong association degree refers to the degree to which the similarity between the data table names corresponding to different data is greater than a preset threshold.

[0052] In the above step S103, the weak correlation degree is determined by the similarity between the data table names corresponding to different data and the similarity between the attribute information of the corresponding data tables. The weak correlation degree more represents the similarity of the data stored in the two data tables. The strong correlation degree refers to the degree to which the similarity between the data table names corresponding to different data is less than or equal to the preset threshold.

[0053] In a specific implementation, in addition to determining the similarity between data table names of different data by the same characters, the similarity between different data tables can also be determined by the semantics represented by the data table names. In addition to the similarity between the data table names corresponding to the data tables can represent the blood relationship between the two data tables, the blood relationship between the two data tables can also be determined by the similarity between the data stored in different data tables. Therefore, the weak association degree between the first data table and the second data table can be jointly determined based on the second name similarity between the data table name of the first data information and the data table name of the second data information, and the attribute similarity between the attribute information of the first data information and the attribute information of the second data information.

[0054] In the above step S104, after determining the strong correlation degree and weak correlation degree between the first data table of the first data to be detected and the second data table of the second data to be detected, the blood relationship between the first data to be detected and the second data to be detected can be jointly determined based on the strong correlation degree and the weak correlation degree. In addition to considering the strong correlation degree between the first data table and the second data table, the present application also considers the weak correlation degree between the first data table and the second data table, that is, increases the similarity between the data, so that the determined blood relationship is more accurate.

[0055] Through the above four steps, the data table name of some data tables may be just a number, or a random string. Therefore, the similarity between the data table names alone cannot accurately determine the blood relationship between different data. Therefore, when determining the blood relationship between different data, in addition to using the data table name of the data table to which the data belongs, this application also uses the data stored in the two data tables to determine it. That is, when the data table names of different data tables are quite different, the blood relationship between different data tables can also be accurately determined based on the similarity of the data stored in different data tables.

[0056] A strong correlation is merely the similarity between the names of different data tables. A strong correlation is a relatively intuitive relationship that can be reflected by the number of identical characters. This application provides a method for determining a strong correlation, such as Figure 2 As shown, step S102 includes:

[0057] Step S1021, calculating the first name similarity between the data table names corresponding to the first data information and the second data information respectively according to the same keywords appearing in the data table name of the first data information and the data table name of the second data information;

[0058] Step S1022: Determine the first name similarity between the data table names corresponding to the first data information and the second data information respectively as the strong association degree between the first data table and the second data table.

[0059] In the above step S1021 , the keyword is a word that can characterize the features of the data stored in the data table.

[0060] In a specific implementation, keywords are extracted from the data table name of the first data information and the data table name of the second data information respectively through a trained keyword extraction model, and then the first name similarity between the data table name of the first data information and the data table name of the second data information is calculated based on the number of identical keywords in the data table name of the first data information and the data table name of the second data information. When calculating the first name similarity, the keywords corresponding to the data table name of the first data information can be formed into a first set, and the keywords corresponding to the data table name of the second data information can be formed into a second set. The ratio between the intersection of the first set and the second set and the union of the first set and the second set is determined as the first name similarity between the data table name of the first data information and the data table name of the second data information.

[0061] The trained keyword extraction model is obtained through training with a large amount of training data. The training process includes the following steps:

[0062] Acquire a keyword training set; the key detection training set includes at least one training sample, the training sample includes a training text and a keyword label; wherein the keyword label is a keyword in the training text;

[0063] For each training sample, the training text in the training sample is used as the input of the keyword extraction model to be trained, and the keyword label in the training sample is used as the output of the keyword extraction model to be trained. The keyword extraction model to be trained is trained until the training accuracy reaches the preset accuracy.

[0064] The keyword tags mentioned above are manually marked in the training text. The training accuracy is manually preset, for example, the training accuracy can be that the similarity between the training result and the keyword tag reaches 80%.

[0065] For example, the data table name of the first data information is "2015 AB Department Annual Meeting Statistical List", and the data table name of the second data information is "2015 AB Department Physical Examination Statistical List". Based on the trained keyword extraction model, the keywords extracted from the data table name of the first data information are "2015", "AB Department", "Annual Meeting", and "Statistical List". The keywords extracted from the data table name of the first data information form a first set ["2015", "AB Department", "Annual Meeting", "Statistical List"]. The keywords extracted from the name are "2015", "AB Department", "Physical Examination", and "Statistical List". The keywords extracted from the data table name based on the second data information constitute the second set ["2015", "AB Department", "Physical Examination", "Statistical List"], among which the intersection of the first set and the second set is ["2015", "AB Department", "Statistical List"], and the union of the first set and the second set is ["2015", "AB Department", "Annual Meeting", "Physical Examination", "Statistical List"], then the similarity of the first name is 3 / 5=0.6.

[0066] In the above step S1022, after the first name similarity is calculated, the value of the first name similarity can be directly used to represent the strong correlation between the first data table and the second data table.

[0067] Words are a broad and profound product. There may be multiple words to express the same thing. Different words correspond to different characters, but the semantics they express may be the same or similar. Therefore, the similarity between two data tables can be determined by the data table names, not only by the same keywords, but also by the strings with the same semantics. That is, in step S103, determining the second name similarity includes:

[0068] Step 1031 : Calculate the second name similarity between the data table names corresponding to the first data information and the second data information respectively according to the semantics corresponding to the data table names of the first data information and the data table names of the second data information respectively.

[0069] In the above step 1031, semantics refers to the meaning contained in the data table name.

[0070] In a specific implementation, a corresponding semantic vector is first pre-set for each character, and a first vector corresponding to the data table name of the first data information and a second vector corresponding to the data table name of the second data information are constructed according to the semantic vector corresponding to each character. Then, the first vector and the second vector are input into the trained semantic similarity calculation model to calculate the second name similarity between the data table names corresponding to the first data information and the second data information, respectively.

[0071] The trained semantic similarity calculation model mentioned above is obtained through training with a large amount of training data. The semantic similarity calculation model is a deep learning network model.

[0072] In addition to determining the blood relationship between two data tables by the data table names, the blood relationship between two data tables can also be determined based on the attribute information of the data stored in the data tables, and the similarity between the attribute information can better reflect the blood relationship between the data tables. That is, step S103 includes:

[0073] Step 1032, calculating the first attribute similarity between the data fields corresponding to the first data table and the second data table respectively according to the distance between the character strings of the data fields of the first data table and the data fields of the second data table;

[0074] Step 1033, calculating the second attribute similarity between the data fields corresponding to the first data table and the second data table respectively according to the intersection and union of the field values ​​of the first data table and the field values ​​of the second data table;

[0075] Step 1034: Calculate the third attribute similarity between the field value ranges corresponding to the first data table and the second data table respectively according to the intersection of the field value ranges of the first data table and the field value ranges of the second data table.

[0076] In the above step 1032, each data table may include multiple data fields. For each data field in the first data table, the string edit distance method is used to calculate the edit distance between the data field and each data field in the second data table, and the first target data field in the second data table with the smallest edit distance to the data field is determined. Then, the first number of data fields in the first data table whose edit distances with the first target data field are less than a preset distance is counted, and the first attribute similarity between the data fields corresponding to the first data table and the second data table is determined based on the ratio of the first number obtained by statistics to the number of data fields in the first data table.

[0077] In the above step 1033, for each data field in the first data table, the repetition rate of the field value corresponding to the data field and the field value of each data field in the second data table is calculated, and the second target data field in the second data table with the largest repetition rate of the field value of the data field is determined. Then, the second number of the field values ​​between the data field and the second target data field in the first data table whose repetition rate is greater than the preset repetition rate is counted, and the second attribute similarity between the field values ​​corresponding to the first data table and the second data table is determined based on the ratio of the second number obtained by statistics to the number of data fields in the first data table.

[0078] In the above step 1034, for each data field in the first data table, the intersection ratio of the field value range corresponding to the data field and the field value range of each data field in the second data table is calculated, and the third target data field in the second data table with the field value range of the data field having the largest intersection ratio is determined. Then, the third number of the intersection ratios of the field value ranges between the data field and the second target data field in the first data table that is greater than the preset intersection ratio is counted, and the third attribute similarity between the field value ranges corresponding to the first data table and the second data table is determined based on the ratio of the third number obtained by statistics to the number of data fields in the first data table.

[0079] The intersection ratio refers to the ratio of the intersection of the field value range of the data field in the first data table and the field value range of the data field in the second data table to the field value range of the data field in the first data table.

[0080] The weak correlation degree is determined by the data table names and attribute information corresponding to the first data table and the second data table, respectively. Therefore, step S103 includes:

[0081] Step 1035: Determine the weak association degree between the first data table and the second data table based on the weighted sum of the second name similarity, the first attribute similarity, the second attribute similarity, and the third attribute similarity.

[0082] In the above step 1035, weighted values ​​are pre-set for the second name similarity, the first attribute similarity, the second attribute similarity, and the third attribute similarity, and finally the second name similarity, the first attribute similarity, the second attribute similarity, and the third attribute similarity are weighted and summed to obtain the weak association degree between the first data table and the second data table. The weighted values ​​corresponding to the second name similarity, the first attribute similarity, the second attribute similarity, and the third attribute similarity can be set according to actual conditions.

[0083] After obtaining the strong correlation degree and the weak correlation degree, the blood relationship between the first data to be detected and the second data to be detected can be jointly determined. Step S104 includes:

[0084] Step 1041, determine the blood relationship between the first data to be detected and the second data to be detected based on the sum of the product of the first weighted value and the strong correlation degree and the product of the second weighted value and the weak correlation degree; the first weighted value is greater than the second weighted value.

[0085] In the above step 1041, the strong correlation degree can more intuitively characterize the blood relationship between the first data to be detected and the second data to be detected, so the first weighted value can be set to be greater than the second weighted value. Of course, if the staff pays more attention to the blood relationship between the data, the first weighted value can also be set to be less than the second weighted value according to actual needs.

[0086] After determining the blood relationship between the first data to be detected and the second data to be detected, the dependency relationship between the data can be clarified according to the blood relationship, and then the data can be quickly classified, and then the data characteristics of a type of data can be analyzed based on the data, and a targeted implementation plan can be generated based on the data characteristics. For example, the first data to be detected is a physical examination form for junior high school students, and the second data to be detected is all the data in the database. The above method is used to determine the physical examination statistics form of a certain middle school that has a close blood relationship with the physical examination form for junior high school students, and then based on the data in the physical examination form for junior high school students and the physical examination statistics form for a certain middle school, physical training strategy information is generated for the students of the certain middle school to improve the physical fitness of the students of the certain middle school.

[0087] The present application embodiment provides a device for determining data lineage relationship, such as Figure 3 As shown, including:

[0088] The first determination module 301 is used to determine the first data to be detected and the second data to be detected in the demand according to data lineage, and determine the first data information of the first data table where the first data to be detected is located and the second data information of the second data table where the second data to be detected is located;

[0089] A second determination module 302, configured to determine a strong association degree between the first data table and the second data table according to a first name similarity between a data table name of the first data information and a data table name of the second data information;

[0090] A third determination module 303 is used to determine the weak association degree between the first data table and the second data table according to the second name similarity between the data table name of the first data information and the data table name of the second data information, and according to the attribute similarity between the attribute information of the first data table where the first data information is located and the attribute information of the second data table where the second data information is located;

[0091] The fourth determination module 304 is used to determine the blood relationship between the first to-be-detected data and the second to-be-detected data according to the strong correlation degree and the weak correlation degree.

[0092] Optionally, the second determining module includes:

[0093] A first calculation unit, configured to calculate a first name similarity between data table names corresponding to the first data information and the second data information respectively, based on the same keywords appearing in the data table name of the first data information and the data table name of the second data information;

[0094] The first determining unit is configured to determine a first name similarity between data table names corresponding to the first data information and the second data information respectively as a strong association degree between the first data table and the second data table.

[0095] Optionally, the third determining module includes:

[0096] The second determining unit is used to calculate a second name similarity between the data table names corresponding to the first data information and the second data information respectively according to the semantics corresponding to the data table names of the first data information and the data table names of the second data information respectively.

[0097] Optionally, the third determining module includes:

[0098] A third determining unit, configured to calculate a first attribute similarity between the data fields respectively corresponding to the first data table and the second data table according to a distance between character strings of the data fields of the first data table and the data fields of the second data table;

[0099] a fourth determining unit, configured to calculate a second attribute similarity between data fields corresponding to the first data table and the second data table, respectively, according to an intersection and a union of field values ​​of the first data table and field values ​​of the second data table;

[0100] The fifth determining unit is used to calculate the third attribute similarity between the field value ranges corresponding to the first data table and the second data table respectively according to the intersection of the field value range of the first data table and the field value range of the second data table.

[0101] Optionally, the third determining module includes:

[0102] The first calculation unit is used to determine the weak association degree between the first data table and the second data table based on the weighted sum of the second name similarity, the first attribute similarity, the second attribute similarity and the third attribute similarity.

[0103] Optionally, the fourth determining module includes:

[0104] The second calculation unit is used to determine the blood relationship between the first data to be detected and the second data to be detected based on the sum of the product of the first weighted value and the strong association degree and the product of the second weighted value and the weak association degree; the first weighted value is greater than the second weighted value.

[0105] Corresponds to Figure 1 The method for determining data lineage relationship in the present application embodiment also provides a computer device 400, such as Figure 4 As shown, the device includes a memory 401, a processor 402, and a computer program stored in the memory 401 and executable on the processor 402, wherein the processor 402 implements the method for determining data lineage when executing the computer program.

[0106] Specifically, the above-mentioned memory 401 and processor 402 can be general-purpose memory and processor, which are not specifically limited here. When the processor 402 runs the computer program stored in the memory 401, it can execute the above-mentioned method for determining the blood relationship of data, thereby solving the problem of inaccurate determination of blood relationship between data in the prior art.

[0107] Corresponds to Figure 1 In the method for determining data lineage relationships, an embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the method for determining data lineage relationships are executed.

[0108] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is run, the above-mentioned method for determining the blood relationship of data can be executed, which solves the problem of inaccurate determination of the blood relationship between data in the prior art. The data table name of some data tables in the present application may be just a number, or composed of random strings. Therefore, the blood relationship between different data cannot be accurately determined by the similarity between the data table names alone. Therefore, when the present application determines the blood relationship between different data, in addition to using the data table name of the data table to which the data belongs, it also uses the data stored in the two data tables to determine it. That is, when the data table names of different data tables are quite different, the blood relationship between different data tables can also be accurately determined based on the similarity of the data stored in different data tables.

[0109] In the embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0112] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0113] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0114] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for determining data lineage, characterized in that: include: Determine the first data to be detected and the second data to be detected in the demand according to the data lineage, determine the first data information of the first data table where the first data to be detected is located and the second data information of the second data table where the second data to be detected is located; determining a strong association degree between the first data table and the second data table according to a first name similarity of the same keyword between the data table name of the first data information and the data table name of the second data information; Determine the weak association degree between the first data table and the second data table according to the second name similarity determined by the semantic correspondence between the data table name of the first data information and the data table name of the second data information, and according to the attribute similarity between the attribute information of the first data table where the first data information is located and the attribute information of the second data table where the second data information is located; Determining the blood relationship between the first to-be-detected data and the second to-be-detected data according to the strong correlation degree and the weak correlation degree; The attribute similarity between the attribute information of the first data table where the first data information is located and the attribute information of the second data table where the second data information is located is calculated by the following steps: Calculating the first attribute similarity between the data fields corresponding to the first data table and the second data table respectively according to the distance between the character strings of the data fields of the first data table and the data fields of the second data table; Calculate the second attribute similarity between the data fields corresponding to the first data table and the second data table respectively according to the intersection and union of the field values ​​of the first data table and the field values ​​of the second data table; Calculating the third attribute similarity between the field value ranges corresponding to the first data table and the second data table respectively according to the intersection of the field value range of the first data table and the field value range of the second data table; Determining the weak association degree between the first data table and the second data table according to the second name similarity determined by the semantics corresponding to the data table name of the first data information and the data table name of the second data information, and according to the attribute similarity between the attribute information of the first data table where the first data information is located and the attribute information of the second data table where the second data information is located, including: Determining a weak association degree between the first data table and the second data table based on a weighted sum of the second name similarity, the first attribute similarity, the second attribute similarity, and the third attribute similarity; Determining the strong association degree between the first data table and the second data table according to the first name similarity of the same keyword between the data table name of the first data information and the data table name of the second data information includes: Calculating first name similarity between data table names corresponding to the first data information and the second data information, respectively, based on identical keywords appearing in the data table name of the first data information and the data table name of the second data information; Determine a first name similarity between data table names corresponding to the first data information and the second data information respectively as a strong association degree between the first data table and the second data table; Determining the blood relationship between the first to-be-detected data and the second to-be-detected data according to the strong correlation degree and the weak correlation degree includes: The blood relationship between the first data to be detected and the second data to be detected is determined based on the sum of the product of the first weighted value and the strong association degree and the product of the second weighted value and the weak association degree; the first weighted value is greater than the second weighted value.

2. A device for determining data lineage, characterized in that: include: A first determination module, configured to determine the first data to be detected and the second data to be detected in the data lineage determination requirement, and determine the first data information of the first data table where the first data to be detected is located and the second data information of the second data table where the second data to be detected is located; A second determination module, configured to determine a strong association degree between the first data table and the second data table according to a first name similarity of the same keyword between the data table name of the first data information and the data table name of the second data information; a third determination module, configured to determine a weak association degree between the first data table and the second data table according to a second name similarity between a data table name of the first data information and a data table name of the second data information, and according to an attribute similarity between attribute information of the first data table where the first data information is located and attribute information of the second data table where the second data information is located; a fourth determination module, configured to determine the blood relationship between the first to-be-detected data and the second to-be-detected data according to the strong association degree and the weak association degree; The third determining module comprises: A third determining unit, configured to calculate a first attribute similarity between the data fields corresponding to the first data table and the second data table, respectively, according to a distance between character strings of the data fields of the first data table and the data fields of the second data table; a fourth determining unit, configured to calculate a second attribute similarity between data fields corresponding to the first data table and the second data table respectively according to an intersection and a union of field values ​​of the first data table and field values ​​of the second data table; a fifth determining unit, configured to calculate a third attribute similarity between the field value ranges corresponding to the first data table and the second data table respectively according to an intersection of the field value ranges of the first data table and the field value ranges of the second data table; The third determining module comprises: a first calculation unit, configured to determine a weak association degree between the first data table and the second data table based on a weighted sum of the second name similarity, the first attribute similarity, the second attribute similarity, and the third attribute similarity; The second determining module includes: A first calculation unit, configured to calculate a first name similarity between data table names corresponding to the first data information and the second data information respectively, based on the same keywords appearing in the data table name of the first data information and the data table name of the second data information; A first determining unit, configured to determine a first name similarity between data table names corresponding to the first data information and the second data information respectively as a strong association degree between the first data table and the second data table; The fourth determining module includes: The second calculation unit is used to determine the blood relationship between the first data to be detected and the second data to be detected based on the sum of the product of the first weighted value and the strong association degree and the product of the second weighted value and the weak association degree; the first weighted value is greater than the second weighted value.

3. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method described in claim 1 are implemented.

4. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method described in claim 1 are performed.