Intelligent data governance cleaning method and device based on knowledge graph

Abstract

The invention discloses an intelligent data governance cleaning method and device based on a knowledge graph, and the method comprises the steps: constructing a data cleaning graph database which comprises a plurality of standard data tables and cleaning mapping information corresponding to each standard field in each standard data table, and combining the semantic recognition and table name recognition of a to-be-cleaned table; determining cleaning mapping information corresponding to each effective field in the to-be-cleaned table; then, a cleaning script can be automatically generated based on the determined cleaning mapping information; therefore, the cleaning script of the metadata and the data items in the production library can be automatically generated under the condition that the cleaning script does not need to be written manually, the operation process of data management engineers on data item standardization is effectively simplified, the difficulty of cleaning operation and the labor cost are reduced, and the efficiency and the accuracy of data cleaning are improved; therefore, the method is very suitable for large-scale application and popularization.

Description

Technical Field

[0001] This invention belongs to the field of data cleaning technology in data governance, specifically relating to an intelligent data governance and cleaning method and apparatus based on knowledge graphs. Background Technology

[0002] In recent years, data governance has become an important means and tool for mining the value of data. At the same time, with the development of big data platforms and the Internet industry, enterprises, government agencies and other departments have integrated a large amount of data and are actively optimizing and upgrading their existing big data collection systems to break down data silos, expand collection channels, and extend their sensing reach. This allows for the comprehensive and high-quality collection and aggregation of basic data resources such as people, places, things, events, organizations, and time and space, as well as dynamic data resources such as trajectories, behaviors, and content. Therefore, in order to fully realize the value of data, it is necessary to build a standard system for existing data and conduct data governance based on this standard.

[0003] Data cleaning is an indispensable part of data governance. It involves delving into the data tables to identify existing data quality issues. This is achieved through manual data quality surveys to uncover data problems. Then, data standardization tools are used to perform a series of operations, such as data meta-alignment, data deduplication, data field value standardization, and data dictionary conversion. This transforms the raw data into high-quality data that is described by data standard meta-elements and can provide standard data services.

[0004] Currently, data cleaning usually involves manually writing cleaning scripts such as ETL or SQL, which has the following shortcomings: (1) Manually writing cleaning scripts requires a high level of experience from governance engineers, which increases labor costs and is prone to errors, thereby reducing the reliability of data cleaning; (2) Manually writing cleaning scripts takes a long time, which reduces cleaning efficiency. Therefore, based on the aforementioned shortcomings, how to provide an intelligent data governance and cleaning method based on knowledge graphs that can automatically generate cleaning scripts has become an urgent problem to be solved. Summary of the Invention

[0005] The purpose of this invention is to provide an intelligent data governance and cleaning method and apparatus based on knowledge graphs, in order to solve the problems of high labor costs, low cleaning reliability and low efficiency in the existing technology.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: Firstly, a knowledge graph-based intelligent data governance and cleaning method is provided, including: A data cleaning graph database is constructed, wherein the data cleaning graph database stores several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table, and the cleaning mapping information of any standard field includes the cleaning method corresponding to the standard field, the standard data element corresponding to the standard field, the connection relationship of the standard data elements, and the corresponding identification code set. Obtain the table to be cleaned, and determine the standard table name corresponding to the table to be cleaned based on several standard data tables; Semantic recognition is performed on each field to be cleaned in the table to be cleaned to obtain the true semantics of each field to be cleaned. Then, the semantic information of the table to be cleaned is formed by using the standard table name corresponding to the table to be cleaned and the true semantics of each field to be cleaned. Based on the semantic information, the cleaning mapping information corresponding to each valid field in the table to be cleaned is matched from the data cleaning graph database; Based on the cleaning mapping information of each valid field, a cleaning script is generated for the table to be cleaned, so that the data cleaning of the table to be cleaned can be completed by running the cleaning script.

[0007] Based on the above-disclosed content, this invention first constructs a data cleaning graph database. This database contains several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table. The cleaning mapping information for any standard field includes the cleaning method corresponding to that standard field, the standard data element corresponding to that standard field, the connection relationship between the standard data elements, and the corresponding identifier code set. Thus, this invention effectively constructs the association relationship between fields and data elements in different data tables, as well as the mapping relationship between fields and cleaning methods. Based on this, upon receiving a table to be cleaned, the cleaning method can be directly matched according to this graph database, i.e., based on the aforementioned constructed graph database... First, a standard table is used to determine the standard table name of the table to be cleaned. Then, semantic recognition is performed on the table to be cleaned to obtain the true semantics of each field to be cleaned. Using the true semantics of each field to be cleaned and the previously determined standard table name, semantic information of the table to be cleaned is generated. Next, based on this semantic information, the cleaning mapping information corresponding to each valid field in the table to be cleaned is matched from the data cleaning graph database (i.e., the corresponding cleaning method, standard data element, the connection relationship of standard data element and the identification code set are matched). Finally, the cleaning mapping information corresponding to each valid field can be used to automatically generate a cleaning script for the table to be cleaned, and the data cleaning of the table to be cleaned is completed based on the cleaning script.

[0008] Through the above design, this invention constructs a data cleaning graph database containing several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table. By combining semantic recognition and table name recognition of the table to be cleaned, the cleaning mapping information corresponding to each valid field in the table to be cleaned is determined. Then, based on the determined cleaning mapping information, a cleaning script can be automatically generated. Thus, this invention can automatically generate cleaning scripts for metadata and data items in the production database without requiring manual writing of cleaning scripts. This effectively simplifies the data governance engineer's standardization process for data items, reduces the difficulty and labor costs of cleaning operations, and improves the efficiency and accuracy of data cleaning. Therefore, this invention is highly suitable for large-scale application and promotion.

[0009] In one possible design, a data cleaning graph database is constructed, including: Obtain the table names and corresponding table structures of several standard tables. The table structure includes field names and field name comments, and any table name is in Chinese. Based on the table names of each departmental standard table, the English names of each departmental standard table are determined. Data splitting is performed on the field names in the table structure of each departmental standard table to obtain the standard data element corresponding to each field in each departmental standard table; Based on the field names in the table structure of each departmental standard table, determine the connection relationship and identification code set of the standard data elements corresponding to each field in each departmental standard table; Obtain the cleaning and verification methods corresponding to each standard data element, and determine the cleaning and verification methods for each field in each departmental standard table based on the cleaning and verification methods corresponding to each standard data element. Based on the table name, English name, field name and field name comments of each departmental standard table, each standard data table is generated. Then, by using the cleaning and verification methods of each field in each departmental standard table, as well as the standard data elements corresponding to each field, the connection relationship of standard data elements and the identification code set, the cleaning mapping information of each standard field in each standard data table is determined. Each standard data table and the cleaning mapping information of each standard field in each standard data table are stored in a graph database to obtain the data cleaning graph database.

[0010] In one possible design, based on several standard data tables, the standard table name corresponding to the table to be cleaned is determined, including: Obtain the standard table names of all standard data tables and the original table names of the tables to be cleaned, and preprocess the original table names to obtain the preprocessed original table names. Calculate the edit distance between the preprocessed original table name and each standard table name; Determine if there exists any edit distance equal to 1; If not, then the preprocessed original table names are segmented into words to obtain a word set; Calculate the semantic similarity between the word set and each standard table name; The total similarity between the original table name and each standard table name is calculated based on the edit distance between the preprocessed original table name and each standard table name, as well as the semantic similarity between the word set and each standard table name. From all standard table names, select those with a total similarity greater than the similarity threshold, and use the standard table name with the highest total similarity among the selected standard table names as the standard table name of the table to be cleaned.

[0011] In one possible design, the cleaning mapping information for any field also includes: the validation method corresponding to that field; Specifically, based on the semantic information, the cleaning mapping information corresponding to each valid field in the table to be cleaned is matched from the data cleaning graph database, including: Based on the true semantics of each field to be cleaned in the semantic information, the standard field corresponding to each field to be cleaned is determined from the data cleaning graph database, and the verification method of the standard field corresponding to each field to be cleaned is used as the verification method of each field to be cleaned. Based on the validation method of each field to be cleaned, the table to be cleaned is subjected to content probing to obtain content probing results, wherein the content probing results include the field null value rate; Based on the field null value rate in the content exploration results, select valid fields from each field to be cleaned; Based on the standard table name and each valid field corresponding to the table to be cleaned, the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database, or the true semantics of each valid field is determined according to the semantic information, and the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database according to the true semantics of each valid field.

[0012] In one possible design, based on the standard table name and each valid field corresponding to the table to be cleaned, the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database, including: Using the standard fields in each standard data table of the data cleaning graph database, several table sets are generated; Based on the valid fields and several table sets, a valid field distribution matrix is ​​generated. The size of the valid field distribution matrix is ​​n×m, where n is the total number of valid fields and m is the number of table sets. The elements in the valid field distribution matrix... This indicates the number of times the nth valid field appears in the mth table set; Based on the effective field distribution matrix, the total number of effective fields appearing in each table set is determined, and the proportion of effective fields in each table set is calculated using the total number of effective fields appearing in each table set. Sort the table sets in descending order of proportion, and take the standard data table corresponding to the top three table sets as the target table. Determine whether the target table contains a standard table name corresponding to the table to be cleaned; If so, the cleaning mapping information of each valid field is determined based on the cleaning mapping information of each standard field in the standard data table corresponding to the standard table name of the table to be cleaned. Otherwise, the true semantics of each valid field are determined based on the semantic information, and the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database based on the true semantics of each valid field.

[0013] In one possible design, based on the true semantics of each valid field, the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database, including: Based on the true semantics of each valid field, standard fields with the same semantics as each valid field are matched from several standard data tables and used as target fields. The cleaning mapping information corresponding to each target field is used as the cleaning mapping information for each valid field.

[0014] In one possible design, the content exploration results also include: field value range distribution, data element exploration results, data type and format exploration results, and business exploration results; The cleaning script for the table to be cleaned is generated based on the cleaning mapping information of each valid field, including: Based on the field value range distribution, data element exploration results, data type and format exploration results, and business exploration results in the content exploration results, the actual cleaned fields in each valid field are determined. Based on the cleaning method in the cleaning mapping information of each actual cleaning field, the standard data elements corresponding to each actual cleaning field, the connection relationship of the standard data elements, and the corresponding identification code set, the actual cleaning fields in the table to be cleaned are transformed into a dictionary to obtain a transformation table. The cleaning script is generated using the transformation table.

[0015] Secondly, a knowledge graph-based intelligent data governance and cleaning device is provided, comprising: The graph database construction unit is used to construct a data cleaning graph database. The data cleaning graph database stores several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table. The cleaning mapping information of any standard field includes the cleaning method corresponding to the standard field, the standard data element corresponding to the standard field, the connection relationship of the standard data elements, and the corresponding identification code set. The table name identification unit is used to obtain the table to be cleaned and, based on several standard data tables, determine the standard table name corresponding to the table to be cleaned. The semantic recognition unit is used to perform semantic recognition on each field to be cleaned in the table to be cleaned, obtain the true semantics of each field to be cleaned, and use the standard table name corresponding to the table to be cleaned and the true semantics of each field to be cleaned to form the semantic information of the table to be cleaned. The data cleaning unit is used to match the cleaning mapping information corresponding to each valid field in the table to be cleaned from the data cleaning graph database based on the semantic information. The data cleaning unit is also used to generate a cleaning script for the table to be cleaned based on the cleaning mapping information of each valid field, so as to complete the data cleaning of the table to be cleaned by running the cleaning script.

[0016] Thirdly, another knowledge graph-based intelligent data governance and cleaning device is provided. Taking the device as an electronic device as an example, it includes a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the knowledge graph-based intelligent data governance and cleaning method as described in the first aspect or any possible design in the first aspect.

[0017] Fourthly, a storage medium is provided, on which instructions are stored, which, when executed on a computer, perform the knowledge graph-based intelligent data governance and cleaning method as described in the first aspect or any possible design of the first aspect.

[0018] Fifthly, a computer program product containing instructions is provided, which, when executed on a computer, causes the computer to perform the knowledge graph-based intelligent data governance and cleaning method as described in the first aspect or any possible design of the first aspect.

[0019] Beneficial effects: (1) This invention constructs a data cleaning graph database containing several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table. Combined with semantic recognition and table name recognition of the table to be cleaned, the cleaning mapping information corresponding to each valid field in the table to be cleaned is determined. Then, based on the determined cleaning mapping information, the cleaning script can be automatically generated. Thus, this invention can automatically generate cleaning scripts for metadata and data items in the production database without the need to manually write cleaning scripts. This effectively simplifies the data governance engineer's operation process for standardizing data items, reduces the difficulty and labor cost of cleaning operations, and improves the efficiency and accuracy of data cleaning. Therefore, this invention is very suitable for large-scale application and promotion. Attached Figure Description

[0020] Figure 1 A schematic diagram of the steps of the knowledge graph-based intelligent data governance and cleaning method provided in the embodiments of the present invention; Figure 2 A structural diagram of the knowledge graph-based intelligent data governance and cleaning device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the present invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the accompanying drawings is only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of helping to understand the present invention, but does not constitute a limitation of the present invention.

[0022] It should be understood that although the terms first, second, etc., may be used herein to describe various units, these units should not be limited by these terms. These terms are only used to distinguish one unit from another. For example, a first unit may be referred to as a second unit, and similarly, a second unit may be referred to as a first unit, without departing from the scope of the exemplary embodiments of the invention.

[0023] It should be understood that the term "and / or" that may appear in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" that may appear in this document describes another relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " that may appear in this document generally indicates that the related objects before and after it are in an "or" relationship.

[0024] Example: See Figure 1 As shown, the knowledge graph-based intelligent data governance and cleaning method provided in this embodiment constructs a data cleaning graph database containing several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table. Combined with semantic recognition and table name recognition of the table to be cleaned, the cleaning mapping information corresponding to each valid field in the table to be cleaned is determined. Then, based on the determined cleaning mapping information, a cleaning script can be automatically generated. Thus, this method can automatically generate cleaning scripts for metadata and data items in the production database without requiring manual writing of cleaning scripts. This effectively simplifies the data governance engineer's standardization process for data items, reduces the difficulty and labor costs of cleaning operations, and improves the efficiency and accuracy of data cleaning. Therefore, this method is very suitable for large-scale application and promotion. For example, this method can be run on the data cleaning end, but is not limited to. Optionally, the data cleaning end can be, but is not limited to, a personal computer (PC) or a server. It is understood that the aforementioned execution subject does not constitute a limitation on the embodiments of this application. Accordingly, the operation steps of this method can be, but are not limited to, the steps S1 to S5 below.

[0025] S1. Construct a data cleaning graph database, wherein the data cleaning graph database stores several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table, and the cleaning mapping information of any standard field includes the cleaning method corresponding to the standard field, the standard data element corresponding to the standard field, the connection relationship of the standard data elements, and the corresponding identification code set; in this embodiment, the mapping relationship between standard data elements and conversion methods (i.e. cleaning methods) and verification methods is constructed based on the table names and table structures of the national standard tables collected under different data governance systems, thereby forming a data governance knowledge base; finally, the aforementioned data is stored in the graph database to construct the data cleaning graph database.

[0026] Optionally, the aforementioned data cleaning graph database construction process can be, but is not limited to, the steps S11 to S17 below.

[0027] S11. Obtain the table names and corresponding table structures of several standard tables. The table structure includes field names and field name comments, and any table name can be in Chinese.

[0028] In this embodiment, the departmental standard table is a collection of different tables in advance according to the data governance scenario. For example, in the regional personnel data governance scenario, the departmental standard table may include a population table, and its table name and table structure are shown in Table 1 below.

[0029] Table 1 shows the table names and structure corresponding to the population table.

[0030] Table 1

[0031] Of course, the above is only an illustration, and the names and structures of the other standard tables will not be listed one by one.

[0032] After obtaining the table names and corresponding table structures of several standard tables, the graph database can be constructed, as shown in steps S12 to S17 below.

[0033] S12. Based on the table names of each departmental standard table, determine the English names of each departmental standard table. In this embodiment, different table names are pre-configured with corresponding English names. Therefore, in actual use, the matching can be performed according to the table names of each departmental standard table. For example, the English name corresponding to the table name "Actual Population Information" is "ysk_ry_syrkxx_dd". Of course, the above example is only illustrative and this embodiment is not limited to this. The English names of each departmental standard table are determined to facilitate the reference of the tables when generating the cleaning script.

[0034] After determining the English names of the standard tables for each department, field segmentation can be performed to obtain the standard data elements corresponding to each field in the table; the field segmentation process is shown in step S13 below.

[0035] S13. Perform data segmentation on the field names in the table structure of each departmental standard table to obtain the standard data elements corresponding to each field in each departmental standard table. In specific implementation, for example, but not limited to, segmenting the field names in each departmental standard table according to preset symbols to obtain the standard data elements corresponding to each field; where the preset symbol is "_" for example; for example, the field name "HJDZ_XZQHDM", "XZQHDM" in the field name is a data element, which is the basic unit constituting the field (data item), while "HJDZ" is a qualifier that modifies the data element "XZQHDM"; in this way, after completing the segmentation of the field names based on the aforementioned preset symbols and obtaining the standard data elements corresponding to each field, the connection relationship of the data elements and the identification code set can be determined, and the process is shown in step S14 below.

[0036] S14. Based on the field names in the table structure of each departmental standard table, determine the connection relationship and identification code set of the standard data elements corresponding to each field in each departmental standard table. In specific implementation, the field names of the code categories are expanded to form data elements with connection relationships. That is, the connection data elements and identification code sets corresponding to each field name are pre-stored. When using them, they can be matched according to the field names. For example, "XBDM (gender code)" and "XB (gender)" have a connection relationship, and the identification code set (that is, the code identifier, used for code identification) corresponding to XBDM (gender code) is XBDM. In this way, after determining the connection relationship and identification code set of the standard data elements corresponding to each field based on the above method, the cleaning method and the verification method can be matched. The process is shown in step S15 below.

[0037] S15. Obtain the cleaning and verification methods corresponding to each standard data element, and determine the cleaning and verification methods for each field in each departmental standard table based on the cleaning and verification methods corresponding to each standard data element.

[0038] In practical applications, this embodiment pre-stores cleaning and verification methods corresponding to different data elements. Therefore, in actual use, matching can be performed according to each standard data element. After obtaining the cleaning and verification methods corresponding to each standard data element, the cleaning and verification methods of the field can be determined. Among them, the verification method of any field is used for content verification, and the process is described in detail below.

[0039] Therefore, after determining the cleaning and verification methods for each field, the standard data table and the cleaning mapping information for each standard field can be generated by combining the aforementioned Chinese table name, English name, field name, field name comment, standard data element, the connection relationship of standard data element, and the corresponding identification code set. The process is shown in step S16 below.

[0040] S16. Based on the table name, English name, field name, and field name comments of each departmental standard table, generate each standard data table, and use the cleaning and verification methods of each field in each departmental standard table, as well as the standard data elements corresponding to each field, the connection relationship of standard data elements, and the identification code set, determine the cleaning mapping information of each standard field in each standard data table; in this embodiment, one departmental standard table corresponds to one standard data table, and the cleaning mapping information of the fields in the departmental standard table is the cleaning mapping information of each standard field in its corresponding data standard table.

[0041] Thus, by using the aforementioned standard data table and the cleaning mapping information of the standard fields in the table as a data governance knowledge base, and storing the data in a graph database, a data cleaning graph database can be generated, as shown in step S17 below.

[0042] S17. Store the cleaning mapping information of each standard data table and each standard field in each standard data table into a graph database to obtain the data cleaning graph database after storage. In this embodiment, the standard table name and standard field of each standard data table can also be converted into an embedding vector and then stored in the Faiss vector library to prepare for the later standard table recommendation.

[0043] Optionally, taking the aforementioned population table as an example, its corresponding standard data table is shown in Table 2 below.

[0044] Table 2 is the standard data table corresponding to the population table.

[0045] Table 2

[0046] Meanwhile, the cleaning mapping information corresponding to each standard field can be stored in tabular form, but is not limited to tabular form. For example, standard fields and their corresponding cleaning and verification methods can be stored in tabular form, and the connection relationship of data elements can also be stored in tabular form.

[0047] Optionally, the cleaning and verification methods corresponding to different standard fields, the association table of standard data elements, and the identification code set table of standard data elements are shown in Tables 3, 4, and 5 below, respectively.

[0048] Table 3 shows the correspondence between standard fields and cleaning and verification methods.

[0049] Table 3

[0050] Table 4 shows the connection relationships between standard data elements.

[0051] Table 4

[0052] In Table 4 above, the source field is a standard data element, while the target field is the contiguous data element corresponding to that standard data element.

[0053] Table 5 is a set of identification codes for standard data elements.

[0054] Table 5

[0055] Of course, the examples mentioned above are merely illustrative, and this embodiment is not limited thereto.

[0056] Thus, after constructing the data cleaning graph database based on the aforementioned steps S11 to S17, cleaning scripts can be automatically generated based on this data cleaning graph database, as shown in the following steps S2 to S5.

[0057] S2. Obtain the table to be cleaned, and determine the standard table name corresponding to the table to be cleaned based on several standard data tables; in specific applications, this embodiment automatically identifies the true meaning of the table name of the table to be cleaned through semantics, thereby matching its corresponding standard table name (e.g., the table name "Total Table of Actual Population Information" is identified as "Actual Population Information"); wherein, the identification process of the standard table name corresponding to the table to be cleaned may be, but is not limited to, as shown in steps S21 to S27 below.

[0058] S21. Obtain the standard table names of all standard data tables and the original table names of the tables to be cleaned, and preprocess the original table names to obtain preprocessed original table names. In specific implementation, the preprocessing of the original table names may include, but is not limited to, removing leading and trailing spaces, special characters (such as the symbol "", comma and symbol "-", etc.), and retaining letters, numbers and Chinese characters or the symbol "_". After the aforementioned preprocessing, the edit distance between table names can be calculated, as shown in step S22 below.

[0059] S22. Calculate the edit distance between the preprocessed original table name and each standard table name; in this embodiment, for any standard table name, for example, but not limited to, counting the number of overlapping characters between the original table name and the standard table name, and then normalizing the number of overlapping characters to obtain the edit distance between the original table name and the standard table name.

[0060] If the original table name is "Total Real Population Information Table" and the standard table name is "Real Population Information", the number of overlapping characters in the two is 6. Normalization is to filter out the maximum length of the original table name and the standard table name, and then divide the number of overlapping characters by the maximum value to obtain the edit distance between the two. Thus, the edit distance between the aforementioned "Total Real Population Information Table" and "Real Population Information" is: 6 / max(6,8) = 6 / 8 = 0.75.

[0061] Therefore, based on the aforementioned method, after calculating the edit distance between the preprocessed original table name and each standard table name, it is necessary to determine whether there is any edit distance equal to 1, so that different methods can be selected to determine the standard table name of the table to be cleaned based on the judgment result; the judgment process is as shown in step S23 below.

[0062] S23. Determine if there exists any edit distance equal to 1; in specific implementation, if there exists any edit distance of 1, then the standard table name corresponding to the edit distance of 1 is used as the standard table name of the table to be cleaned; otherwise, it is necessary to combine semantic similarity to determine the standard table name of the table to be cleaned, and the process is as shown in steps S24 to S27 below.

[0063] S24. If not, then perform word segmentation on the preprocessed original table names to obtain a word set.

[0064] After word segmentation of the preprocessed original table names is completed, semantic similarity can be calculated, as shown in step S25 below.

[0065] S25. Calculate the semantic similarity between the word set and each standard table name; In this embodiment, for any standard table name, the cosine similarity (i.e., converted to an embedding vector) between the word vector of each word in the word set and the text vector of the standard table name is calculated; Then, the cosine similarity between all words and the standard table name is summed to obtain the semantic similarity between the word set and the standard table name; Thus, after calculating the semantic similarity between the word set and each standard table name based on the aforementioned method, the total similarity between the original table name and each standard table name can be calculated by combining the aforementioned edit distance, as shown in step S26 below.

[0066] S26. Calculate the total similarity between the original table name and each standard table name based on the edit distance between the preprocessed original table name and each standard table name, and the semantic similarity between the word set and each standard table name. In this embodiment, for any standard table name, the total similarity between the original table name and that standard table name can be obtained by weighted summation of the edit distance between the preprocessed original table name and that standard table name, and the semantic similarity between the word set and that standard table name. Optionally, for example, the weight of semantic similarity is 0.6, and the weight of edit distance is 0.4.

[0067] Thus, after calculating the total similarity between the original table name and each standard table name, the standard table name of the table to be cleaned can be determined based on the total similarity, as shown in step S27 below.

[0068] S27. From all standard table names, filter out the standard table names with a total similarity greater than the similarity threshold, and use the standard table name with the highest total similarity among the filtered standard table names as the standard table name of the table to be cleaned; in this embodiment, for example, the similarity threshold can be set to 0.6, thus filtering out standard table names with a total similarity greater than 0.6, and then selecting the table name with the highest total similarity among the filtered standard table names as the standard table name of the table to be cleaned.

[0069] Therefore, based on the aforementioned steps S21 to S27, after determining the standard table name of the table to be cleaned, semantic recognition of the fields in the table to be cleaned can be performed, as shown in step S3 below.

[0070] S3. Perform semantic recognition on each field to be cleaned in the table to be cleaned to obtain the true semantic meaning of each field. Then, use the standard table name corresponding to the table to be cleaned and the true semantic meaning of each field to be cleaned to form the semantic information of the table to be cleaned. In specific implementation, the true meaning of each field to be cleaned in the table to be cleaned is automatically recognized through semantic recognition (e.g., the field "name" is recognized as "name", and "ID number" is recognized as "citizen identity number"). The cosine similarity between each field to be cleaned and the standard fields in each standard data table is also calculated. Then, standard fields with a cosine similarity greater than or equal to 0.8 are selected. Among the selected standard fields, the standard field with the highest cosine similarity is selected as the true semantic meaning of the field to be cleaned.

[0071] After completing the semantic recognition of each field to be cleaned in the table to be cleaned, the semantic information of the table to be cleaned can be formed by combining the standard table name of the table to be cleaned. Then, based on the semantic information and with the help of the aforementioned data cleaning graph database, the cleaning mapping information of each valid field in the table to be cleaned can be matched. The matching process of the cleaning mapping information can be, but is not limited to, the following step S4.

[0072] S4. Based on the semantic information, the cleaning mapping information corresponding to each valid field in the table to be cleaned is matched from the data cleaning graph database. In specific implementation, as mentioned above, the cleaning mapping information also includes a verification method. Therefore, in this embodiment, the corresponding verification method is first matched based on the semantic information of each field to be cleaned. Then, the matched verification method is used to perform content probing on the table to be cleaned, thereby obtaining the content probing result. Next, based on the content verification result, the valid fields in the table to be cleaned are filtered out. Finally, the cleaning mapping information of each valid field is determined based on the data cleaning graph database. The aforementioned process can be, but is not limited to, the steps S41 to S44 below.

[0073] S41. Based on the true semantics of each field to be cleaned in the semantic information, determine the standard field corresponding to each field to be cleaned from the data cleaning graph database, and use the verification method of the standard field corresponding to each field to be cleaned as the verification method of each field to be cleaned; in this embodiment, assuming that the true semantics of the field to be cleaned, name, is a name, then the verification method corresponding to name is determined from the data cleaning graph database, and then the verification method corresponding to name is used as the verification method of name; of course, the above example is only an example, and this embodiment is not limited to this.

[0074] After obtaining the validation method for each field to be cleaned, the content of the table to be cleaned can be explored based on this. The process can be, but is not limited to, the steps shown in S42 below.

[0075] S42. Based on the validation methods of each field to be cleaned, the content of the table to be cleaned is explored to obtain the content exploration results, wherein the content exploration results include the field null value rate; in this embodiment, by exploring the content of each field in the table to be cleaned through the validation methods corresponding to each field to be cleaned, the meaning and statistical distribution of each field can be identified, thereby facilitating a better understanding of the data. The purpose of the content exploration is to provide guidance for subsequent data cleaning.

[0076] Examples of content exploration can include, but are not limited to: null value rate exploration (referring to the exploration of whether the content of each field is empty, null value rate = total number of rows with null values ​​in each field / total number of rows in the table) × 100%), value range distribution exploration, data element exploration, type and format exploration, and business exploration; among them, value range distribution exploration includes dictionary information exploration (i.e., obtaining dictionary table information of data items, such as the dictionary table of gender is "0-unknown gender; 1-male; 2-female; 9-unspecified gender") and value range exploration (exploring the value range based on the field content to determine the maximum and minimum values, such as the value range of Earth longitude is "-180 to 180 degrees", and the value range of current address_administrative division code, except for the specified region, its first digit is between 1 and 6).

[0077] Furthermore, data element exploration refers to determining the actual semantics of the fields to be cleaned based on their names and contents, and mapping them to standard data elements. This can be achieved by combining semantic recognition with manual analysis to explore the associations of data elements in the table, recommending corresponding standard data elements and qualifiers for the source table fields, thereby confirming the corresponding standard fields (data items).

[0078] Meanwhile, type and format probing involves probing the field type and data format, such as determining whether the data type of the date field is YYYYMMDDhh24mmss; and whether the length of the citizen identification number is 18 digits. Business probing involves probing the field logic of the table data, such as whether the online time is <= the offline time, and probing for duplicate records in the table data, such as whether there are two identical records in the actual population.

[0079] Thus, the content exploration results obtained through the aforementioned content exploration can also include: field value range distribution, data element exploration results, data type and format exploration results, and business exploration results. Based on this, it is possible to determine whether there are problems with the content corresponding to the field to be cleaned, such as value range problems, type and format problems, business logic problems, or duplicate data. Therefore, data cleaning can be performed on the problematic fields to be cleaned, that is, to generate corresponding cleaning scripts, the process of which is described below.

[0080] After the content exploration is completed, the effective fields can be filtered based on this, as shown in step S43 below.

[0081] S43. Based on the field null value rate in the content exploration results, select valid fields from each field to be cleaned. In specific implementation, fields with a null value rate lower than a preset threshold are considered valid fields. After selecting valid fields, the true semantics of the valid fields can be combined with the standard fields in the standard data table for similarity matching. If the similarity match is less than a set value, a manual verification prompt is output. Otherwise, the data cleaning graph database constructed above can be used directly to match the cleaning mapping information of each valid field. The process is shown in step S44 below.

[0082] S44. Based on the standard table name and each valid field corresponding to the table to be cleaned, the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database, or the true semantics of each valid field are determined according to the semantic information, and the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database according to the true semantics of each valid field. In specific applications, this embodiment first searches for standard data tables in the data cleaning graph database based on the standard table name to determine whether the standard table name corresponding to the table to be cleaned can be retrieved. If it can be retrieved, the cleaning mapping information can be matched directly using the standard table name; if it cannot be retrieved, the cleaning mapping information needs to be matched using the true semantics of each valid field.

[0083] Optionally, the matching process of cleaning mapping information described above can, but is not limited to, adopt the steps S44a to S44f shown below.

[0084] S44a. Using the standard fields in each standard data table of the data cleaning graph database, generate several table sets; in this embodiment, it is equivalent to using the standard fields in each standard data table to form the table sets corresponding to each standard data table, and a table set is considered as a document.

[0085] After obtaining several sets of tables, the effective field distribution can be statistically analyzed, as shown in step S44b below.

[0086] S44b. Based on each valid field and several table sets, generate a valid field distribution matrix, wherein the size of the valid field distribution matrix is ​​n×m, where n is the total number of valid fields, m is the number of table sets, and the elements in the valid field distribution matrix... This indicates the number of times the nth valid field appears in the mth table set.

[0087] In this embodiment, for any table set 1, firstly, if the i-th valid field appears in any table set 1, then the number of times the i-th valid field appears in any table set 1 is counted. Record it as 1, otherwise record it as 0; then, increment i by 1 and redefine it. The value of i is used until i equals n, thus obtaining the number of times each valid field appears in any table set 1.

[0088] Thus, the effective field distribution matrix can be represented as: ; In the formula, Let m be the set of tables. This indicates the nth valid field. This indicates the number of times the nth valid field appears in the first set of tables. Similarly, This indicates the number of times the first and second valid fields appear in the m-th table set.

[0089] Thus, after generating the effective field distribution matrix, the proportion of effective fields in each table set can be determined based on it, as shown in step S44c below.

[0090] S44c. Based on the effective field distribution matrix, determine the total number of effective fields appearing in each table set, and use the total number of effective fields appearing in each table set to calculate the proportion of effective fields in each table set.

[0091] In practice, the summation of each column element in the effective field distribution matrix is ​​calculated to obtain the total number of effective fields appearing in each table set. Then, the total number is divided by the total number of standard fields in the corresponding table set to obtain the proportion of effective fields in each table set. Subsequently, the standard data tables can be filtered based on the proportion, as shown in step S44d below.

[0092] S44d. Sort each set of tables in descending order of proportion, and take the standard data tables corresponding to the top three sets of tables as the target tables. In this embodiment, after filtering out the standard data tables corresponding to the top three sets of tables with the largest proportion, it can be determined whether the table name of the filtered standard data table contains the table name of the table to be cleaned. The process is as shown in step S44e below.

[0093] S44e. Determine whether the target table name contains a standard table name corresponding to the table to be cleaned. In this embodiment, if it exists, the cleaning mapping information corresponding to each standard field in the standard data table corresponding to the standard table name of the table to be cleaned is directly used as the cleaning mapping information for each valid field. Otherwise, semantic matching is required to determine the cleaning mapping information for the valid fields. The process is shown in step S44f below.

[0094] S44f. If yes, then based on the cleaning mapping information of each standard field in the standard data table corresponding to the standard table name of the table to be cleaned, determine the cleaning mapping information of each valid field; otherwise, based on the semantic information, determine the true semantics of each valid field, and based on the true semantics of each valid field, match the cleaning mapping information corresponding to each valid field from the data cleaning graph database.

[0095] In this embodiment, it is assumed that the table names of the standard data tables corresponding to the top three table sets contain the standard table names of the tables to be cleaned. For example, if the standard table name of the table to be cleaned is the same as the standard data table name of the second set of table sets, then the cleaning mapping relationship of each standard field in the standard data table corresponding to the second set of table sets is directly mapped to the cleaning mapping relationship of each valid field (that is, find the standard field corresponding to each valid field in the standard data table corresponding to the second set of table sets, and then obtain the corresponding cleaning mapping relationship).

[0096] Meanwhile, if the aforementioned condition is not met, that is, the table names of the standard data tables corresponding to the top three table sets do not contain the standard table names corresponding to the table to be cleaned; then, semantic matching is performed directly based on the true semantics, that is: according to the true semantics of each valid field, standard fields with the same semantics as each valid field are matched from several standard data tables as target fields; then, the cleaning mapping information corresponding to each target field is used as the cleaning mapping information of each valid field.

[0097] Based on the aforementioned steps S44a to S44f, after obtaining the cleaning mapping information of each valid field in the table to be cleaned, a cleaning script can be generated based on this information, as shown in step S5 below.

[0098] S5. Based on the cleaning mapping information of each valid field, a cleaning script for the table to be cleaned is generated so that the data cleaning of the table to be cleaned can be completed by running the cleaning script. In this embodiment, as previously explained, the content exploration results can provide cleaning suggestions. Therefore, this embodiment first determines the actual cleaning fields in each valid field according to the field value range distribution, data source exploration results, data type and format exploration results, and business exploration results in the content exploration results. Then, according to the cleaning method in the cleaning mapping information of each actual cleaning field, the standard data elements corresponding to each actual cleaning field, the connection relationship of the standard data elements, and the corresponding identifier code set, the actual cleaning fields in the table to be cleaned are dictionary-converted to obtain a conversion table. Finally, the cleaning script can be generated using the conversion table.

[0099] Specifically, assuming that content probing reveals the following issues with the population table: The XBDM (gender code) field contains values ​​such as "F", "M", and empty. These values ​​do not match the standard dictionary table for gender and have out-of-bounds value issues. Therefore, the gender field needs to be cleaned. In this case, it is used as an actual cleaned field. During the cleansing process, a mapping relationship is established between the source dictionary table and the standard dictionary table to convert non-standard dictionary values ​​into standard dictionary values.

[0100] For example, the CSRQ (date of birth) field contains abnormal dates such as "2041-12-01" and "1994-2-31". These values ​​are incorrect and are incorrect business field values. Therefore, the date of birth is used as an actual cleaning field. During the cleaning process, it is necessary to extract the 7th to 14th digits of the GMSFHM (citizen identity number) to obtain the correct CSRQ (date of birth) value.

[0101] For example, the XZZ_DZMC (current address_address name) field contains a large number of full-width characters, such as "B Zone 205" and "Xiangxi Road 575". Therefore, it is also used as an actual cleaning field. During data cleaning, the full-width field value is converted to a half-width field value by the full-width to half-width function getcbd compiled in the knowledge graph. Furthermore, assuming that the source contains the field HJ_SSXQH (household registration province, city, county, district), the field name of which does not conform to the requirements of the Ministry's standards; therefore, it is used as an actual cleaning field, and during cleaning, it is mapped to the standard field (data item) HJD_XZQHDM (household registration location_administrative division code) through data benchmarking operation. Furthermore, the GMSFHM (Citizen Identity Number) field contains values ​​that do not meet the citizen identity number verification rules, and there are also 15-digit citizen identity numbers. Therefore, it can also be used as a practical cleaning field. At the same time, during cleaning, the get18sfzh function compiled in the knowledge graph (i.e., graph database) is used to process abnormal citizen identity numbers into standard 18-digit numbers, and those that do not meet the verification rules are set to null values.

[0102] Finally, if there are two identical data entries in the actual population, the fields corresponding to these two identical data entries will also be used as actual cleaning fields. During the cleaning process, a unique XXZJBH (information primary key number) field needs to be generated to remove duplicate data and ensure the uniqueness of business data.

[0103] Thus, based on the content exploration results, the fields that actually need to be cleaned are determined. Then, the cleansing mapping information of the fields that actually need to be cleaned can be combined to perform dictionary transformation of the fields in the table, which is the process of converting dictionary data of one form or state into another form or state (such as gender code 1 to gender male, gender code 2 to gender female, etc.). Finally, the cleansing script of the table to be cleaned can be generated based on the cleansing methods contained in the transformation table.

[0104] Optionally, the above technologies can be encapsulated into a data governance and cleaning API service. When a new non-standard table is received, a request can be sent to the API, passing in the table name and related fields, and a cleaning script can be automatically generated.

[0105] Furthermore, taking the aforementioned population table as an example, the generated cleansing script can be, but is not limited to, the following: set hive.exec.dynamic.partition = true; set hive.exec.dynamic.partition.mode=nonstrict; set mapred.max.split.size=67108864; set hive.merge.mapfiles=true; set hive.merge.mapredfiles=true; set hive.merge.size.per.task=134217728; set hive.merge.smallfiles.avgsize=134217728; INSERT OVERWRITE TABLEysk.ysk_syrkjbxx_dd PARTITION (fq_day) select RKBM, SYRKGLLBDM, SYRKGLLB, GMSFHM, XM, XBDM, XB, MZDM, MZ, CSRQ, HJDZ_XZQHDM, HJDZ_XZQHMC, HJDZ_DZMC, XZZ_XZQHDM, XZZ_XZQHMC, XZZ_DZMC, RKGLZXLBDM, RKGLZXLB, ZXSJ, GXSJ, XXRKSJ, XXZJBH, FQ_DAY from ( select RKBM, SYRKGLLBDM, SYRKGLLB, GMSFHM, XM, XBDM, XB, MZDM, MZ, CSRQ, HJDZ_XZQHDM, HJDZ_XZQHMC, HJDZ_DZMC, XZZ_XZQHDM, XZZ_XZQHMC, XZZ_DZMC, RKGLZXLBDM, RKGLZXLB, ZXSJ, GXSJ, XXRKSJ, XXZJBH, FQ_DAY, row_number () over (partition by xxzjbh order by xxrksj desc)as rn from ( select RKBM, SYRKGLLBDM, SYRKGLLB, GMSFHM, XM, XBDM, XB, MZDM, MZ, CSRQ, HJDZ_XZQHDM, HJDZ_XZQHMC, HJDZ_DZMC, XZZ_XZQHDM, XZZ_XZQHMC, XZZ_DZMC, RKGLZXLBDM, RKGLZXLB, ZXSJ, GXSJ, XXRKSJ, XXZJBH, FQ_DAY from ysk.ysk_syrkjbxx_ddd union all select RKBM, SYRKGLLBDM, SYRKGLLB, GMSFHM, XM, XBDM, XB, MZDM, MZ, CSRQ, HJDZ_XZQHDM, HJDZ_XZQHMC, HJDZ_DZMC, XZZ_XZQHDM, XZZ_XZQHMC, XZZ_DZMC, RKGLZXLBDM, RKGLZXLB, ZXSJ, GXSJ, XXRKSJ, XXZJBH, FQ_DAY from ( a.rkbm asRKBM, a.syrjgllbdm asSYRKGLLBDM, a.syrkgllb asSYRKGLLB, default . get18sfzh ( a . sfzh ) asGMSFHM , a.name asXM, case when a.xb = 'M' then '1' when a.xb = 'F' then '2' else '9' end as XBDM, case when a.xb = 'M' when a.xb = 'F' else 'Update lock' end as XB, a.mzdm asMZDM, a.mz asMZ, default . getcsrq ( a . sfzh ) asCSRQ , a.HJ_SSXQH asHJDZ_XZQHDM, a.HJ_SSXQHMC asHJDZ_XZQHMC, default . getcbd ( a . HJ_DZ ) asHJDZ_DZMC , . a.XZZ_SSXQH asXZZ_XZQHDM, a.XZZ_SSXQHMC asXZZ_XZQHMC, default . getcbd ( a . HJXZZ_DZ ) asXZZ_DZMC , . a.rkglzxlbdm asRKGLZXLBDM, a.rkglzxl asRKGLZXLB, default . gettime ( a . zxsj , 14 ) asZXSJ , default . gettime ( a . gxsj , 14 ) asGXSJ , a.xxrksj as xxrksj , a.xxzjbh as xxzjbh , a.fq_day as fq_day from hck.hck_czrkjbxx_dd a where fq_day=cast(default.gettime(CURRENT_TIMESTAMP,8) as INT) ) t1 ) t2 ) t3 where t3.rn = 1.

[0106] Of course, the examples mentioned above are merely illustrative, and this embodiment is not limited thereto.

[0107] Therefore, through the knowledge graph-based intelligent data governance and cleaning method described in detail in steps S1 to S5 above, this invention can automatically generate cleaning scripts for metadata and data items in the production database without the need for manual writing of cleaning scripts. This effectively simplifies the data governance engineer's operation process in standardizing data items, reduces the difficulty and labor costs of cleaning operations, and improves the efficiency and accuracy of data cleaning. Therefore, this invention is very suitable for large-scale application and promotion.

[0108] like Figure 2 As shown, the second aspect of this embodiment provides a hardware device for implementing the knowledge graph-based intelligent data governance and cleaning method described in the first aspect of the embodiment, comprising: The graph database construction unit is used to construct a data cleaning graph database. The data cleaning graph database stores several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table. The cleaning mapping information of any standard field includes the cleaning method corresponding to the standard field, the standard data element corresponding to the standard field, the connection relationship of the standard data elements, and the corresponding identification code set.

[0109] The table name identification unit is used to obtain the table to be cleaned and, based on several standard data tables, determine the standard table name corresponding to the table to be cleaned.

[0110] The semantic recognition unit is used to perform semantic recognition on each field to be cleaned in the table to be cleaned, obtain the true semantics of each field to be cleaned, and use the standard table name corresponding to the table to be cleaned and the true semantics of each field to be cleaned to form the semantic information of the table to be cleaned.

[0111] The data cleaning unit is used to match the cleaning mapping information corresponding to each valid field in the table to be cleaned from the data cleaning graph database based on the semantic information.

[0112] The data cleaning unit is also used to generate a cleaning script for the table to be cleaned based on the cleaning mapping information of each valid field, so as to complete the data cleaning of the table to be cleaned by running the cleaning script.

[0113] The working process, working details and technical effects of the device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0114] like Figure 3 As shown, the third aspect of this embodiment provides another intelligent data governance and cleaning device based on knowledge graphs. Taking the device as an electronic device as an example, it includes: a memory, a processor, and a transceiver that are connected in sequence. The memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the intelligent data governance and cleaning method based on knowledge graphs as described in the first aspect of the embodiment.

[0115] For specific examples, the memory may include, but is not limited to, random access memory (RAM), read-only memory (ROM), flash memory, first-in-first-out (FIFO) memory, and / or first-in-last-out (FILO) memory, etc.; specifically, the processor may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor may be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state.

[0116] In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. For example, the processor may not be limited to microprocessors of the STM32F105 series, reduced instruction set computer (RISC) microprocessors, x86 architecture processors, or processors with integrated neural network processing units (NPUs). The transceiver may be, but is not limited to, a Wi-Fi transceiver, a Bluetooth transceiver, a General Packet Radio Service (GPRS) transceiver, a ZigBee (a low-power LAN protocol based on the IEEE 802.15.4 standard) transceiver, a 3G transceiver, a 4G transceiver, and / or a 5G transceiver. Furthermore, the device may also include, but is not limited to, a power module, a display screen, and other necessary components.

[0117] The working process, working details and technical effects of the electronic device provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0118] The fourth aspect of this embodiment provides a storage medium for storing instructions containing the knowledge graph-based intelligent data governance and cleaning method described in the first aspect of the embodiment. That is, the storage medium stores instructions that, when executed on a computer, perform the knowledge graph-based intelligent data governance and cleaning method as described in the first aspect of the embodiment.

[0119] The storage medium refers to a carrier for storing data, which may include, but is not limited to, floppy disks, optical disks, hard disks, flash memory, USB flash drives, and / or memory sticks. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.

[0120] The working process, working details and technical effects of the storage medium provided in this embodiment can be found in the first aspect of the embodiment, and will not be repeated here.

[0121] The fifth aspect of this embodiment provides a computer program product containing instructions that, when executed on a computer, cause the computer to perform the knowledge graph-based intelligent data governance and cleaning method as described in the first aspect of this embodiment, wherein the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device.

[0122] Finally, it should be noted that the above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A knowledge graph-based intelligent data governance and cleaning method, characterized in that, include: A data cleaning graph database is constructed, wherein the data cleaning graph database stores several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table, and the cleaning mapping information of any standard field includes the cleaning method corresponding to the standard field, the standard data element corresponding to the standard field, the connection relationship of the standard data elements, and the corresponding identification code set. Obtain the table to be cleaned, and determine the standard table name corresponding to the table to be cleaned based on several standard data tables; Semantic recognition is performed on each field to be cleaned in the table to be cleaned to obtain the true semantics of each field to be cleaned. Then, the semantic information of the table to be cleaned is formed by using the standard table name corresponding to the table to be cleaned and the true semantics of each field to be cleaned. Based on the semantic information, the cleaning mapping information corresponding to each valid field in the table to be cleaned is matched from the data cleaning graph database; Based on the cleaning mapping information of each valid field, a cleaning script is generated for the table to be cleaned, so that the data cleaning of the table to be cleaned can be completed by running the cleaning script.

2. The method according to claim 1, characterized in that, A data cleaning graph database was constructed, including: Obtain the table names and corresponding table structures of several standard tables. The table structure includes field names and field name comments, and any table name is in Chinese. Based on the table names of each departmental standard table, the English names of each departmental standard table are determined. Data splitting is performed on the field names in the table structure of each departmental standard table to obtain the standard data element corresponding to each field in each departmental standard table; Based on the field names in the table structure of each departmental standard table, determine the connection relationship and identification code set of the standard data elements corresponding to each field in each departmental standard table; Obtain the cleaning and verification methods corresponding to each standard data element, and determine the cleaning and verification methods for each field in each departmental standard table based on the cleaning and verification methods corresponding to each standard data element. Based on the table name, English name, field name and field name comments of each departmental standard table, each standard data table is generated. Then, by using the cleaning and verification methods of each field in each departmental standard table, as well as the standard data elements corresponding to each field, the connection relationship of standard data elements and the identification code set, the cleaning mapping information of each standard field in each standard data table is determined. Each standard data table and the cleaning mapping information of each standard field in each standard data table are stored in a graph database to obtain the data cleaning graph database.

3. The method according to claim 1, characterized in that, Based on several standard data tables, the standard table names corresponding to the table to be cleaned are determined, including: Obtain the standard table names of all standard data tables and the original table names of the tables to be cleaned, and preprocess the original table names to obtain the preprocessed original table names. Calculate the edit distance between the preprocessed original table name and each standard table name; Determine if there exists any edit distance equal to 1; If not, then the preprocessed original table names are segmented into words to obtain a word set; Calculate the semantic similarity between the word set and each standard table name; The total similarity between the original table name and each standard table name is calculated based on the edit distance between the preprocessed original table name and each standard table name, as well as the semantic similarity between the word set and each standard table name. From all standard table names, select those with a total similarity greater than the similarity threshold, and use the standard table name with the highest total similarity among the selected standard table names as the standard table name of the table to be cleaned.

4. The method according to claim 1, characterized in that, The cleaning mapping information for any field also includes: the validation method corresponding to that field; Specifically, based on the semantic information, the cleaning mapping information corresponding to each valid field in the table to be cleaned is matched from the data cleaning graph database, including: Based on the true semantics of each field to be cleaned in the semantic information, the standard field corresponding to each field to be cleaned is determined from the data cleaning graph database, and the verification method of the standard field corresponding to each field to be cleaned is used as the verification method of each field to be cleaned. Based on the validation method of each field to be cleaned, the table to be cleaned is subjected to content probing to obtain content probing results, wherein the content probing results include the field null value rate; Based on the field null value rate in the content exploration results, select valid fields from each field to be cleaned; Based on the standard table name and each valid field corresponding to the table to be cleaned, the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database, or the true semantics of each valid field is determined according to the semantic information, and the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database according to the true semantics of each valid field.

5. The method according to claim 4, characterized in that, Based on the standard table name and each valid field corresponding to the table to be cleaned, the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database, including: Using the standard fields in each standard data table of the data cleaning graph database, several table sets are generated; Based on the valid fields and several table sets, a valid field distribution matrix is ​​generated. The size of the valid field distribution matrix is ​​n×m, where n is the total number of valid fields and m is the number of table sets. The elements in the valid field distribution matrix... This indicates the number of times the nth valid field appears in the mth table set; Based on the effective field distribution matrix, the total number of effective fields appearing in each table set is determined, and the proportion of effective fields in each table set is calculated using the total number of effective fields appearing in each table set. Sort the table sets in descending order of proportion, and take the standard data table corresponding to the top three table sets as the target table. Determine whether the target table contains a standard table name corresponding to the table to be cleaned; If so, the cleaning mapping information of each valid field is determined based on the cleaning mapping information of each standard field in the standard data table corresponding to the standard table name of the table to be cleaned. Otherwise, the true semantics of each valid field are determined based on the semantic information, and the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database based on the true semantics of each valid field.

6. The method according to claim 5, characterized in that, Based on the true semantics of each valid field, the cleaning mapping information corresponding to each valid field is matched from the data cleaning graph database, including: Based on the true semantics of each valid field, standard fields with the same semantics as each valid field are matched from several standard data tables and used as target fields. The cleaning mapping information corresponding to each target field is used as the cleaning mapping information for each valid field.

7. The method according to claim 4, characterized in that, The content exploration results also include: field value range distribution, data element exploration results, data type and format exploration results, and business exploration results; The cleaning script for the table to be cleaned is generated based on the cleaning mapping information of each valid field, including: Based on the field value range distribution, data element exploration results, data type and format exploration results, and business exploration results in the content exploration results, the actual cleaned fields in each valid field are determined. Based on the cleaning method in the cleaning mapping information of each actual cleaning field, the standard data elements corresponding to each actual cleaning field, the connection relationship of the standard data elements, and the corresponding identification code set, the actual cleaning fields in the table to be cleaned are transformed into a dictionary to obtain a transformation table. The cleaning script is generated using the transformation table.

8. A knowledge graph-based intelligent data governance and cleaning device, characterized in that, include: The graph database construction unit is used to construct a data cleaning graph database. The data cleaning graph database stores several standard data tables and cleaning mapping information corresponding to each standard field in each standard data table. The cleaning mapping information of any standard field includes the cleaning method corresponding to the standard field, the standard data element corresponding to the standard field, the connection relationship of the standard data elements, and the corresponding identification code set. The table name identification unit is used to obtain the table to be cleaned and, based on several standard data tables, determine the standard table name corresponding to the table to be cleaned. The semantic recognition unit is used to perform semantic recognition on each field to be cleaned in the table to be cleaned, obtain the true semantics of each field to be cleaned, and use the standard table name corresponding to the table to be cleaned and the true semantics of each field to be cleaned to form the semantic information of the table to be cleaned. The data cleaning unit is used to match the cleaning mapping information corresponding to each valid field in the table to be cleaned from the data cleaning graph database based on the semantic information. The data cleaning unit is also used to generate a cleaning script for the table to be cleaned based on the cleaning mapping information of each valid field, so as to complete the data cleaning of the table to be cleaned by running the cleaning script.

9. An electronic device, characterized in that, include: A memory, a processor, and a transceiver are sequentially connected in communication, wherein the memory is used to store computer programs, the transceiver is used to send and receive messages, and the processor is used to read the computer programs and execute the knowledge graph-based intelligent data governance and cleaning method as described in any one of claims 1 to 7.

10. A computer program product containing instructions, characterized in that, When the instructions are executed on the computer, the computer performs the knowledge graph-based intelligent data governance and cleaning method as described in any one of claims 1 to 7.

Images