Method, device and storage medium for data governance

Abstract

The application discloses a data governance method and device and a storage medium, and belongs to the field of communication. The method comprises the following steps: reading a storage file from a database server, wherein the storage file is used for storing data belonging to at least one service; acquiring at least one data mode of a first data set, wherein the first data set comprises data belonging to the same service saved by the storage file, and the at least one data mode is used for indicating the structure of each data included in the first data set; acquiring at least one data feature of the first data set based on the at least one data mode; and acquiring a first data standard based on the at least one data feature, wherein the first data standard is used for constraining the standardization of each data included in the first data set. The application can improve the efficiency and accuracy of acquiring data standards.

Description

[0001] This application claims priority to Chinese Patent Application No. 202111422755.7, filed on November 26, 2021, entitled "A Data Processing Method and Computer", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computers, and in particular to a method, apparatus and storage medium for data governance. Background Technology

[0003] Database tables typically include at least one column, each used to store at least one piece of data. Any column in a database table can be associated with a data standard, which provides normative constraints on the data to be stored in that column. For example, the data standard defines the format, type, and / or length of the data that can be stored in that column.

[0004] Currently, technicians can manually generate data standards for any column of data in a database table and associate that column of data with that data standard. Alternatively, technicians can manually select a data standard for any column of data in a database table from a large pool of pre-generated data standards and associate the selected data standard with that column of data.

[0005] Currently, the data standards associated with any column of data in a database table are either manually generated or manually selected by technical personnel, resulting in low efficiency and accuracy in obtaining data standards. Summary of the Invention

[0006] This application provides a data governance method, apparatus, and storage medium to improve the efficiency and accuracy of data standard acquisition. The technical solution is as follows:

[0007] Firstly, this application provides a data governance method. In this method, a storage file is read from a database server, the storage file storing data belonging to at least one business function. At least one data pattern of a first data set is obtained. The first data set includes data belonging to the same business function stored in the storage file, and the at least one data pattern indicates the structure of each piece of data included in the first data set. At least one data feature of the first data set is obtained based on the at least one data pattern. A first data standard is obtained based on the at least one data feature, the first data standard constraining the standardization of each piece of data included in the first data set. By obtaining at least one data pattern of the first data set in this way, and automatically obtaining the first data standard associated with the first data set based on the at least one data pattern, the efficiency and accuracy of obtaining the data standard are improved.

[0008] In one possible implementation, the storage file is a structured data file. This structured data file stores data belonging to at least one business function in a list format, with data belonging to the same business function stored in the same column of the structured data file. This simplifies the complexity of retrieving the first data set by directly retrieving a column of data from the storage file.

[0009] In one possible implementation, the storage file is a semi-structured data file. This semi-structured data file uses tag blocks to store data belonging to at least one business function. Data belonging to the same business function is stored in the same tag block of the semi-structured data file. This simplifies the complexity of retrieving the first data set by directly retrieving data located in the same tag block from the storage file.

[0010] In another possible implementation, the at least one data pattern includes a basic data pattern used to indicate the basic structure of each data item included in the first data set. This basic structure includes one or more of the following: a mixed alphanumeric structure, an integer structure, a floating-point structure, a Boolean structure, an address structure, an identifier structure, or a date structure. Because the basic data pattern indicates the basic structure of each data item included in the first data set, a first data standard can be obtained based on the data pattern of the first data set.

[0011] In another possible implementation, each piece of data in the first data set is identified based on at least one specified regular expression to obtain the basic data pattern.

[0012] In another possible implementation, the at least one data pattern further includes a prefix data pattern, which is used to indicate multiple data in the first data set that include a first prefix, the first prefix being the longest common prefix of the multiple data, and the basic data pattern corresponding to the portion of each of the multiple data except for the first prefix being the same.

[0013] In another possible implementation, data corresponding to a first data pattern are obtained from a first dataset, where at least one data pattern includes the first data pattern. Based on the obtained data, the number of occurrences and / or frequency of occurrence of the data corresponding to the first data pattern are obtained, and the at least one data feature includes the number of occurrences and / or frequency of occurrence of the data corresponding to the first data pattern. This approach achieves the acquisition of data features based on data patterns, and the data features include the distribution characteristics corresponding to the data patterns, thus enriching the content of the data features.

[0014] In another possible implementation, the data type in the first data set is obtained based on at least one data pattern. Based on the data type in the first data set, at least one data feature of the first data set is obtained. This approach, which obtains data types based on data patterns and then derives data features from data types, enriches the methods for acquiring data features.

[0015] In another possible implementation, the first data set includes first data, the type of which includes one or more of the following: a basic type or a technical type of the first data, the basic type describing the data components of the first data, and the technical type describing the functionality implemented by the first data. This results in a rich variety of data types.

[0016] In another possible implementation, the basic type of the first data is an integer, floating-point number, or boolean, and the technical type of the first data is a code type, encoding type, flag type, category type, description type, or measurement type, thus enriching the content of the data type.

[0017] In another possible implementation, statistical features are obtained based on the data belonging to a first type included in the first dataset. These statistical features include the maximum, minimum, mean, deviation, variance, median, percentile, and / or standard deviation of the data belonging to the first type. The first type includes integer, floating-point, measure, and / or code types, and at least one data feature includes this statistical feature. And / or, distribution features are obtained based on the data belonging to a second type included in the first dataset. These distribution features include the number of occurrences and / or frequency of occurrence of the data belonging to the second type. The second type includes flag, Boolean, category, and / or code types, and at least one data feature includes this distribution feature. This implementation obtains data features based on data types, enriching the methods for acquiring data features.

[0018] In another possible implementation, the first data set includes the second data set, which has a mixed alphanumeric structure. Data belonging to the first type includes the numeric portion of the second data set, and data belonging to the second type includes the alphanumeric portion. This enriches the content of the data features.

[0019] In another possible implementation, the data features of the first data set include at least one word segmentation, which is obtained by segmenting the third data and removing stop words. The third data includes business attribute description information corresponding to the first data set and / or data of description types in the first data set. This enriches the content of the data features, and removing stop words avoids them affecting the accuracy of the data acquisition standards.

[0020] In another possible implementation, the data features of the first dataset include the number of occurrences and / or frequency of data corresponding to the first language, and the language to which the data describing the type in the first dataset belongs includes the first language. This enriches the content of the data features.

[0021] In another possible implementation, based on at least one data feature, a connected graph to which a first data set belongs is determined. Each node in this connected graph represents a different data set, and the similarity between the neighboring nodes of the first data set and the first data set exceeds a specified threshold. A first data standard is obtained based on the data sets included in this connected graph. Since the data sets in this connected graph are similar, the data standard can be accurately summarized based on the data sets included in the connected graph, thus improving the accuracy of obtaining the data standard.

[0022] In another possible implementation, based on the at least one data feature and the service type of the data in the first data set, the connected graph to which the first data set belongs is determined. The service type of the data is date, region, IP address, or identifier. By incorporating the service type into the determination of the connected graph, the accuracy of the connected graph determination is improved.

[0023] In another possible implementation, at least one data standard is obtained, which includes data standards associated with other data sets in the connected graph besides the first data set. Based on the association frequency of each data standard in the at least one data standard, one data standard is selected as the first data standard. This improves the accuracy of obtaining the first data standard by utilizing the association frequency.

[0024] In another possible implementation, the data standard associated with the other data set is updated to the first data standard. Since the data standard associated with the other data set is the same as the first data standard, updating the data standard associated with the other data set to the first data standard can remove the duplicate data standard.

[0025] In another possible implementation, a first data standard is generated based on at least one data pattern and / or at least one data feature of each data set in the connected graph, thereby achieving automated generation of data standards and improving the efficiency and accuracy of data standard generation.

[0026] In another possible implementation, the normativity includes data integrity, data consistency, and / or data accuracy. The first data standard includes a first standard content, a second standard content, and / or a third standard content. The first standard content is used to constrain the data integrity of the first data set, the second standard content is used to constrain the data consistency of the first data set, and the third standard content is used to constrain the data accuracy of the first data set.

[0027] In another possible implementation, a first standard content is generated based on the distribution characteristics of the data in each dataset; and / or, a second standard content is generated based on at least one data pattern in each dataset; and / or, a third standard content is generated based on the distribution characteristics and / or statistical characteristics of the data in each dataset. This achieves automated generation of data standards, improving the efficiency and accuracy of data standard generation.

[0028] Secondly, this application provides a data governance apparatus for performing the method in the first aspect or any possible implementation thereof. Specifically, the apparatus includes units for performing the method in the first aspect or any possible implementation thereof.

[0029] Thirdly, this application provides a computer device including at least one processor and a memory, wherein the at least one processor is configured to be coupled to the memory, read and execute instructions in the memory to implement the method in the first aspect or any possible implementation of the first aspect.

[0030] Fourthly, this application provides a computer program product comprising a computer program stored in a computer-readable storage medium, wherein the computer program is loaded by a processor to implement the method described in the first aspect or any possible implementation thereof.

[0031] Fifthly, this application provides a computer-readable storage medium for storing a computer program, which is loaded by a processor to execute the method described in the first aspect or any possible implementation thereof.

[0032] In a sixth aspect, this application provides a chip including a memory and a processor, wherein the memory is used to store computer instructions, and the processor is used to call and execute the computer instructions from the memory to perform the method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of a network architecture provided in an embodiment of this application;

[0034] Figure 2 This is a schematic diagram of another network architecture provided in an embodiment of this application;

[0035] Figure 3 This is a schematic diagram of a data standard provided in an embodiment of this application;

[0036] Figure 4 This is a flowchart of a data governance method provided in an embodiment of this application;

[0037] Figure 5 This is a flowchart of another data governance method provided in an embodiment of this application;

[0038] Figure 6 This is a schematic diagram of a display interface provided in an embodiment of this application;

[0039] Figure 7 This is a flowchart of a method for obtaining data types provided in an embodiment of this application;

[0040] Figure 8 This is a connected graph provided in an embodiment of this application;

[0041] Figure 9 This is another connected graph provided in the embodiments of this application;

[0042] Figure 10 This is a schematic diagram of another display interface provided in an embodiment of this application;

[0043] Figure 11 This is a schematic diagram of a data governance device structure provided in an embodiment of this application;

[0044] Figure 12 This is a schematic diagram of another computer device structure provided in an embodiment of this application. Detailed Implementation

[0045] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0046] See Figure 1 This application provides a network architecture 100, including:

[0047] The management device 101 and the database server 102 communicate with each other.

[0048] In some embodiments, the number of database servers 102 is greater than or equal to one, that is, the network architecture 100 includes at least one database server 102. The management device 101 and the at least one database server 102 are connected to a communication network, and the management device 101 communicates with each database server 102 through the communication network.

[0049] In some embodiments, each database server 102 includes a database for storing data. For each database server 102, the database includes at least one storage file for storing data. The storage file stores data belonging to at least one business function. Optionally, the storage file stores at least one data set, each data set including data belonging to the same business function.

[0050] In some embodiments, the management device 101 is used to manage the at least one database server 102.

[0051] For example, see Figure 2 The at least one database server 102 includes a data lake, a Structured Query Language (SQL) server, and / or an Oracle server, where Oracle is a relational database management system. The data lake, SQL server, and Oracle server are all database servers used to store data, and the management device 101 is used to manage the data lake, SQL server, and Oracle server, etc.

[0052] For any database server 102, the storage files in the database server 102 include structured data files and / or semi-structured data files, both of which are files used to store data.

[0053] This structured data file uses a list format to store data. Each column in the structured data file stores data for one business function; that is, each column is a data set. Optionally, this structured data file can be a database table, for example, see the structured data file shown in Table 1 below. This structured data file uses lists to store data. The structured data file shown in Table 1 is a database table, which includes four columns of data, each representing a different data set. The first column stores the customer code, meaning the first column stores the customer code; the second column stores the contract number, meaning the second column stores the contract number; the third column stores the customer profile, meaning the third column stores the customer introduction; and the fourth column stores the customer name, meaning the fourth column stores the customer name.

[0054] Table 1: Customer Information List

[0055]

[0056] The semi-structured data file is configured to store data in the form of tag blocks, and is a non-list-form storage file. In said semi-structured data file, data belonging to the same service is stored centrally in tag blocks, that is, each data item in a data set is stored centrally in one tag block of the semi-structured data file. Optionally, the structured data file is an extensible markup language (xml) file or the like. For example, in the following semi-structured data file 1, the customer code data "KH000001", "KH000002", "AH100001", "AH200002", ..., "KH100001" are stored centrally in one tag block of the xml file, wherein <客户编码> is the start tag of the tag block, and < / 客户编码> is the end tag of the tag block; the contract number data "SG1234", "SG1235", "SG1278", "SG1456", ..., "SG1689" are stored centrally in one tag block of the xml file, wherein <合同号> is the start tag of the tag block, and < / 合同号> is the end tag of the tag block; the customer profile data "large enterprise established in 1991 specializing in patent business", "an enterprise operating Sichuan cuisine", "main business: doors, windows and cement", "world-leading ICT infrastructure and intelligent terminal provider", ..., "main business: game development and social services" are stored centrally in one tag block of the xml file, wherein <客户简介> is the start tag of the tag block, and < / 客户简介> is the end tag of the tag block; the customer name data "Enterprise 1", "Enterprise 2", "Enterprise 3", "Enterprise 4", ..., "Enterprise 100" are stored centrally in one tag block of the xml file, wherein <客户名称> is the start tag of the tag block, and < / 客户名称> is the end tag of the tag block.

[0057] That is, the semi-structured data file 1 shown below includes four data sets, which are data set 1, data set 2, data set 3 and data set 4. Data set 1 includes the following customer codes: "KH000001", "KH000002", "AH100001", "AH200002", ..., "KH100001"; data set 2 includes the following contract numbers: "SG1234", "SG1235", "SG1278", "SG1456", ..., "SG1689"; data set 3 includes the following customer profiles: "a large enterprise established in 1991 that specializes in patent business", "an enterprise operating Sichuan cuisine", "main business: doors, windows and cement", "a global leading provider of ICT infrastructure and smart terminals", ..., "main business: game development and social services"; data set 4 includes the following customer names: "Enterprise 1", "Enterprise 2", "Enterprise 3", "Enterprise 4", ..., "Enterprise 100".

[0058] Semi-structured data file 1:

[0059] <title> Customer Information< / title>

[0060] <Customer Code>(Data Set 1)

[0061] KH000001;

[0062] KH000002;

[0063] AH100001;

[0064] AH200002;

[0065] ……

[0066] KH100001;

[0067] < / Customer Code>

[0068] <Contract Number>(Data Set 2)

[0069] SG1234;

[0070] SG1235;

[0071] SG1278;

[0072] SG1456;

[0073] ……

[0074] SG1689;

[0075] < / Contract Number>

[0076] <Customer Profile>(Data Set 3)

[0077] a large enterprise established in 1991 that specializes in patent services;

[0078] is an enterprise operating Sichuan cuisine;

[0079] mainly deals in doors, windows and cement;

[0080] a world-leading provider of ICT infrastructure and smart terminals;

[0081] ……

[0082] mainly engages in game development and social services;

[0083] < / Customer Profile>

[0084] <Customer Name>(Data Set 4)

[0085] Enterprise 1;

[0086] Enterprise 2;

[0087] Enterprise 3;

[0088] Enterprise 4;

[0089] ……

[0090] Enterprise 100;

[0091] < / Customer Name>

[0092]

[0093] It should be noted that, for a row of data in any column in the structured data file shown in Table 1, the row of data may be a null value. The null value indicates that the user does not store data in that row of the column, and the database system automatically stores a preset null value in that row of the column. Similarly, the database system can also store null values in semi-structured data files.

[0094] For any storage file included in any of the foregoing servers 102, for any data set included in the storage file, the data set is associated with a data standard. The data standard is used to constrain the normalization of each piece of data included in the data set. That is, when storing data in the data set, the data needs to comply with the normalization specified by the data standard, and the data can only be saved to the data set if the data complies with the normalization specified by the data standard.

[0095] In some embodiments, the normativity includes data integrity, data consistency, and / or data accuracy. The data standard includes first standard content, second standard content, and / or third standard content. The first standard content constrains the data integrity of the data set, the second standard content constrains the data consistency of the data set, and the third standard content constrains the data accuracy of the data set. Optionally, the data standard may also include other content such as a standard name and / or standard number.

[0096] For example, see Figure 3 For a dataset 1 that includes customer codes, the data standard 1 associated with dataset 1 is as follows. The first standard content in data standard 1 includes a data length of "8 bits". For data to be saved to dataset 1, the first standard content defines the length of the data to be saved as 8 bits, thus constraining the integrity of the data to be saved. Optionally, the first standard content also indicates whether null values ​​are allowed to be input into dataset 1. The second standard content in data standard 1 includes a prefix data pattern "KH{number}[6 bits], AH{number}[6 bits]", a general pattern "{English size}[2 bits]{number}[6 bits]", and a data type "character". The second standard content defines that the data to be saved consists of "KH" and 6 digits or "AH" and 6 digits, thus constraining the consistency of the data to be saved. The third standard in this data standard 1 includes the value range "KH[000000-999999], AH[000000-999999]". The third standard defines that the 6 digits in the data to be saved are greater than or equal to 000000 and less than or equal to 999999, thereby constraining the accuracy of the data to be saved.

[0097] Data Standard 1

[0098]

[0099] for Figure 3 The meaning of data standard 2 can be found in the description of data standard 1 above, and will not be repeated here.

[0100] For any data set in server 102, management device 101 can obtain a data standard for that data set and associate the data standard with that data set. Thus, when saving data to that data set, server 102 first determines whether the data conforms to the specifications defined by the data standard. If the data conforms to the specifications defined by the data standard, the data is saved to the data set. If the data does not conform to the specifications defined by the data standard, the data is discarded, or a prompt is given to the user to modify the data.

[0101] See Figure 4 and Figure 5 This application provides a data governance method 400, which is applied to... Figure 1 or Figure 2 The network architecture 100 shown above, and the execution entity of the method 400 are the aforementioned Figure 1 or Figure 2 The management device 101 in the network architecture 100 shown. The method 400 includes:

[0102] Step 401: Read the storage file from the database server. The storage file is used to store data belonging to at least one business function. Obtain the first data set from the storage file.

[0103] In step 401, a storage file is retrieved from the database server, and a dataset is retrieved from the storage file to form a first data set. The first data set includes data belonging to the same business. Optionally, the first data set is a data set in the storage file that has not yet been associated with a data standard, or the first data set is a data set in the storage file that has been associated with a data standard.

[0104] See Figure 1 or Figure 2 The management device communicates with at least one server, each server including a storage file for storing data. The management device reads the storage file stored on any one of the servers and retrieves a first set of data from that storage file.

[0105] For example, see Figure 2 The management device retrieves storage files from a data lake, SQL Server, or Oracle Server. These storage files may be structured, such as a database table; or they may be semi-structured, such as an XML file. Assuming the storage file is a database table, the management device retrieves a column of data from that table as the first data set.

[0106] In some embodiments, the management device also displays the storage file and highlights the first data set within it. For example, see... Figure 6 Suppose that the management device retrieves a storage file from the data lake, which is a database table as shown in Table 1. The first data set is a column of data in the database table (e.g., the first column of data in the database table shown in Table 1, where the first column of data is the customer code). The management device displays the database table and bolds the column of data in the database table to highlight the first data set for easy browsing by technicians.

[0107] In some embodiments, the stored file is a structured data file. A column of data not associated with a data standard is retrieved from the structured data file as a first data set; alternatively, a column of data associated with a data standard is retrieved from the structured data file as a first data set. For example, referring to the structured data file shown in Table 1 above, assume that none of the four columns of data in the structured data file are associated with a data standard. The first column of data is read from the structured data file as the first data set, i.e., the first data set includes customer codes “KH000001”, “KH000002”, “AH100001”, “AH200002”, ..., “KH100001”.

[0108] In some embodiments, the stored file is a semi-structured data file. A dataset not associated with a data standard is obtained from the semi-structured data file to form a first data set; alternatively, a column of data associated with a data standard is obtained from the structured data file to form the first data set. For example, referring to the semi-structured data file 1 described above, assume that none of the four data sets in the structured data file 1 are associated with a data standard. Data set 1 is read from the structured data file 1 as the first data set, i.e., the first data set includes customer codes “KH000001”, “KH000002”, “AH100001”, “AH200002”, ..., “KH100001”.

[0109] Step 402: Obtain at least one data pattern of the first data set, the at least one data pattern being used to indicate the structure of the data included in the first data set.

[0110] The at least one data pattern includes a basic data pattern. Optionally, the basic data pattern is used to indicate the basic structure of each data item included in the first data set. Optionally, the basic structure includes one or more of the following: a mixed alphanumeric structure, an integer structure, a floating-point number structure, a text structure, a Boolean structure, an IP address structure, an identifier structure, or a date structure.

[0111] In some embodiments, the management device includes at least one specified regular expression. In step 402, based on the specified at least one regular expression, each piece of data in the first data set is identified to obtain at least one basic data pattern corresponding to the first data set.

[0112] For example, the management device includes regular expressions for numbers such as "\d", lowercase English regular expressions such as "[az]", uppercase English regular expressions such as "[AZ]", regular expressions for regions such as "^China$|^UK$..." and / or regular expressions for dates such as "^(19|20)\d\d(0[1-9]|1[0-2])(0[1-9]|[1-2]\d|3[0-1])$", etc.

[0113] In step 402, for any data in the first data set (hereinafter referred to as the first data for ease of explanation), the first data is identified using each of the at least one regular expression to obtain at least one basic data pattern. This at least one basic data pattern includes the basic data pattern corresponding to the first data. Optionally, the at least one basic data pattern also includes the basic data pattern corresponding to each sub-data in the first data.

[0114] The first data may include at least one sub-data, and for each sub-data, the sub-data includes at least one consecutive data element, and each data element in the sub-data is a data element of the same type.

[0115] For example, suppose the first data is "KH000001". The first data "KH000001" includes a first sub-data and a second sub-data. The first sub-data is "KH", and the second sub-data is "000001". The first sub-data includes two data elements, "K" and "H", both uppercase. The second sub-data includes six data elements: five numeric elements "0" and one numeric element "1". Using at least one regular expression, the first data "KH000001" is identified, resulting in three basic data patterns: basic data pattern 1 corresponding to the first data "KH000001", basic data pattern 2 corresponding to the first sub-data "KH", and basic data pattern 3 corresponding to the second sub-data "000001".

[0116] Basic data pattern 1 is {uppercase English letters} [2 digits] {numbers} [6 digits]. Basic data pattern 1 indicates that the first data "KH000001" consists of two uppercase English letters and six number digits, for a total of eight data elements. Basic data pattern 2 is {uppercase English letters} [2 digits]. Basic data pattern 2 indicates that the first sub-data "KH" consists of two uppercase English letters. Basic data pattern 3 is {numbers} [6 digits]. Basic data pattern 3 indicates that the second sub-data "000001" consists of six data elements.

[0117] For example, suppose the first data is "a leading global ICT infrastructure and smart terminal provider." This first data includes a third, fourth, and fifth sub-data. The third sub-data is "leading globally," the fourth sub-data is "ICT," and the fifth sub-data is "infrastructure and smart terminal provider." The third sub-data, "leading globally," includes five data elements, all of which are Chinese characters. The fourth sub-data, "ICT," includes three data elements, all of which are uppercase English letters. The fifth sub-data, "infrastructure and smart terminal provider," includes twelve data elements, all of which are Chinese characters. By using at least one regular expression to identify the first data, four basic data patterns are obtained: basic data pattern 4 corresponding to the first data, basic data pattern 5 corresponding to the third sub-data, basic data pattern 6 corresponding to the fourth sub-data, and basic data pattern 7 corresponding to the fifth sub-data.

[0118] Basic data pattern 4 is {Chinese}[5 characters]{uppercase English}[3 characters]{Chinese}[12 characters]. Basic data pattern 4 indicates that the first data element consists of 5 Chinese characters, 3 uppercase English letters, and 12 Chinese characters, totaling 20 data elements. Basic data pattern 5 is {Chinese}[5 characters]. Basic data pattern 5 indicates that the third sub-data element consists of 5 Chinese characters. Basic data pattern 6 is {uppercase English}[3 characters]. Basic data pattern 6 indicates that the fourth sub-data element consists of 3 uppercase English letters. Basic data pattern 7 is {Chinese}[12 characters]. Basic data pattern 7 indicates that the fifth sub-data element consists of 12 Chinese characters.

[0119] Specifically, if the underlying data pattern corresponding to the first data indicates that the first data includes data elements of one type, for example, if the underlying data pattern indicates that the first data includes numeric data elements, then the underlying structure of the first data is an integer structure. If the underlying data pattern indicates that the first data includes both numeric and alphanumeric data elements, then the underlying structure of the first data is a mixed alphanumeric structure. If the underlying data pattern indicates that the first data includes both numeric and decimal data elements, and the data elements before and after the decimal point are all numeric, then the underlying structure of the first data is a floating-point structure. If the underlying data pattern indicates that the first data is "true or false", then the underlying structure of the first data is a Boolean structure. If the underlying data pattern indicates that the first data is a date type, then the underlying structure of the first data is a date structure. If the underlying data pattern indicates that the first data is an address, then the underlying structure of the first data is an address structure. If the underlying data pattern indicates that the first data is an identifier, then the underlying structure of the first data is an identifier structure.

[0120] In some embodiments, the at least one data pattern may further include a prefix data pattern, which indicates multiple data in a first data set that include a first prefix, the first prefix being the longest common prefix of the multiple data, and the underlying data pattern corresponding to the portion of each of the multiple data excluding the first prefix being the same. The prefix data pattern includes the first prefix and the underlying data pattern.

[0121] The prefix data pattern corresponding to the first data set may include one or more prefix data patterns, each prefix data pattern corresponding to a different prefix. Optionally, based on the above at least one basic data pattern, multiple data that include the same prefix are determined in the first data set. For ease of explanation, the longest common prefix included by these multiple data is called the first prefix, and the basic data pattern corresponding to the part of each of these multiple data except for the first prefix is ​​the same.

[0122] For example, suppose the first data set includes multiple data such as “KH000001”, “KH000002”, ..., “KH100001”. Based on the above basic data pattern 1, basic data pattern 2 and basic data pattern 3, “KH000001”, “KH000002”, ..., “KH100001” including the first prefix “KH” are determined in the first data set. Among them, the first prefix “KH” is the longest common prefix among “KH000001”, “KH000002”, ..., “KH100001”, and the basic data pattern corresponding to each data except “KH” is {digit} [6 bits]. Therefore, the prefix data pattern 1 used to indicate "KH000001", "KH000002", ..., "KH100001" is KH{digit}[6 bits], that is, prefix data pattern 1 includes the first prefix "KH" and the basic data pattern "{digit}[6 bits]" corresponding to each data except the first prefix "KH". Similarly, prefix data pattern 2 is also obtained from the first data set, and prefix data pattern 2 is AH{digit}[6 bits].

[0123] In some embodiments, for multiple data corresponding to any given prefix data pattern, the prefix data pattern can be further subdivided into finer-grained data patterns based on the suffix of each data point, resulting in multiple fine-grained data patterns corresponding to the prefix data pattern. For the multiple data corresponding to the prefix data pattern, each set of data includes data corresponding to a fine-grained data pattern. The data corresponding to any fine-grained data pattern includes a first suffix, which is the longest common suffix included in the data corresponding to the fine-grained data pattern.

[0124] In summary, the data pattern of the first data includes the basic data pattern corresponding to the first data, the basic data pattern corresponding to each sub-data in the first data, the prefix data pattern corresponding to the first data, and / or the fine-grained data pattern corresponding to the first data, etc. Furthermore, at least one data pattern of the first data set includes the data pattern corresponding to each data in the first data set. The data patterns corresponding to each data in the first data set may be the same, or some data in the first data set may have the same data pattern.

[0125] Step 403: Based on the at least one data pattern, obtain at least one data feature of the first data set.

[0126] The first dataset has at least one data feature, which includes different data features. These different data features are obtained in different ways. The process of obtaining data features will be explained in detail below, in the following cases.

[0127] First, the at least one data feature includes statistical features corresponding to data patterns in the first data set. For any data pattern in the first data set, for ease of explanation, this data pattern is referred to as the first data pattern. The statistical features corresponding to the first data pattern are used to reflect the statistical situation of the data corresponding to the first data pattern in the first data set. Optionally, the statistical features corresponding to the first data pattern include the number of occurrences and / or the frequency of occurrence of the data corresponding to the first data pattern.

[0128] In some embodiments, for the number of data occurrences and / or the frequency of data occurrences corresponding to a first data pattern, each data corresponding to the first data pattern is obtained from a first data set. Based on each data, the number of data occurrences and / or the frequency of data occurrences corresponding to the first data pattern is obtained. Optionally, the number of each data is counted to obtain the number of data occurrences corresponding to the first data pattern, and the frequency of data occurrences corresponding to the first data pattern is calculated based on the number of each data and the total number of data included in the first data set.

[0129] The first data pattern includes at least one basic data pattern, at least one prefix data pattern, and / or at least one fine-grained data pattern. Therefore, the statistical characteristics corresponding to the first data pattern include the number of data occurrences and / or the frequency of data occurrences for each basic data pattern, the number of data occurrences and / or the frequency of data occurrences for each prefix data pattern, and / or, the number of data occurrences and / or the frequency of data occurrences for each fine-grained data pattern.

[0130] For example, suppose the first dataset includes 100 customer codes, namely “KH000001”, “KH000002”, “AH100001”, “AH200002”, ..., “KH100001”. At least one data pattern in the first dataset includes basic data pattern 1 “{English size}[2 digits]{number}[6 digits]”, basic data pattern 2 “{English size}[2 digits]”, basic data pattern 3 “{number}[6 digits]”, prefix data pattern 1 “KH{number}[6 digits]”, and prefix data pattern 2 “AH{number}[6 digits]”. The data corresponding to basic data pattern 1 includes the aforementioned 100 customer codes, so the number of occurrences of data corresponding to basic data pattern 1 is 100, and the frequency of occurrence of data corresponding to basic data pattern 1 is 100%. Each of the 100 customer codes includes data corresponding to Basic Data Pattern 2. For example, "KH000001" includes the data "KH" corresponding to Basic Data Pattern 2, and "AH100001" includes the data "AH" corresponding to Basic Data Pattern 2. Therefore, the number of occurrences of data corresponding to Basic Data Pattern 2 is 100, and the frequency of occurrence of data corresponding to Basic Data Pattern 2 is 100%. Each of the 100 customer codes also includes data corresponding to Basic Data Pattern 3. For example, "KH000001" includes the data "000001" corresponding to Basic Data Pattern 3, and "AH100001" includes the data "100001" corresponding to Basic Data Pattern 3. Therefore, the number of occurrences of data corresponding to Basic Data Pattern 3 is 100, and the frequency of occurrence of data corresponding to Basic Data Pattern 3 is 100%. Assuming that the data corresponding to prefix data pattern 1 includes 80 customer codes, the number of occurrences of the data corresponding to prefix data pattern 1 is 80, and the frequency of occurrence of the data corresponding to prefix data pattern 1 is 80%; the data corresponding to prefix data pattern 2 includes 20 customer codes, so the number of occurrences of the data corresponding to prefix data pattern 2 is 20%, and the frequency of occurrence of the data corresponding to prefix data pattern 2 is 20%.

[0131] Second, the at least one data feature includes the distribution characteristics and / or statistical characteristics of the data in the first data set.

[0132] In some embodiments, the following operations 4031-4032 are used to obtain at least one data feature of the first data set, namely, to obtain the distribution features and / or statistical features of the data in the first data set.

[0133] 4031: Based on at least one data pattern of the first data set, obtain the type of data in the first data set.

[0134] The first data set will still be referred to as the first data, and the type of the first data includes one or more of the following: the basic type or the technical type of the first data.

[0135] The basic type of the first data describes its data components. The basic type of the first data can be integer, floating-point, character, or boolean, etc. The technical type of the first data describes the functionality implemented by the first data. The technical type of the first data can be code type, encoding type, flag type, category type, description type, or measurement type.

[0136] This code type indicates that the first piece of data is used to identify a certain object. For example, the first piece of data might be the area code of a region. The area code is a code type of data used to indicate that region. For example, the area code for Beijing is "010", which indicates Beijing, so "010" is a code type of data.

[0137] This encoding type indicates that the first data is the number of a certain object, such as the ID column. 00001 is used to represent the data in the first row, 00002 is used to represent the data in the second row, and so on. 00001 is the unique number of the first row, and 00002 is the unique number of the second row.

[0138] The first data for this label type is binary classification data, including "true", "false", "yes" or "no", etc.

[0139] The first data of this description type is used to describe an object. For example, the data listed above, "a large enterprise established in 1991 specializing in patent business", is data of the description type used to describe enterprise 1.

[0140] The first data in a metric class is used to measure an object. For example, the first data in a metric class might be height or weight, used to measure a person's body.

[0141] In some embodiments, the type of the first data further includes a service type, which describes the service to which the first data belongs. The service type of the first data may be a date type, region, IP address, or identifier, etc. The identifier may be a universally unique identifier (UUID), etc.

[0142] See Figure 7 In 4031, the type of the first data is obtained through the following operations 1-10, which are as follows.

[0143] 1. Based on at least one data pattern, determine whether the type of the first data is date, boolean, or other types, where other types are types other than date and boolean.

[0144] The at least one data pattern includes the basic data pattern corresponding to the first data and / or the basic data pattern corresponding to each sub-data in the first data. The basic data pattern corresponding to date type data and the basic data pattern corresponding to boolean type data are both fixed specified patterns. Therefore, based on at least one data pattern, it can be determined whether the type of the first data is date type, boolean type, or other type.

[0145] Boolean type is a type of flag type; that is, Boolean type is a flag type. In other words, if the first data is of Boolean type, the type of the first data is determined to be a flag type.

[0146] If the type of the first data is other than other types, the specific type of the first data will be determined by the following steps 2-10.

[0147] 2. If the type of the first data is other types, determine the structure of the first data based on the basic data pattern of the first data.

[0148] The first data may be null, and may be a mixed structure of English letters and numbers, an integer structure, a text structure, or a floating-point structure, etc.

[0149] 3. If the structure of the first data is a mixed structure of English letters and numbers, determine the length of the number part in the first data. If the length exceeds N, determine that the type of the first data is an encoding type and end. If the length does not exceed N, determine that the type of the first data is a code type and end.

[0150] N is the first threshold, where N is an integer greater than 1, such as N = 4, 5, or 6. If the length of the numeric portion in the first data exceeds N bits, the type of the first data is determined to be an encoding type; if the length of the numeric portion in the first data does not exceed N bits, the type of the first data is determined to be a code type.

[0151] 4. If the first data is an integer structure, determine the length of the first data. If the length of the first data is 1 digit, perform operation 5. If the length of the first data is more than 1 digit, perform operation 6.

[0152] The first data is an integer structure, which determines that the type of the first data is integer.

[0153] 5. If the first data is 0 or 1, then the type of the first data is determined to be a flag type, and the process ends; if the first data is any value from 2 to 9, then the type of the first data is determined to be a code type, and the process ends.

[0154] If the first data is 0 or 1, that is, 1 or 0 may be used to represent yes or no, or 1 or 0 may be used to represent true or false, so if the first data is 0 or 1, then the type of the first data is determined to be a flag type.

[0155] 6. Obtain the frequency of occurrence of the first data in the first data set. If the frequency of occurrence of the first data exceeds the second threshold and the length of the first data does not exceed M, then determine the type of the first data as code type, where M is the third threshold and M is an integer greater than 1. If the frequency of occurrence of the first data does not exceed the second threshold or the length of the first data exceeds M, execute operation 7.

[0156] Alternatively, M = 3, 4, or 5, etc.

[0157] If the frequency of the first data exceeds the second threshold and the length of the first data does not exceed M, it indicates that the first data is frequently occurring and relatively short, so the type of the first data is determined to be a code type. For example, telephone area codes are often frequently occurring and relatively short, so telephone area codes are code types.

[0158] 7. Based on the business attribute description information corresponding to the first data set, determine the type of the first data, and then end.

[0159] If the first data set is a single column of data, then the business attribute description information corresponding to the first data set is the column name and / or the table name of the database table containing that column of data. The business attribute description information corresponding to the first data set is read from the structured data file used to store the first data set.

[0160] If the first data set is data belonging to the same business in a semi-structured data file, the semi-structured data file includes a tag block for storing the first data set. The start tag and / or end tag of the tag block include business attribute description information corresponding to the first data set. The business attribute description information corresponding to the first data set is read from the start tag or end tag of the tag block in the semi-structured data file.

[0161] In operation 7, if the business attribute description information corresponding to the first data set includes the specified keyword, then the type of the first data set is determined to be a measurement type; if the business attribute description information corresponding to the first data set does not include the specified keyword, then the type of the first data set is determined to be an encoding type. Optionally, the specified keyword is the name of the data used to measure the object, including height, weight, and length, etc.

[0162] 8. If the structure of the first data is a floating-point number structure, then determine the type of the first data as a metric class, and end.

[0163] 9. If the structure of the first data is text, obtain the frequency of occurrence of the first data. If the frequency of occurrence of the first data does not exceed Q, where Q is the fourth threshold and Q is an integer greater than 0, then determine that the type of the first data is a description type. If the frequency of occurrence of the first data exceeds Q, then execute operation 10.

[0164] Q = 1, 2, or 3, etc.

[0165] If the frequency of the first data does not exceed Q, it means that the frequency of the first data is low. Since the first data is a text structure, it is determined that the first data is descriptive information describing a certain object, that is, the type of the first data is a description type.

[0166] If the frequency of the first data exceeds Q, then the high frequency of the first data indicates that the first data may be a flag type (such as yes or no, true or false, etc.), or it may indicate that the first data may be a code type.

[0167] 10. Determine whether the data range corresponding to the binary classification data includes the first data. If the data range includes the first data, determine that the type of the first data is the flag type. If it does not include the first data, determine that the type of the first data is the code type.

[0168] The data range corresponding to binary classification data includes yes, no, true, false, not, correct, incorrect, etc.

[0169] In some embodiments, after obtaining the type of the first data, the first data and its type are combined to form a training sample. Multiple different training samples can be obtained by following steps 1-10 above, or multiple training samples can be manually set. These multiple training samples are then used to train the intelligent algorithm to obtain a type recognition model. Thus, when it is necessary to obtain the type of data, the data is input into the type recognition model, which then identifies the type of the data, and the type of data output by the type recognition model is obtained.

[0170] 4032: Based on the type of data in the first data set, obtain at least one data feature of the first data set.

[0171] In 4032, data belonging to a first type are obtained from a first data set. Based on these data belonging to the first type, statistical characteristics of the data in the first data set are obtained. These statistical characteristics include the maximum, minimum, average, deviation, variance, median, percentile, and / or standard deviation of the data belonging to the first type. The first type includes integer, floating-point, measure type, and / or encoding type. At least one data feature includes the statistical characteristics of the data in the first data set. And / or,

[0172] Obtain each data belonging to the second type from the first data set, and obtain the distribution characteristics of the data in the first data set based on each data belonging to the second type. The distribution characteristics include the number of occurrences and / or frequency of occurrence of each data belonging to the second type. The second type includes flag type, Boolean type, category type and / or code type. The at least one data feature includes the distribution characteristics of the data in the first data set.

[0173] In some embodiments, the first data set may include data with a mixed structure of English letters and numbers. For ease of explanation, this data is referred to as the second data. That is, the data structure of the second data is a mixed structure of English letters and numbers. Each data belonging to the first type includes the number part of the second data, and each data belonging to the second type includes the English part of the second data.

[0174] For example, assuming the second data is "KH000001", then each data belonging to the first type includes the numeric part "000001" in the second data, and each data belonging to the second type includes the English part "KH" in the second data.

[0175] Third, the at least one data feature includes at least one word segmentation, which is obtained by segmenting the third data and removing stop words. The third data includes business attribute description information corresponding to the first data set and / or description type data in the first data set.

[0176] In some embodiments, the third data is segmented to obtain multiple segments, and stop words belonging to the stop word list are removed from the multiple segments. The remaining segments are used as data features of the first data set.

[0177] For stop words, the data of a determined text type can be segmented, the frequency of each segment can be counted, and the segments with a frequency exceeding a frequency threshold can be selected to obtain a stop word list. Optionally, the selected segments can be displayed so that technicians can filter out non-stop words and compile the remaining segments into a stop word list.

[0178] Fourth, the at least one data feature includes statistical features corresponding to the first language, which are used to reflect the statistical situation of the data corresponding to the first language. Optionally, the statistical features include the number of occurrences of data corresponding to the first language and / or the frequency of occurrence of data, and the language to which the data of the descriptive type in the first dataset belongs includes the first language.

[0179] For any data description type in the first dataset, the data includes English letters, Chinese characters, and / or Japanese characters, meaning the data includes text from one or more languages. For example, for the data listed above, "a leading global ICT infrastructure and smart device provider," the data includes English letters and Chinese characters, corresponding to both English and Chinese languages; that is, the data belongs to languages ​​including English and Chinese.

[0180] In the first dataset, identify the language or one language to which each description type of data belongs, thus obtaining the language to which the description type of data in the first dataset belongs. For any given language, for ease of explanation, this language is referred to as the first language. Obtain the data corresponding to the first language from the first dataset, count the number of data obtained, and obtain the frequency of occurrence of the data corresponding to the first language. And / or, based on the count and the total number of data included in the first dataset, calculate the frequency of occurrence of the data corresponding to the first language.

[0181] In summary, at least one data feature of the first dataset includes statistical features corresponding to the data patterns of the first dataset (the number of data occurrences and / or the frequency of data occurrences corresponding to the data patterns), distribution features of the data in the first dataset, statistical features of the data in the first dataset, at least one word segmentation, and / or statistical features corresponding to the language in the first dataset (the number of data occurrences and / or the frequency of data occurrences corresponding to the language), etc.

[0182] Step 404: Based on the at least one data feature, determine the connected graph to which the first data set belongs, where each node in the connected graph is a different data set, and the similarity between the neighboring nodes of the first data set and the first data set exceeds a specified threshold.

[0183] If the similarity between any two adjacent nodes in the connected graph exceeds a specified threshold, then there is an edge connecting the two adjacent nodes.

[0184] In some embodiments, the connected graph to which the first data set belongs is determined based on the at least one data feature and the business type of the data in the first data set.

[0185] In step 404, the connected graph to which the first data set belongs is determined by the following operations 4041-4044, which are respectively:

[0186] 4041, Obtain at least one feature set of the first data set, each feature set including data features of the same feature type.

[0187] For example, a feature set can be formed by combining the statistical features of the data in the first data set, the statistical features corresponding to the data patterns in the first data set, and / or the statistical features corresponding to the language in the first data set; a feature set can be formed by combining the distribution features of the data in the first data set; and / or, a feature set can be formed by combining at least one of the above word segments.

[0188] In some embodiments, the business types of the data in the first data set are also grouped into a feature set, thereby enabling the determination of the connected graph to which the first data set belongs based on the at least one data feature and the business types of the data in the first data set.

[0189] In some embodiments, each feature set is a feature vector or feature matrix, and each feature set corresponds to a different feature type. For example, the statistical features of the data in the first dataset are used as the first row of a matrix, the statistical features corresponding to the data patterns in the first dataset are used as the second row, and the statistical features corresponding to the language in the first dataset are used as the third row, resulting in a three-row feature matrix. As another example, the distribution features of the data in the first dataset can be combined into a feature vector, and / or at least one of the aforementioned word segments can be combined into a feature vector, resulting in two different feature sets.

[0190] In this embodiment, there may be at least one connected graph, or there may be no connected graph. If there is no connected graph, the first data set is directly treated as a node in the connected graph, and the process restarts from step 401, i.e., other data sets are obtained, and processing of these other data sets begins again from step 401 to obtain at least one feature set. This other data set is then used as the second data set. If there is at least one connected graph, each node in the connected graph is a data set. For example, referring to 5, the management device already has three connected graphs, namely connected... Figure 1 , connectivity Figure 2 and connectivity Figure 3 Each node in these three connected graphs represents a different data set. For any data set in a connected graph, for ease of explanation, this data set is referred to as the second data set. Before adding the second data set to the connected graph, at least one feature set of the second data set has been obtained according to steps 401-404 above. The second data set is different from the first data set. The first data set and the second data set may be two different data sets in the same storage file, or they may be data sets in two different storage files.

[0191] The second dataset may be associated with a data standard, or it may not be associated with a data standard. In other words, for any connected graph, any dataset in that graph may or may not be associated with a data standard.

[0192] 4042: Based on at least one feature set of a first data set and at least one feature set of a second data set, obtain the similarity between the first data set and the second data set.

[0193] In step 4042, a feature set is selected from at least one feature set in the first data set, and a feature set is selected from at least one feature set in the second data set. The two selected feature sets are of the same feature type. Based on these two feature sets, the similarity corresponding to the feature type is obtained using a similarity algorithm. Following this method, the similarity corresponding to at least one feature type can be obtained. Based on the similarity corresponding to each feature type and the weight corresponding to each feature type, the similarity between the first data set and the second data set is obtained.

[0194] See Figure 5 For example, the feature types corresponding to the two datasets are statistical features of the data in the two datasets, and the similarity algorithms used include K-means (a distance algorithm based on Euclidean distance) or K-means+ clustering algorithms. Another example is that the feature types corresponding to the two datasets are distribution features of the data in the two datasets, distribution features corresponding to data patterns, and / or distribution features corresponding to language, and the similarity algorithms used include Mahalanobis distance or Jaccard distance. Yet another example is that the feature types corresponding to the two datasets are word segmentation and / or business types of the data, and the similarity algorithms used include edit distance or fuzzy matching text similarity algorithms.

[0195] In some embodiments, based on a thesaurus, it is determined whether there are synonyms in the word segments included in the business attribute description information corresponding to the first data set and the word segments included in the business attribute description information corresponding to the second data set. If synonyms exist, the similarity between the first data set and the second data set is increased based on the number of synonyms.

[0196] In some embodiments, based on an antonym list, it is determined whether there are antonyms in the word segments included in the business attribute description information corresponding to the first data set and the word segments included in the business attribute description information corresponding to the second data set. If antonyms exist, the similarity between the first data set and the second data set is reduced based on the number of antonyms.

[0197] The synonym and antonym lists are pre-defined word lists.

[0198] By repeating the process described in 4041-4042, the similarity between the first data set and other data sets in the at least one connected graph can also be obtained.

[0199] 4043: Identify at least one target node in the at least one connected graph, wherein the similarity between the first data set and each target node exceeds a specified similarity threshold.

[0200] In 4043, at least one node in the at least one connected graph is selected whose similarity to the first data set exceeds a specified similarity threshold, and the at least one node is the target node.

[0201] 4044: Connect the nodes corresponding to the first data set with each target node to determine the connected graph in which the nodes corresponding to the first data set are located.

[0202] Connect the node corresponding to the first data set to each target node to add the node corresponding to the first data set to a certain connected graph.

[0203] For example, see Figure 5 Assuming in connectivity Figure 1 , connectivity Figure 2 and connectivity Figure 3 Select at least one node whose similarity to the first dataset exceeds a specified similarity threshold, wherein the at least one node is connected. Figure 2 The three nodes in the diagram are node1, node2, and node3. See also... Figure 8 Connect the node corresponding to the first data set to these three nodes, thereby adding the node corresponding to the first data set to the connected nodes. Figure 2 middle.

[0204] In some embodiments, the connected graph containing the node corresponding to the first data set and the at least one target node are displayed.

[0205] After display, the technician may adjust the connection relationship between the node corresponding to the first data set and one or more target nodes in the connected graph. Based on the target node adjusted by the technician and the original target node, the values ​​of the first, second, third, and / or fourth thresholds are adjusted.

[0206] For example, display Figure 8 The connection shown Figure 2 See also Figure 9Technicians can cancel the connection between the node corresponding to the first data set and node 3, and connect the node corresponding to the first data set and node 4. Based on the adjusted nodes 3 and 4, and the nodes 1, 2, and 3 before the adjustment, the management device adjusts the values ​​of the first threshold, second threshold, third threshold, and / or fourth threshold.

[0207] Step 405: Based on the data set included in the connected graph, obtain the first data standard and associate the first data standard with the first data set.

[0208] In some embodiments, the first data standard is displayed before associating the first data standard and the first data set. The first data standard and the first data set are associated upon receiving a command triggered by a technician confirming the association of the first data standard and the first data set.

[0209] For example, see Figure 6 The management device displays the data standard, recommending the data standard with standard number DB0001 to the technician. The technician triggers a command to confirm the association of the first data standard and the first data set by clicking the confirmation button. Upon receiving this command, the management device will associate the data standard with the first data set (i.e., associate the data standard DB0001 with the column containing the customer code). Optionally, the technician can also modify the displayed data standard before clicking the confirmation button.

[0210] Each data set in the connected graph may not be associated with a data standard, or some data sets in the connected graph may be associated with a data standard.

[0211] In the case where some data sets in the connected graph are already associated with data standards, in step 405, at least one data standard is obtained, which includes data standards associated with other data sets in the connected graph besides the first data set; based on the association frequency of each data standard in the at least one data standard, a data standard is selected from the at least one data standard as the first data standard.

[0212] In some embodiments, one or more data sets in the connectivity graph are associated with a data standard, and the association frequency of the data standard includes the number of data sets associated with the data standard. The data standard with the highest association frequency can be selected as the first data standard, and the first data standard is associated with the first data set.

[0213] Data sets within the same connected graph are similar, indicating that at least one data criterion is a duplicate. To avoid duplicate data criterions, deduplication can be performed on this at least one data criterion. In implementation,

[0214] The data standard associated with each data set in the connected graph except for the first data set is updated to the first data standard. This associates each data set in the connected graph with the first data standard, thereby removing duplicate and redundant data standards.

[0215] See Figure 10 The management device can also display the data sets in the connectivity graph and the numbers of the data standards associated with the data sets in the connectivity graph. For the first data standard with the highest association frequency, the management device uses a marker to distinguish the first data standard from other data standards, so that technicians can know which data standard is the first data standard and which data standards are duplicates of the first data standard.

[0216] In some embodiments, for data sets in the connected graph that are not associated with a data standard, the data set is also associated with the first data standard.

[0217] In some embodiments, where no data set in the connected graph is associated with a data standard, in step 405, a first data standard is generated based on at least one data pattern and / or at least one data feature of each data set in the connected graph, and each data set in the connected graph is associated with the first data standard.

[0218] See Figure 5 In implementation, a first standard content is generated based on the distribution characteristics of the data in each data set of the connected graph; a second standard content is generated based on at least one data pattern of each data set of the connected graph; and / or a third standard content is generated based on the distribution characteristics and / or statistical characteristics of the data in each data set of the connected graph, thereby obtaining the first data standard.

[0219] In some embodiments, for the first standard content, a first frequency mean is calculated based on the frequency of occurrence of null values ​​in each dataset. When the first frequency mean exceeds a first frequency threshold, the first standard content is determined to indicate that null values ​​are allowed to be input into the first dataset; when the first frequency mean does not exceed the first frequency threshold, the first standard content is determined to indicate that null values ​​are not allowed to be input into the first dataset.

[0220] In some embodiments, at least one data pattern in the first data set includes a basic data pattern corresponding to each data in the first data set and / or one or more prefix data patterns corresponding to the first data set. Thus, for any basic data pattern, based on the data occurrence frequency corresponding to that basic data pattern in each data set, the average data occurrence frequency corresponding to that basic data pattern is calculated, thereby obtaining the average data occurrence frequency corresponding to at least one basic data pattern. For any prefix data pattern, based on the data occurrence frequency corresponding to that prefix data pattern in each data set, the average data occurrence frequency corresponding to that prefix data pattern is calculated, thereby obtaining the average data occurrence frequency corresponding to at least one prefix data pattern. Basic data patterns whose average data occurrence frequency exceeds a second frequency threshold are selected as general patterns in the second standard content, and prefix data patterns whose average data occurrence frequency exceeds a third frequency threshold are selected as prefix data patterns in the second standard content.

[0221] In some embodiments, the data type is determined based on the infrastructure indicated by the selected underlying data pattern, and the first data standard further includes the data type. For example, when the infrastructure indicated by the underlying data pattern is a mixed alphanumeric structure or a text structure, the determined data type is character. As another example, when the infrastructure indicated by the underlying data pattern is an integer structure, the determined data type is integer. Yet another example, when the infrastructure indicated by the underlying data pattern is a floating-point structure, the determined data type is floating-point.

[0222] In some embodiments, the data length is determined based on a general pattern (the selected basic data pattern) in the second standard content, and the first standard content also includes this data length. For example, if the selected basic data pattern is {uppercase English}[2 digits]{numbers}[6 digits], then the determined data length is 8 digits.

[0223] For the third standard content, select the largest maximum value from the maximum values ​​of the data in each data set as the upper limit of the value range, and select the smallest minimum value from the minimum values ​​of the data in each data set as the lower limit of the value range, thus obtaining the value range in the third standard content.

[0224] The aforementioned first, second, and third standard contents may also include other contents, which can be obtained based on the data patterns and data characteristics of each data set in the connected graph.

[0225] In some embodiments, for the standard name of the first data standard, the business attribute description information corresponding to each data set in the connectivity graph includes multiple word segments. For each word segment, the average occurrence frequency of the word is calculated based on the occurrence frequency of the word corresponding to each data set. One or more words with the highest average occurrence frequency are selected from the multiple word segments as the standard name of the first data standard.

[0226] In some embodiments, the data sets in the connected graph include floating-point numbers. The decimal places of each floating-point number are counted, and the longest decimal place is selected from the counted decimal places. This longest decimal place is used as the data precision included in the first data standard. For example, assuming the longest decimal place is 2, it means that the decimal places allowed to be stored in the first data set cannot exceed 2. If the data sets in the connected graph do not include floating-point numbers, then the data precision included in the first data standard is zero.

[0227] In step 401, the management device obtains the storage file from the server. Optionally, after obtaining the first data standard, the management device sends association information to the server. This association information includes the file identifier of the storage file, the set identifier of the first data set, and the first data standard. The server receives the association information, associates the first data standard with the first data set in the storage file based on the association information, and then uses the first data standard to apply normative constraints to the data to be stored in the first data set included in the storage file.

[0228] For example, the management device retrieves a database table as shown in Table 1 from the data lake. The management device uses the first column of data in this database table (the first column being the customer code) as a first data set. After associating the first data set with a first data standard, the management device sends association information to the data lake. This association information includes the file identifier of the stored file (e.g., the filename and / or storage path), the set identifier of the first data set (e.g., the column name and / or column number), and the first data standard. The data lake receives this association information and, based on it, associates the first column of data with the first data standard in the database table shown in Table 1.

[0229] When the data source receives a customer code that needs to be stored in the database table shown in Table 1, the data lake uses a first data standard to check whether the customer code meets the normative constraints of the first data standard. If the customer code meets the normative constraints of the first data standard, the data lake saves the customer code to the first column of the database table shown in Table 1. If the customer code does not meet the normative constraints of the first data standard, the data lake discards the customer code or prompts the user to modify the customer code.

[0230] In some embodiments, after associating the first data set with the first data standard, the server detects each data in the first data set based on the first data standard. If data that does not conform to the specification indicated by the first data standard is detected, the detected data is cleared or the detected data is transformed so that the transformed data conforms to the specification indicated by the first data standard.

[0231] In this embodiment, at least one data pattern of a first data set is obtained, and at least one data feature of the first data set is obtained based on the at least one data pattern. The at least one data feature includes the distribution characteristics and / or statistical characteristics of the data in the first data set and the distribution characteristics corresponding to the data patterns of the first data set, thus enriching the content of the at least one data feature. This allows for the automatic determination of the connected graph in which the first data set resides. Since the data sets in the connected graph are similar, there are associations between data sets and data standards within the connected graph. At least one data standard associated with the data sets in the connected graph is obtained; this at least one data standard is a similar data standard. Based on the association frequency of each data standard, one data standard is selected as the first data standard, improving the accuracy of obtaining the first data standard. Each data set in the connected graph is associated with the first data standard, and duplicate data standards are removed. Since there are associations between data sets and data standards within the connected graph, and since the data sets in the connected graph are similar, the first data standard is automatically generated based on the data patterns and data features of each data set in the connected graph, improving the accuracy and efficiency of generating the first data standard.

[0232] See Figure 11 This application provides a data governance apparatus 1100, which can be deployed on the management device in any of the above embodiments. For example, the apparatus 1100 is deployed on... Figure 1 or Figure 2 On the management device 101 in the network architecture 100 shown, or deployed on Figure 4 The management device of the method 400 shown. The device 1100 includes.

[0233] The file reading module 1101 is used to read a storage file from the database server, the storage file being used to store data belonging to at least one business function;

[0234] Pattern acquisition module 1102 is used to acquire at least one data pattern of a first data set, the first data set including data belonging to the same business stored in the storage file, and the at least one data pattern is used to indicate the structure of each data included in the first data set;

[0235] The feature acquisition module 1103 is used to acquire at least one data feature of the first data set based on the at least one data pattern;

[0236] The standard acquisition module 1104 is further configured to acquire a first data standard based on the at least one data feature, wherein the first data standard is used to constrain the standardization of each data included in the first data set.

[0237] Optionally, for the detailed implementation process of the file reading module 1101 reading the stored file, please refer to [link / reference]. Figure 4 The relevant content in step 401 of method 400 shown will not be described in detail here.

[0238] Optionally, for the detailed implementation process of the pattern acquisition module 1102 acquiring at least one data pattern, see [link to documentation]. Figure 4 The relevant content in step 402 of method 400 shown will not be described in detail here.

[0239] Optionally, for a detailed implementation process of feature acquisition module 1103 acquiring at least one data feature, see [link to documentation]. Figure 4 The relevant content in step 403 of method 400 shown will not be described in detail here.

[0240] Optionally, for the detailed implementation process of the standard acquisition module 1104 acquiring the first data standard, please refer to [link / reference]. Figure 4 The relevant content in steps 404-405 of method 400 shown will not be described in detail here.

[0241] Optionally, the storage file is a structured data file, which stores data belonging to at least one business in a list format, with data belonging to the same business stored in the same column of the structured data file.

[0242] Optionally, the storage file is a semi-structured data file. The semi-structured data file uses tag blocks to store data belonging to at least one business. Data belonging to the same business is stored in the same tag block of the semi-structured data file.

[0243] Optionally, the at least one data pattern includes a basic data pattern, which is used to indicate the basic structure of each data included in the first data set. The basic structure includes one or more of the following: a mixed alphanumeric structure, an integer structure, a floating-point structure, a Boolean structure, an address structure, an identifier structure, or a date structure.

[0244] Optionally, the pattern acquisition module 1102 is used to identify each data in the first data set based on at least one specified regular expression to obtain a basic data pattern.

[0245] Optionally, for the detailed implementation process of the pattern acquisition module 1102 acquiring the basic data pattern, please refer to [link / reference]. Figure 4 The relevant content in step 402 of method 400 shown will not be described in detail here.

[0246] Optionally, the at least one data pattern further includes a prefix data pattern, which is used to indicate multiple data in the first data set that include a first prefix, the first prefix being the longest common prefix of the multiple data, and the basic data pattern corresponding to the portion of each of the multiple data except for the first prefix being the same.

[0247] Optionally, the feature acquisition module 1103 is used for:

[0248] From a first data set, obtain each data corresponding to a first data pattern, wherein the at least one data pattern includes the first data pattern;

[0249] Based on the acquired data, the number of data occurrences and / or the frequency of data occurrences corresponding to the first data pattern are obtained, and the at least one data feature includes the number of data occurrences and / or the frequency of data occurrences corresponding to the first data pattern.

[0250] Optionally, the feature acquisition module 1103 acquires the data occurrence quantity and / or data occurrence frequency corresponding to the first data pattern. For a detailed implementation process, please refer to [link to documentation]. Figure 4 The relevant content in step 403 of method 400 shown will not be described in detail here.

[0251] Optionally, the feature acquisition module 1103 is used for:

[0252] Based on at least one data pattern, obtain the data type in the first data set;

[0253] Based on the type of data in the first data set, obtain at least one data feature of the first data set.

[0254] Optionally, for the detailed implementation process of feature acquisition module 1103 acquiring the data type in the first data set, see [link to documentation]. Figure 4 The relevant content in step 4031 of method 400 shown will not be described in detail here.

[0255] Optionally, for a detailed implementation process of feature acquisition module 1103 acquiring at least one data feature, see [link to documentation]. Figure 4 The relevant content in step 4032 of method 400 shown will not be described in detail here.

[0256] Optionally, the first data set includes first data, and the type of the first data includes one or more of the following: a basic type or a technical type of the first data, wherein the basic type is used to describe the data components of the first data, and the technical type is used to describe the functions implemented by the first data.

[0257] Optionally, the basic type of the first data is an integer, floating-point number, or Boolean, and the technical type of the first data is a code type, encoding type, flag type, category type, description type, or measurement type.

[0258] Optionally, the feature acquisition module 1103 is used for:

[0259] Statistical features are obtained based on the data belonging to a first type included in the first dataset. These statistical features include the maximum, minimum, average, deviation, variance, median, percentile, and / or standard deviation of the data belonging to the first type. The first type includes integer, floating-point, measure type, and / or encoding type. At least one data feature includes this statistical feature; and / or...

[0260] Distribution characteristics are obtained based on the data belonging to the second type in the first dataset. The distribution characteristics include the number of occurrences and / or frequency of occurrence of each data belonging to the second type. The second type includes flag type, Boolean type, category type and / or code type. The at least one data feature includes the distribution characteristics.

[0261] Optionally, for the detailed implementation process of the feature acquisition module 1103 acquiring statistical features, please refer to [link / reference]. Figure 4 The relevant content in step 4032 of method 400 shown will not be described in detail here.

[0262] Optionally, for the detailed implementation process of the feature acquisition module 1103 acquiring distribution features, please refer to [link to documentation]. Figure 4 The relevant content in step 4032 of method 400 shown will not be described in detail here.

[0263] Optionally, the first data set includes the second data, the data structure of the second data is a mixed alphanumeric structure, each data belonging to the first type includes the numeric part of the second data, and each data belonging to the second type includes the alphanumeric part of the second data.

[0264] Optionally, the data features of the first data set include at least one word segmentation, which is obtained by segmenting the third data and removing stop words. The third data includes business attribute description information corresponding to the first data set and / or description type data in the first data set.

[0265] Optionally, the data characteristics of the first dataset include the number of data occurrences and / or the frequency of data occurrences corresponding to the first language, and the language to which the data of the description type in the first dataset belongs includes the first language.

[0266] Optionally, the standard acquisition module 1104 is used for:

[0267] Based on at least one data feature, determine the connected graph to which the first data set belongs, where each node in the connected graph is a different data set, and the similarity between the neighboring nodes of the first data set and the first data set exceeds a specified threshold.

[0268] Based on the data set included in the connected graph, the first data standard is obtained.

[0269] Optionally, the standard acquisition module 1104 determines the detailed implementation process of the connected graph, see [link to documentation]. Figure 4 The relevant content in step 404 of method 400 shown will not be described in detail here.

[0270] Optionally, for the detailed implementation process of the standard acquisition module 1104 acquiring the first data standard, please refer to [link / reference]. Figure 4 The relevant content in step 405 of method 400 shown will not be described in detail here.

[0271] Optionally, the standard acquisition module 1104 is used to determine the connected graph to which the first data set belongs based on at least one data feature and the business type of the data in the first data set, wherein the business type of the data is date, region, address or identifier.

[0272] Optionally, the standard acquisition module 1104 is used for:

[0273] Obtain at least one data standard, which includes data standards associated with other data sets in the connected graph besides the first data set;

[0274] Based on the correlation frequency of each data standard in at least one data standard, select one data standard as the first data standard from at least one data standard.

[0275] Optionally, the detailed implementation process of the standard acquisition module 1104 acquiring at least one data standard and selecting a first data standard from the at least one data standard can be found in [reference needed]. Figure 4 The relevant content in step 405 of method 400 shown will not be described in detail here.

[0276] Optionally, the standard acquisition module 1104 is used to update the data standard associated with the other data set to the first data standard.

[0277] Optionally, the standard acquisition module 1104 is used to generate a first data standard based on at least one data pattern and / or at least one data feature of each data set in the connected graph.

[0278] Optionally, for a detailed implementation process of the standard acquisition module 1104 generating the first data standard, see [link to documentation]. Figure 4 The relevant content in step 405 of method 400 shown will not be described in detail here.

[0279] Optionally, the standardization includes data integrity, data consistency, and / or data accuracy. The first data standard includes first standard content, second standard content, and / or third standard content. The first standard content is used to constrain the data integrity of the first data set, the second standard content is used to constrain the data consistency of the first data set, and the third standard content is used to constrain the data accuracy of the first data set.

[0280] Optionally, the standard acquisition module 1104 is used for:

[0281] Based on the distribution characteristics of the data in each dataset, generate first standard content; and / or,

[0282] Based on at least one data pattern for each dataset, generate second standard content; and / or,

[0283] Third standard content is generated based on the distribution and / or statistical characteristics of the data in each dataset.

[0284] In this embodiment, the acquisition unit acquires at least one data pattern of a first data set, which includes data belonging to the same business. The at least one data pattern indicates the structure of each piece of data included in the first data set. The processing unit acquires a first data standard based on the at least one data pattern. The first data standard constrains the standardization of each piece of data included in the first data set. Thus, the acquisition unit acquires at least one data pattern of the first data set, and the processing unit automatically acquires the first data standard associated with the first data set based on the at least one data pattern, thereby improving the efficiency and accuracy of acquiring the data standard.

[0285] See Figure 12 This application provides a schematic diagram of a computer device 1200. The computer device 1200 can be any of the management devices described in the above embodiments. For example, the device 1200 can be one of the aforementioned... Figure 1 or Figure 2 The management device 101 in the network architecture 100 shown, or the one described above. Figure 4The management device in method 400 shown. The device 1200 includes at least one processor 1201, internal connections 1202, memory 1203, and at least one transceiver 1204.

[0286] The device 1200 is a hardware-structured device that can be used to implement... Figure 11 The functional modules in the device 1100. For example, those skilled in the art will conceive of them. Figure 11 The file reading module 1101 in the device 1100 shown, Figure 11 The file reading module 1101 in the illustrated device 1100 can be implemented by the at least one transceiver 1204. The pattern acquisition module 1102, feature acquisition module 1103, and standard acquisition module 1104 can be implemented by the at least one processor 1201 calling code in the memory 1203.

[0287] Optionally, the device 1200 can also be used to implement the function of managing the device in any of the above embodiments.

[0288] Optionally, the processor 1201 described above may be a general-purpose central processing unit (CPU), a network processor (NP), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present application.

[0289] The aforementioned internal connection 1202 may include a pathway for transmitting information between the aforementioned components. Optionally, the internal connection 1202 may be a single board or a bus, etc.

[0290] The transceiver 1204 described above is used for communicating with other devices or communication networks.

[0291] The aforementioned memory 1203 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or it may be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto. The memory may exist independently and be connected to the processor via a bus. The memory may also be integrated with the processor.

[0292] The memory 1203 stores the application code that executes the solution of this application, and its execution is controlled by the processor 1201. The processor 1201 executes the application code stored in the memory 1203, and cooperates with at least one transceiver 1204, thereby enabling the device 1200 to realize the functions of the method of this patent.

[0293] In a specific implementation, as one example, the processor 1201 may include one or more CPUs, for example... Figure 12 CPU0 and CPU1 in the CPU.

[0294] In a specific implementation, as one embodiment, the device 1200 may include multiple processors, for example... Figure 12 Processors 1201 and 1207 are mentioned. Each of these processors can be a single-core (single-CPU) processor or a multi-core (multi-CPU) processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0295] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0296] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method of data governance, characterized by, The method includes: Read a storage file from the database server, the storage file being used to store data belonging to at least one business function; Obtain at least one data pattern of a first data set, the first data set including data belonging to the same business stored in the storage file, the at least one data pattern being used to indicate the structure of each data included in the first data set; At least one data feature of the first data set is obtained based on the at least one data pattern, wherein the at least one data feature includes statistical features and / or distribution features; A first data standard is obtained based on the at least one data feature, and the first data standard is used to constrain the standardization of each data included in the first data set; The database server sends association information, which includes the file identifier of the stored file, the set identifier of the first data set, and the first data standard. The association information is used to enable the database server to associate the first data standard with the first data set and to use the first data standard to perform normative constraints on the data to be stored in the first data set.

2. The method of claim 1, wherein, The storage file is a structured data file, which stores data belonging to at least one business in a list format, with data belonging to the same business stored in the same column of the structured data file; or, The storage file is a semi-structured data file. The semi-structured data file stores data belonging to at least one business in the form of tag blocks. Data belonging to the same business is stored in the same tag block of the semi-structured data file.

3. The method of claim 1 or 2, wherein, The at least one data pattern includes a basic data pattern, which is used to indicate the basic structure of each data included in the first data set. The basic structure includes one or more of the following: a mixed alphanumeric structure, an integer structure, a floating-point number structure, a Boolean structure, an address structure, an identifier structure, or a date structure.

4. The method of claim 3, wherein, The acquisition of at least one data pattern from the first data set includes: Based on at least one specified regular expression, each piece of data in the first data set is identified to obtain the basic data pattern.

5. The method of claim 3, wherein, The at least one data pattern further includes a prefix data pattern, which is used to indicate multiple data in the first data set that include a first prefix, wherein the first prefix is ​​the longest common prefix of the multiple data, and the basic data pattern corresponding to the portion of each of the multiple data except for the first prefix is ​​the same.

6. The method of claim 1, 2, 4 or 5, wherein, The step of obtaining at least one data feature of the first data set based on the at least one data pattern includes: From the first data set, obtain each data corresponding to the first data pattern, wherein the at least one data pattern includes the first data pattern; Based on the acquired data, the number of data occurrences and / or the frequency of data occurrences corresponding to the first data pattern are obtained, and the at least one data feature includes the number of data occurrences and / or the frequency of data occurrences corresponding to the first data pattern.

7. The method of claim 1, 2, 4 or 5, wherein, The step of obtaining at least one data feature of the first data set based on the at least one data pattern includes: Based on the at least one data pattern, obtain the type of data in the first data set; Based on the type of data in the first data set, at least one data feature of the first data set is obtained.

8. The method of claim 7, wherein, The first data set includes first data, and the type of the first data includes one or more of the following: the basic type or the technical type of the first data, wherein the basic type is used to describe the data components of the first data, and the technical type is used to describe the functions implemented by the first data.

9. The method of claim 8, wherein, The basic type of the first data is integer, floating-point number or Boolean, and the technical type of the first data is code type, encoding type, flag type, category type, description type or measurement type.

10. The method of claim 7, wherein, The step of obtaining at least one data feature of the first data set based on the data type in the first data set includes: Statistical features are obtained based on the data belonging to the first type in the first dataset. These statistical features include the maximum, minimum, average, deviation, variance, median, percentile, and / or standard deviation of the data belonging to the first type. The first type includes integer, floating-point, measure type, and / or encoding type; and / or... Based on the data belonging to the second type included in the first data set, the distribution characteristics are obtained. The distribution characteristics include the number of occurrences and / or frequency of occurrence of each data belonging to the second type. The second type includes flag type, Boolean type, category type and / or code type.

11. The method of claim 10, wherein, The first data set includes the second data, the data structure of the second data is a mixed alphanumeric structure, each data belonging to the first type includes the numeric part of the second data, and each data belonging to the second type includes the alphanumeric part of the second data.

12. The method of claim 1, 2, 4, 5, 8, 9, 10, or 11, wherein, The data features of the first data set include at least one word segmentation, which is obtained by segmenting the third data and removing stop words. The third data includes business attribute description information corresponding to the first data set and / or description type data in the first data set.

13. The method of claim 1, 2, 4, 5, 8, 9, 10, or 11, wherein, The data characteristics of the first dataset include the number of data occurrences and / or the frequency of data occurrences corresponding to the first language, and the language to which the data of the description type in the first dataset belongs includes the first language.

14. The method of claims 1, 2, 4, 5, 8, 9, 10, or 11, wherein, The process of obtaining the first data standard based on the at least one data feature includes: Based on the at least one data feature, the connected graph to which the first data set belongs is determined, where each node in the connected graph is a different data set, and the similarity between the neighboring nodes of the first data set and the first data set exceeds a specified threshold. Based on the data set included in the connected graph, a first data standard is obtained.

15. The method as described in claim 14, characterized in that, Determining the connected graph to which the first data set belongs based on the at least one data feature includes: Based on the at least one data feature and the business type of the data in the first data set, the connected graph to which the first data set belongs is determined, and the business type of the data is date, region, address, or identifier.

16. The method as described in claim 14, characterized in that, The step of obtaining the first data standard based on the data set included in the connected graph includes: Obtain at least one data standard, the at least one data standard including data standards associated with other data sets in the connected graph besides the first data set; Based on the correlation frequency of each data standard in the at least one data standard, a data standard is selected from the at least one data standard as the first data standard.

17. The method as described in claim 16, characterized in that, The method further includes: Update the data standard associated with the other data sets to the first data standard.

18. The method as described in claim 14, characterized in that, The step of obtaining the first data standard based on the data set included in the connected graph includes: The first data standard is generated based on at least one data pattern and / or at least one data feature of each data set in the connected graph.

19. The method as described in claim 18, characterized in that, The standardization includes data integrity, data consistency, and / or data accuracy. The first data standard includes a first standard content, a second standard content, and / or a third standard content. The first standard content is used to constrain the data integrity of the first data set, the second standard content is used to constrain the data consistency of the first data set, and the third standard content is used to constrain the data accuracy of the first data set.

20. The method as described in claim 19, characterized in that, The generation of the first data standard based on at least one data pattern and / or at least one data feature of each data set in the connected graph includes: Based on the distribution characteristics of the data in each dataset, the first standard content is generated; and / or, Based on at least one data pattern of each of the data sets, the second standard content is generated; and / or, The third standard content is generated based on the distribution characteristics and / or statistical characteristics of the data in each dataset.

21. A data governance apparatus, characterized in that, The device includes: The file reading module is used to read storage files from the database server, the storage files being used to store data belonging to at least one business function; A pattern acquisition module is used to acquire at least one data pattern of a first data set, the first data set including data belonging to the same business stored in the storage file, and the at least one data pattern is used to indicate the structure of each data included in the first data set. The feature acquisition module is used to acquire at least one data feature of the first data set based on the at least one data pattern, wherein the at least one data feature includes statistical features and / or distribution features; A standard acquisition module is used to acquire a first data standard based on the at least one data feature, wherein the first data standard is used to constrain the standardization of each data included in the first data set; The device is further configured to send association information to the database server. The association information includes the file identifier of the storage file, the set identifier of the first data set, and the first data standard. The association information is used to enable the database server to associate the first data standard with the first data set in the storage file and to use the first data standard to perform normative constraints on the data to be stored in the first data set in the storage file.

22. The apparatus as claimed in claim 21, characterized in that, The storage file is a structured data file, which stores data belonging to at least one business in a list format, with data belonging to the same business stored in the same column of the structured data file; or, The storage file is a semi-structured data file. The semi-structured data file stores data belonging to at least one business in the form of tag blocks. Data belonging to the same business is stored in the same tag block of the semi-structured data file.

23. The apparatus as claimed in claim 21 or 22, characterized in that, The at least one data pattern includes a basic data pattern, which is used to indicate the basic structure of each data included in the first data set. The basic structure includes one or more of the following: a mixed alphanumeric structure, an integer structure, a floating-point number structure, a Boolean structure, an address structure, an identifier structure, or a date structure.

24. The apparatus as claimed in claim 21 or 22, characterized in that, The feature acquisition module is used for: From the first data set, obtain each data corresponding to the first data pattern, wherein the at least one data pattern includes the first data pattern; Based on the acquired data, the number of data occurrences and / or the frequency of data occurrences corresponding to the first data pattern are obtained, and the at least one data feature includes the number of data occurrences and / or the frequency of data occurrences corresponding to the first data pattern.

25. The apparatus as claimed in claim 21 or 22, characterized in that, The standard acquisition module is used for: Based on the at least one data feature, the connected graph to which the first data set belongs is determined, where each node in the connected graph is a different data set, and the similarity between the neighboring nodes of the first data set and the first data set exceeds a specified threshold. Based on the data set included in the connected graph, a first data standard is obtained.

26. A computer device, characterized in that, The method includes at least one processor, which is configured to be coupled to a memory, read and execute instructions in the memory to implement the method as claimed in any one of claims 1-20.

27. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a computer, it implements the method as described in any one of claims 1-20.

28. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, and the computer program is loaded by a processor to implement the method as described in any one of claims 1-20.

Images