A data standard generation method, storage medium and system based on similarity clustering and data exploration

Abstract

The present invention provides a data standard generation method, storage medium and system based on similarity clustering and data exploration. The method comprises the following steps: collecting and extracting multiple metadata to be standardized with the same data objects from multi-source heterogeneous databases; performing standardization preprocessing on the data information of each metadata to be standardized to obtain multiple metadata feature information; obtaining the data type of each metadata to be standardized based on the metadata feature information, performing similarity clustering on the metadata to be standardized with the same data type to generate multiple metadata similarity clusters; eliminating metadata similarity clusters with a metadata quantity less than a first preset value and a metadata occurrence frequency less than a second preset value, thereby generating optional data standards; and for each metadata similarity cluster retained after the elimination, using a preset data exploration statement to explore the field metadata in the metadata similarity cluster according to its data type to generate a corresponding data standard.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data standard generation method, storage medium and system based on similarity clustering and data exploration. Background Art

[0002] Currently, the process of generating data standards requires business and technical experts to first define the generally applicable data standards, then design a data model. The data model is then standardized using the data standards defined by these experts to achieve standardization and uniformity in the data model design. The designed data model is then used to generate reasonable data standards. This data standard generation approach is highly dependent on experts, making it difficult for small businesses that lack relevant experts to design data models and generate reasonable data standards. Summary of the Invention

[0003] The technical problem to be solved by the present invention is how to reduce the reliance on expert definitions in the process of generating reasonable data standards.

[0004] To solve the above technical problems, the present invention provides a data standard generation method based on similarity clustering and data exploration, comprising the following steps:

[0005] A. Collect multiple heterogeneous metadata with different structural types from multi-source heterogeneous databases, and extract multiple metadata to be standardized with the same data objects;

[0006] B. For each metadata to be standardized, obtain multiple pieces of data information, and perform standardization preprocessing on each piece of data information to obtain multiple metadata feature information;

[0007] C. obtaining the data type of each metadata to be standardized based on the metadata feature information of each metadata to be standardized, performing similarity clustering on metadata to be standardized with the same data type, and generating multiple metadata similarity clusters;

[0008] D. based on the number of metadata in each metadata similarity cluster and the number of times each metadata appears in the application system, eliminating metadata similarity clusters where the number of metadata is less than a first preset value and the number of times each metadata appears is less than a second preset value;

[0009] E. Obtain the data type of the metadata similar clusters that remain after elimination, and generate optional data standards based on the data;

[0010] F. For each metadata similarity cluster that remains after elimination, use a preset data exploration statement to explore the field metadata in the metadata similarity cluster according to its data type to obtain data features, and select and generate corresponding data standards from the optional data standards based on the data features.

[0011] Preferably, the data information of the metadata to be standardized includes English names, Chinese names and / or data types. In step B, the standardization preprocessing of the data information of the metadata to be standardized includes: stem extraction or morphological restoration of English names, synonym normalization expression of Chinese names, and / or standardized expression of data types.

[0012] Preferably, in step B, the data types of the metadata to be standardized include character class, numerical class and date class; in step E, optional text class standards and encoding class standards are generated based on the character class data type, optional encoding class standards and numerical class standards are generated based on the numerical class data type, and optional date class standards are generated based on the date class data type.

[0013] Preferably, in step F, the data exploration statements adopted according to the character data type include value grouping statistics and length grouping statistics, the data exploration statements adopted according to the numerical data type include decimal place judgment, value grouping statistics and quartile calculation range, and the data exploration statements adopted according to the date data type include using date regular expressions to identify date formats.

[0014] Preferably, in step C, the similarity of the metadata feature information of the metadata to be standardized with the same data type is calculated using a text similarity algorithm. If the similarity of the metadata feature information of at least two metadata to be standardized reaches a preset threshold, the at least two metadata to be standardized are similarly clustered to generate a metadata similarity cluster.

[0015] Preferably, in step C, the similarity of the metadata feature information of the metadata to be standardized with the same data type is calculated using a text similarity algorithm. If the similarity between the metadata feature information of the metadata to be standardized and the metadata feature information of other metadata to be standardized does not reach a preset threshold, the metadata to be standardized is similarly clustered to generate a metadata similarity cluster.

[0016] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which implements the steps in the above-mentioned data standard generation method when executed by a processor.

[0017] The present invention also provides a data standard generation system based on similarity clustering and data exploration, comprising a computer-readable storage medium and a processor connected to each other, wherein the computer-readable storage medium is as described above.

[0018] The present invention has the following beneficial effects: in the process of generating a data standard, a plurality of metadata to be standardized with the same structural type are first obtained, then the data information of the metadata to be standardized is subjected to standardization preprocessing to obtain metadata feature information, then similarity clustering is performed on the metadata to be standardized with the same data type based on the metadata feature information to generate a plurality of metadata similarity clusters, then metadata similarity clusters whose metadata quantity is less than a first preset value and whose number of occurrences of each metadata is less than a second preset value are eliminated, that is, metadata similarity clusters generated based on metadata to be standardized that are not generally applicable are eliminated, then for each metadata similarity cluster retained after the elimination, a preset data exploration statement is used to explore the field metadata in the metadata similarity cluster according to its data type to obtain data features, and finally, a corresponding data standard is generated based on the data features. This is equivalent to first selecting metadata similarity clusters generated based on generally applicable metadata to be standardized, and then performing data exploration on them to generate generally applicable data standards, that is, the generated data standards are more reasonable. In this process, there is no need for experts to define the content of the generally applicable data standards, thereby reducing dependence on expert definitions. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a flowchart of the data standard generation method based on similarity clustering and data exploration;

[0020] Figure 2 This is a flowchart of character data exploration;

[0021] Figure 3 This is a flowchart for numerical data exploration. DETAILED DESCRIPTION

[0022] The present invention is further described in detail below in conjunction with specific embodiments.

[0023] This embodiment provides a data standard generation system based on similarity clustering and data exploration, which includes a computer-readable storage medium and a processor connected to each other. The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the system realizes the following Figure 1 The data standard generation method based on similarity clustering and data exploration shown in the figure specifically includes the following steps A, B, C, D, E, and F.

[0024] A. Collect multiple heterogeneous metadata with different structural types from a multi-source heterogeneous database, and extract multiple metadata to be standardized of the structural type from the multi-source heterogeneous metadata according to the structural type input by the user.

[0025] A multi-source heterogeneous database refers to a database that synchronizes metadata generated by application systems such as Oracle, MySQL, Hive, MongoDb, and HDFS in real time. This metadata comes from different business systems, and the data structures include structured, semi-structured, and unstructured, which is multi-source heterogeneous metadata. A database that synchronizes multi-source heterogeneous metadata in real time is a multi-source heterogeneous database. Among them, the data objects of structured metadata are, for example, systems, libraries, tables, and fields. In this case, structured metadata includes system metadata, library table metadata, and field metadata. The data objects of unstructured metadata are, for example, documents, images, videos, and audio. In this case, unstructured metadata includes document metadata, image metadata, video metadata, and audio metadata. The data objects of semi-structured metadata are, for example, time series data of a device. In this case, semi-structured metadata includes time series metadata.

[0026] The system collects these multi-source heterogeneous metadata with different structural types from multi-source heterogeneous databases, and then extracts multiple metadata to be standardized with the same data objects from these multi-source heterogeneous metadata according to the data objects input by the user or the data objects preset by the system, for example, extracting multiple field metadata from the multi-source heterogeneous metadata.

[0027] B. For each metadata to be standardized, obtain multiple pieces of data information, and perform standardization preprocessing on each piece of data information to obtain multiple metadata feature information;

[0028] The data information of each field metadata includes English name, Chinese name and data type. Standardization preprocessing of these data information includes: stem extraction or word form restoration of English name, for example, stem extraction of English name effective is performed to obtain metadata feature information effect, for example, word form restoration of English name driving is performed to obtain metadata feature information drive; synonym normalization of Chinese name based on WordNet (Word Meaning Network), for example, the Chinese name "distribution transformer" and the Chinese name "distribution transformer" are synonyms, then these two Chinese names can be normalized to express as one of the Chinese names to obtain metadata feature information, for example, normalized expression is "distribution transformer", or normalized expression For "matching transformation", the specific situation depends on the preset situation of the system; standardize the expression of data types. For example, the type descriptions String, Varchar, Varchar2, char, and nchar represent the same data type, which means character class, so they can be standardized into metadata feature information String. For example, the type descriptions Float, number, decimal, and double represent the same data type, which means numeric class, so they can be standardized into metadata feature information number. For example, Date, time, datetime, and timestamp represent the same data type, which means date class, so they can be standardized into metadata feature information Date.

[0029] C. obtaining the data type of each metadata to be standardized based on the metadata feature information of each metadata to be standardized, performing similarity clustering on metadata to be standardized with the same data type, and generating multiple metadata similarity clusters;

[0030] In each metadata to be standardized, the metadata feature information obtained based on the type description can reflect the data type of the corresponding metadata to be standardized. The system can obtain the data type of each metadata to be standardized based on the multiple metadata feature information of each metadata to be standardized. For example, there are 100 field metadata, the type descriptions of the metadata of the 1st to 50th fields are String, and the type descriptions of the metadata of the 51st to 70th fields are Varchar. Then, the metadata feature information obtained based on the type description of the metadata of the 1st to 70th fields are all String, which means that the data types of the metadata of the 1st to 70th fields are the same, all of which are character classes. Therefore, similarity calculation can be performed on the metadata of the 1st to 70th fields, and then similarity clustering can be performed on the metadata of the 1st to 85th fields respectively according to the similarity calculation results; and the type descriptions of the metadata of the 71st to 80th fields are number, and the type descriptions of the metadata of the 81st to 85th fields are decimal. Then, the metadata feature information obtained based on the type description of the metadata of the 71st to 85th fields are all Float means that the data types of the metadata in the 71st to 85th fields are the same, all of which are numeric types. Therefore, similarity calculation can be performed on the metadata in the 71st to 85th fields, and then similarity clustering can be performed on the metadata in the 71st to 85th fields based on the similarity calculation results. The metadata type descriptions of the 86th to 95th fields are Date, and the metadata type descriptions of the 96th to 100th fields are datetime. Therefore, the metadata feature information obtained based on the type descriptions of the metadata in the 86th to 100th fields is Date, which means that the data types of the metadata in the 86th to 100th fields are the same, all of which are date types. Therefore, similarity calculation can be performed on the metadata in the 86th to 100th fields, and then similarity clustering can be performed on the metadata in the 86th to 100th fields based on the similarity calculation results.

[0031] Taking the similarity calculation of the metadata of the 1st to 85th fields as an example, a detailed description is given: for the English names and Chinese names of the metadata of the 1st to 85th fields with the same data type, a text similarity algorithm (such as the existing conventional simhash algorithm) is used to calculate the similarity between any two English names, and the similarity between any two Chinese names is calculated. The field metadata corresponding to the English names or Chinese names whose similarity reaches a preset level (for example, 80%) are similarly clustered to generate multiple metadata similarity clusters. For example, if the similarity between the English name of the metadata of the first field and the English names of the metadata of the second, third, and fourth fields reaches a preset degree, and the similarity between the Chinese name of the metadata of the first field and the Chinese names of the metadata of the fifth, sixth, seventh, and eighth fields reaches a preset degree, then similarity clustering is performed on the metadata of the first to eighth fields to obtain the first metadata similarity cluster; for example, if the similarity between the Chinese name of the metadata of the ninth field and the Chinese names of the metadata of other fields does not reach the preset degree, and the similarity between the English name of the metadata of the ninth field and the English names of the metadata of other fields does not reach the preset degree, then similarity clustering is performed on the metadata of the ninth field alone to obtain the second metadata similarity cluster; for example, if the similarity between the English names of any two metadata of the tenth to seventieth fields reaches the preset degree, then similarity clustering is performed on the metadata of the tenth to seventieth fields to obtain the third metadata similarity cluster.

[0032] The method for calculating the similarity of the metadata of the 71st to 85th fields is the same as the method for calculating the similarity of the metadata of the 1st to 85th fields. The calculation result is that if the similarity of the English name or Chinese name between any two metadata of the 71st to 85th fields reaches a preset level, the metadata of the 71st to 85th fields are similarly clustered to obtain the fourth metadata similarity cluster.

[0033] The method for calculating the similarity of the metadata of the 86th to 100th fields is the same as the method for calculating the similarity of the metadata of the 1st to 85th fields mentioned above. The calculation result is that if the similarity of the English name or Chinese name between any two metadata of the 86th to 100th fields reaches a preset level, the metadata of the 86th to 100th fields are similarly clustered to obtain the fifth metadata similarity cluster.

[0034] D. Based on the number of metadata in each metadata similarity cluster and the number of times each metadata appears in the application system, eliminate metadata similarity clusters whose number of metadata is less than a first preset value and whose number of times each metadata appears is less than a second preset value.

[0035] After clustering to obtain the above five metadata similarity clusters, the system obtains the number of field metadata in each metadata similarity cluster and the number of times each field metadata appears in the application system, determines whether the number of field metadata is less than a first preset value (specifically 2), and determines whether the number of times each field metadata appears in the application system is less than a second preset value (specifically 3), and then eliminates metadata similarity clusters whose number of field metadata is less than the first preset value and whose number of metadata appears is less than the second preset value. The number of times field metadata appears in the application system can be calculated based on the number of field metadata with exactly the same data information. For example, if two field metadata have exactly the same data information among multiple field metadata extracted from multi-source heterogeneous metadata, the number of times the field metadata appears in the application system is 2. In this embodiment, if the data information between the various field metadata extracted from the multi-source heterogeneous metadata is not exactly the same, the number of times each field metadata appears in the application system is 1.

[0036] In this embodiment, the number of field metadata in the first metadata similarity cluster is 8, which is no less than the first preset value; the number of field metadata in the second metadata similarity cluster is 1, which is no less than the first preset value; the number of field metadata in the third metadata similarity cluster is 61, which is no less than the first preset value; the number of field metadata in the fourth metadata similarity cluster is 15, which is no less than the first preset value; and the number of field metadata in the fifth metadata similarity cluster is 25, which is no less than the first preset value. Since the number of times each field metadata appears in the application system is 1, which is no less than the second preset value, the number of field metadata in the second metadata similarity cluster is no less than the first preset value, and the number of occurrences of each field metadata is no less than the second preset value, the second metadata similarity cluster is eliminated. However, since the number of occurrences of each field metadata in the first, third, fourth, and fifth metadata similarity clusters is no less than the second preset value, but the number of field metadata is no less than the first preset value, the first, third, fourth, and fifth metadata similarity clusters are retained.

[0037] E. Obtain the data type of the metadata similar clusters that remain after elimination, and generate optional data standards based on the data;

[0038] After the system eliminates the second metadata similarity cluster and retains the first, third, fourth, and fifth metadata similarity clusters, it obtains the data types of the first, third, fourth, and fifth metadata similarity clusters, where: the metadata feature information obtained based on the type description of the field metadata in the first metadata similarity cluster is all String, that is, the data type of the first metadata similarity cluster is String, which means character class; the metadata feature information obtained based on the type description of the field metadata in the third metadata similarity cluster is all String, that is, the data type of the third metadata similarity cluster is String, which means character class; the metadata feature information obtained based on the type description of the field metadata in the fourth metadata similarity cluster is all Float, that is, the data type of the third metadata similarity cluster is Float, which means numeric class; the metadata feature information obtained based on the type description of the field metadata in the fifth metadata similarity cluster is all Date, that is, the data type of the fifth metadata similarity cluster is Date, which means date class.

[0039] For character field metadata, it may be text or encoding, so two optional data standards are generated based on the first metadata similarity cluster and the third metadata similarity cluster retained after elimination, namely text standard and encoding standard; for numeric field metadata, it may be encoding or numerical value, so two optional data standards are generated based on the fourth metadata similarity cluster retained after elimination, namely encoding standard and numerical value standard; and for date field metadata, it can only be date, so one optional data standard is generated based on the fifth metadata similarity cluster retained after elimination, specifically date standard.

[0040] F. For each metadata similarity cluster that remains after elimination, use the preset data exploration statement to explore the field metadata in the metadata similarity cluster according to its data type to obtain data characteristics, and select and generate corresponding data standards from the optional data standards based on the data characteristics.

[0041] For the first metadata similarity cluster that remains after elimination, the preset character exploration statement is used to explore the field metadata in the first metadata similarity cluster according to its character data type to obtain data features, and then the corresponding data standards are selected from the optional data standards according to the data features, as follows: Figure 2As shown, the character exploration statement includes value grouping statistics and length grouping statistics. Specifically, the value grouping statistics of the field metadata are first performed, and then the length grouping statistics are performed. Then, it is determined whether the number of value groups is less than the first threshold (specifically 50) and whether the number of length groups is the second threshold (specifically 2). If so, the encoding class standard is defined and generated from the optional data standards based on the length of the largest data volume (that is, the data feature). If not, the text class standard is defined and generated from the optional data standards based on the length of the largest data volume (that is, the data feature).

[0042] For example, the eight field metadata in the first metadata similarity cluster are 0001, 0003, 0002, 0001, 0001, 0002, 0003, and 0004. The value grouping statistics are: (1) 3 0001s; (2) 2 0002s; (3) 2 0003s; (4) 1 0004; that is, the number of value groups is 4. The length grouping statistics are: (1) 8 data with a length of 4, that is, the number of length groups is 1. In this case, the number of value groups is less than 50, but the number of length groups is not 2. Therefore, the text class standard is generated based on the length of the largest data (i.e., data feature): the text character length is 4.

[0043] In other embodiments, if the number of value groups is less than 50 and the number of length groups is 2, the encoding class standard can be generated based on the length X of the largest data volume (ie, data feature): the encoding character length is X.

[0044] Since the data type of the third metadata similarity cluster retained after elimination is also character class, the third metadata similarity cluster is explored using the same character exploration statement as the first metadata similarity cluster. The specific exploration process is the same and will not be repeated here.

[0045] For the fourth metadata similarity cluster that remains after elimination, the preset numerical exploration statement is used to explore the field metadata in the fourth metadata similarity cluster according to its numerical data type to obtain data features, and then the corresponding data standards are selected from the optional data standards according to the data features, as follows: Figure 3As shown, the numerical exploration statement includes decimal place judgment, value grouping statistics and quartile calculation. Specifically, it is first judged whether the field metadata has decimal places. If there are decimal places, the quartiles are used to calculate the value range of the field metadata. Then, according to the value range calculation result (i.e., data characteristics), the numerical class standard is defined and generated from the optional data standards. If there are no decimal places, the field metadata is grouped and counted, and then it is judged whether the number of value groups is less than the first threshold (specifically 50). If so, the coding class standard is defined and generated from the optional data standard according to the length of the largest data volume (i.e., data characteristics). If not, the quartiles are used to calculate the value range of the field metadata. Then, according to the value range calculation result (i.e., data characteristics), the numerical class standard is defined and generated from the optional data standards.

[0046] For example, if the metadata of the 15 fields in the fourth metadata similarity cluster are 0001, 0003, 0002, 0001, 0001, 0002, 0003, 0004, 0001, 0003, 0002, 0001, 0001, 0002, 0003, and none of them have decimal places, then after judging the decimal places, the value grouping statistics are performed, and the value grouping statistics results are: (1) 0001 has 6; (2) 0002 has 4; (3) 0003 has 4; (4) 0004 has 1; that is, the number of value groups is 4, which is less than the first threshold (specifically 50), so the encoding class standard is generated according to the length of the largest data volume (that is, the data feature): the encoding character length is 4.

[0047] If the 15 field metadata in the fourth metadata similarity cluster are 0.001, 0003, 0002, 0001, 0001, 0002, 0003, 0004, 0001, 0003, 0002, 0001, 0001, 0002, 0003, and 0.001 has decimal places, then after judging the decimal places, the quartile range is calculated for the field metadata. Specifically, the 15 field metadata are first arranged from small to large to obtain 0.001, 0001, 0001, 0001, 0001, 0001, 0002, 0002, 0002, 0003, 0003, 0003, 0004, and then the 15 field metadata are divided into four equal parts using three split points, where the first split point is at (n+1 ) / 4 bits, the second split point is at (n+1) / 2 bits, and the third split point is at (n+1) / 4*3 bits, where n is the number of field metadata, specifically 15. In this way, the first split point is at the 4th bit, specifically 0001, the second split point is at the 8th bit, specifically 0002, and the third split point is at the 12th bit, specifically 0003. The median is 0002, and its distance to the first split point is equal to the distance to the third split point, which means that the values ​​of most field metadata are between the first and second split points and between the second and third split points, that is, the values ​​of most field metadata are between the first and third split points, that is, the value range is [0001, 0003]. Therefore, the generation of numerical class standards can be defined: the value range is [0001, 0003].

[0048] In other embodiments, if the distance from the median to the first split point is smaller than the distance to the third split point, it means that the values ​​of most field metadata are between the first split point and the second split point, that is, the value range is [0001, 0002], so a numerical class standard can be defined: the value range is [0001, 0002]; if the distance from the median to the first split point is larger than the distance to the third split point, it means that the values ​​of most field metadata are between the second split point and the third split point, that is, the value range is [0002, 0003], so a numerical class standard can be defined: the value range is [0002, 0003].

[0049] For the fifth metadata similarity cluster retained after elimination, the preset date exploration statement is used to explore the field metadata in the fifth metadata similarity cluster according to its date data type to obtain data features, and then the corresponding data standards are selected from the optional data standards based on the data features. Specifically, the date format "YYYY-MM-DD" or "YYYY-MM-DD-hh24:mm:ss" is first identified for each field metadata in the fifth metadata similarity cluster using a date regular expression. Then, these two date formats are grouped and counted to obtain the data volume of the date format "YYYY-MM-DD" and the data volume of the date format "YYYY-MM-DD-hh24:mm:ss". Then, the date format with the larger data volume is used as the data feature, and the date standard is defined and generated from the optional data standards.

[0050] For example, among the 25 date-related field metadata in the fifth metadata similarity cluster, 15 fields have a date format of "YYYY-MM-DD" and 10 fields have a date format of "YYYY-MM-DD-hh24:mm:ss". Therefore, the data volume of "YYYY-MM-DD" is greater than that of "YYYY-MM-DD-hh24:mm:ss". Therefore, the date format with a larger data volume, "YYYY-MM-DD", is used as the data feature to generate the date standard: YYYY-MM-DD.

[0051] In the above-mentioned data standard generation process, multiple metadata to be standardized with the same structural type are first obtained, and then the data information of the metadata to be standardized is subjected to standardization preprocessing to obtain metadata feature information. Then, based on the metadata feature information, similarity clustering is performed on the metadata to be standardized with the same data type to generate multiple metadata similarity clusters. Then, metadata similarity clusters whose metadata quantity is less than a first preset value and whose number of occurrences of each metadata is less than a second preset value are eliminated. That is, metadata similarity clusters generated based on metadata to be standardized that are not generally applicable are eliminated. Then, for each metadata similarity cluster retained after the elimination, preset data exploration statements are used to explore the field metadata in the metadata similarity cluster according to its data type to obtain data features. Finally, corresponding data standards are generated based on the data features. This is equivalent to first selecting metadata similarity clusters generated based on generally applicable metadata to be standardized, and then performing data exploration on them to generate generally applicable data standards. That is, the generated data standards are more reasonable. In this process, there is no need for experts to define the content of generally applicable data standards, thereby reducing dependence on expert definitions.

[0052] The above is only an embodiment of the invention and does not limit the scope of patent protection. Those skilled in the art can make non-substantial changes or substitutions based on the invention and still fall within the scope of patent protection.

Claims

1. A data standard generation method based on similarity clustering and data exploration, characterized by: The steps include: A. Collect multiple heterogeneous metadata with different structural types from multi-source heterogeneous databases, and extract multiple metadata to be standardized with the same data objects; B. For each metadata to be standardized, obtain multiple pieces of data information, and perform standardization preprocessing on each piece of data information to obtain multiple metadata feature information; C. obtaining the data type of each metadata to be standardized based on the metadata feature information of each metadata to be standardized, performing similarity clustering on metadata to be standardized with the same data type, and generating multiple metadata similarity clusters; D. based on the number of metadata in each metadata similarity cluster and the number of times each metadata appears in the application system, eliminating metadata similarity clusters where the number of metadata is less than a first preset value and the number of times each metadata appears is less than a second preset value; E. Obtain the data type of the metadata similar clusters that remain after elimination, and generate optional data standards based on the data; F. For each metadata similarity cluster that remains after elimination, explore the optional data standards using a preset data exploration statement based on its data type to obtain data features, and generate corresponding data standards based on the data features; In step B, the data types of the metadata to be standardized include character, numerical and date types; in step E, optional text standards and encoding standards are generated based on the character data type, optional encoding standards and numerical standards are generated based on the numerical data type, and optional date standards are generated based on the date data type; in step F, the data exploration statements used according to the character data type include value grouping statistics and length grouping statistics, the data exploration statements used according to the numerical data type include decimal place judgment, value grouping statistics and quartile calculation value range, and the data exploration statements used according to the date data type include using date regular expressions to identify date formats.

2. The data standard generation method based on similarity clustering and data exploration according to claim 1 is characterized in that: The data information of the metadata to be standardized includes English names, Chinese names and / or data types. In step B, the standardization preprocessing of the data information of the metadata to be standardized includes: stem extraction or lemma restoration of the English name, synonym normalization expression of the Chinese name, and / or standardized expression of the data type.

3. The data standard generation method based on similarity clustering and data exploration according to claim 2 is characterized in that: In step C, similarity is calculated for metadata feature information of metadata to be standardized with the same data type using a text similarity algorithm. If the similarity of metadata feature information of at least two metadata to be standardized reaches a preset threshold, similarity clustering is performed on the at least two metadata to be standardized to generate a metadata similarity cluster.

4. The data standard generation method based on similarity clustering and data exploration according to claim 2 is characterized in that: In step C, similarity is calculated for metadata feature information of metadata to be standardized with the same data type using a text similarity algorithm. If the similarity between the metadata feature information of the metadata to be standardized and the metadata feature information of other metadata to be standardized does not reach a preset threshold, similarity clustering is performed on the metadata to be standardized to generate a metadata similarity cluster.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the data standard generation method according to any one of claims 1 to 4 are implemented.

6. A data standard generation system based on similarity clustering and data exploration, comprising a computer-readable storage medium and a processor connected to each other, characterized in that: The computer-readable storage medium as claimed in claim 5.

Images