Data standard determination method and device and electronic equipment

Abstract

The invention discloses a data standard determination method and device and electronic equipment. The method comprises the steps that metadata in a data source is obtained through a stream processing framework, and the metadata at least comprises a database mode and a field attribute which the data belongs to; performing feature extraction on the metadata to obtain target feature data corresponding to the metadata; a knowledge graph corresponding to the metadata is generated according to the target feature data, and the knowledge graph is used for representing a data relation in the metadata; data standards corresponding to the metadata are determined according to the knowledge graph, and the data standards are data rules used for standardizing data in the metadata. The technical problem that data standardization efficiency is low due to the fact that a data standard recommendation method in related technologies generally depends on manual calibration and cannot fully extract data features of metadata is solved.

Description

Technical Field

[0001] The present application relates to the field of data governance, and more specifically, to a method, device and electronic device for determining data standards. Background Art

[0002] Data governance is the core of data management. Its goal is to establish a data management system, eliminate data silos, improve data quality, and provide reliable, accurate, and timely data to meet business needs. Data governance involves the formulation of data standards. Data standards refer to the regulatory requirements for data collection, storage, use, exchange, and destruction throughout its life cycle, and are the foundation of data governance.

[0003] In related technologies, the formulation of data standards usually relies on manual work based on business needs and industry specifications. This is a huge workload, inefficient, and prone to omissions or errors. It is also impossible to effectively extract data features. Existing data governance tools or systems also find it difficult to provide intelligent formulation of data standards.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a method, device and electronic device for determining a data standard, so as to at least solve the technical problem that the data standard recommendation method in the related art usually relies on manual calibration and cannot fully extract the data features of metadata, resulting in low data standardization efficiency.

[0006] According to one aspect of an embodiment of the present application, a method for determining a data standard is provided, comprising: obtaining metadata from a data source through a stream processing framework, wherein the metadata includes at least a database schema and field attributes to which the data belongs; performing feature extraction on the metadata to obtain target feature data corresponding to the metadata; generating a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent data relationships in the metadata; determining a data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to standardize data in the metadata.

[0007] Optionally, after obtaining the metadata in the data source through the stream processing framework, the method also includes: determining the feature extraction logic of the metadata; extracting key data features in the metadata according to the feature extraction logic to obtain first feature data; cleaning the metadata and the first feature data, and combining the cleaned metadata and the first feature data to obtain second feature data.

[0008] Optionally, feature extraction is performed on the metadata to obtain target feature data corresponding to the metadata, including: determining a feature extraction operation for the second feature data, wherein the feature extraction operation includes at least one of the following: missing value processing, feature conversion, feature selection, feature construction, feature dimensionality reduction, feature persistence and feature structuring; and feature extraction is performed on the second feature data according to the feature extraction operation to obtain target feature data.

[0009] Optionally, a knowledge graph corresponding to the metadata is generated based on the target feature data, including: processing the target feature data through a clustering analysis algorithm to obtain a clustering analysis result, wherein the clustering analysis result is used to represent the grouping of data points with similar features in the target feature data; processing the clustering analysis result through preset association rules to obtain an association learning result, wherein the association learning result is used to represent frequent item sets and association rules in the clustering analysis result; determining a knowledge graph based on the association learning result, wherein the nodes in the knowledge graph are used to represent the fields of the association learning result, and the edges in the knowledge graph are used to represent the field relationships of the association learning result.

[0010] Optionally, before determining the knowledge graph based on the association learning results, the method also includes: determining frequent item sets and association rules in the association learning results; determining conversion parameters for reverse feature conversion of the association learning results; and reversely converting the frequent item sets and association rules based on the conversion parameters.

[0011] Optionally, before determining the knowledge graph based on the associated learning results, the method also includes: performing data structure adaptation on the associated learning results, wherein the data structure adaptation is used to adjust the data structure of the associated learning results.

[0012] Optionally, the method further includes: sending the data standard to the target object; receiving confirmation information returned by the target object, or receiving feedback information returned by the target object, and updating the data standard according to the feedback information.

[0013] According to another aspect of an embodiment of the present application, a device for determining a data standard is also provided, including: an acquisition module, used to acquire metadata in a data source through a stream processing framework, wherein the metadata includes at least a database schema and field attributes to which the data belongs; an extraction module, used to perform feature extraction on the metadata to obtain target feature data corresponding to the metadata; a generation module, used to generate a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent data relationships in the metadata; a determination module, used to determine a data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to standardize data in the metadata.

[0014] According to another aspect of the embodiments of the present application, there is also provided an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute a determination method for implementing the above-mentioned data standard.

[0015] According to another aspect of the embodiments of the present application, a non-volatile storage medium is provided, the non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned method for determining the data standard by running the computer program.

[0016] According to another aspect of the embodiments of the present application, a computer program product is provided, including computer instructions, which implement the above-mentioned method for determining the data standard when the computer instructions are executed by a processor.

[0017] In an embodiment of the present application, metadata in a data source is obtained through a stream processing framework, wherein the metadata includes at least a database schema and field attributes to which the data belongs; feature extraction is performed on the metadata to obtain target feature data corresponding to the metadata; a knowledge graph corresponding to the metadata is generated based on the target feature data, wherein the knowledge graph is used to represent data relationships in the metadata; data standards corresponding to the metadata are determined based on the knowledge graph, wherein the data standards are data rules for standardizing data in the metadata, thereby achieving the purpose of automation and intelligence from data feature extraction to data standard formulation, thereby achieving the technical effect of improving the accuracy of data standardization and enhancing the intuitiveness of recommendation results, thereby solving the technical problem of low data standardization efficiency due to the fact that the data standard recommendation method in the related art usually relies on manual calibration and cannot fully extract the data features of the metadata. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a hardware structure diagram of a computer terminal for implementing a method for determining a data standard according to an embodiment of the present application;

[0020] Figure 2 is a flow chart of a method for determining a data standard according to an embodiment of the present application;

[0021] Figure 3 It is a structural diagram of a device for determining a data standard according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] First, some nouns or terms that appear in the process of explaining the embodiments of the present application are subject to the following explanations:

[0025] Apache Flink: An open source framework for large-scale data stream processing that supports batch processing and stream processing for extracting data features.

[0026] Flink Gelly: A graph computing library in Apache Flink that provides a rich set of graph algorithms and operations, simplifies the development of graph analysis applications, and is used to calculate data feature relationships and assist in the generation of data standards.

[0027] Clustering algorithm: An unsupervised learning method that divides the samples in a data set into several clusters, so that the samples in the same cluster have high similarity and the samples in different clusters have low similarity. It is used to identify natural groupings in the data, thus providing a basis for data classification and standard setting.

[0028] Association rule learning: In a large database system, quickly find the potential and valuable associations between things, express them with rules, and draw important related conclusions after reasoning and accumulating knowledge. It is used to discover association rules between fields and provide interpretability of generation standards to assist decision-making.

[0029] In order to solve the problem of poor data standardization efficiency in the related art, the present application embodiment provides a method for determining a data standard. The method can be run on Figure 1 In the computer terminal shown, the computer terminal is described below.

[0030] The data standard determination method embodiment provided in the embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal for implementing a method for determining a data standard. Figure 1 As shown, the computer terminal 10 may include one or more (102a, 102b, ..., 102n are used to illustrate) processors (the processor may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission module 106 for communication functions connected via a wired and / or wireless network. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art can understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0031] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10. As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as program instructions / data storage devices corresponding to the method for determining the data standard in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for determining the data standard described above is implemented. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0033] The transmission module 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission module 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission module 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0034] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 .

[0035] It should be noted that, in some optional embodiments, the above Figure 1 The computer terminal shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. It should be noted that Figure 1 This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the computer terminal described above.

[0036] In the above-mentioned operating environment, an embodiment of the present application provides an embodiment of a method for determining a data standard. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0037] Figure 2 is a flow chart of a method for determining a data standard according to an embodiment of the present application. Figure 2As shown, the method comprises the following steps:

[0038] Step S202: obtaining metadata in the data source through a stream processing framework, wherein the metadata at least includes a database schema and field attributes to which the data belongs.

[0039] In the above step S202, Apache Flink can be used as a stream processing framework to extract metadata from data sources (such as databases, data lakes, etc.), where the metadata includes but is not limited to database schemas and field attributes (such as data types, lengths, whether nulls are allowed, etc.). This step is the basis for the formulation of data standards, ensuring that subsequent steps can be operated based on accurate data source information.

[0040] Step S204: extract features from the metadata to obtain target feature data corresponding to the metadata.

[0041] In the above step S204, the purpose of feature extraction is to identify and extract key information in the metadata, which may include the data type, field length, field uniqueness, and field statistical characteristics (such as mean, standard deviation, etc.) of the field, for subsequent knowledge graph construction and data standard generation. Among them, the means of feature extraction may include, for example: missing value processing, feature conversion, feature selection, feature construction, feature dimensionality reduction, feature persistence, and feature structuring.

[0042] Step S206, generating a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent the data relationship in the metadata.

[0043] In the above step S206, the target feature data extracted from the metadata can be used to construct a knowledge graph. The knowledge graph is a graphical data structure used to represent entities (data fields in this scenario) and the relationships between them. In the knowledge graph, each data field is regarded as a node, and the relationships between fields (such as dependencies, correlations, etc.) are regarded as edges. The construction of the knowledge graph helps to intuitively display the relationships between data and provide visual support for the formulation of data standards.

[0044] Step S208, determining the data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to standardize the data in the metadata.

[0045] In the above step S208, the data standard corresponding to the metadata can be determined based on the knowledge graph through relevant graph algorithms. Among them, the data standard is a set of data rules used to standardize the format, type, processing method, etc. of the data to ensure the consistency and comparability of the data. In this step, the data relationships displayed in the knowledge graph can be used to identify which fields require unified data standards and which fields have dependencies, so as to formulate corresponding data rules. For example, if two fields are shown to be highly correlated in the graph, rules may need to be formulated to ensure that the data format and processing method of the two fields remain consistent.

[0046] Through the above steps S202 to S208, the purpose of automation and intelligence from data feature extraction to data standard formulation is achieved, thereby achieving the technical effect of improving the accuracy of data standardization and enhancing the intuitiveness of recommendation results, thereby solving the technical problem that the data standard recommendation method in the related art usually relies on manual calibration and cannot fully extract the data features of metadata, resulting in low data standardization efficiency. The following is a detailed description.

[0047] In the above step S202, after obtaining the metadata in the data source through the stream processing framework, the method also includes: determining the feature extraction logic of the metadata; extracting key data features in the metadata according to the feature extraction logic to obtain first feature data; cleaning the metadata and the first feature data, and combining the cleaned metadata and the first feature data to obtain second feature data.

[0048] In an embodiment of the present application, the data preparation stage includes obtaining metadata from the data source through a stream processing framework (such as Apache Flink) and performing a series of data processing steps to ensure the quality and applicability of the data.

[0049] The details are as follows:

[0050] 1. Metadata extraction:

[0051] Use Apache Flink to extract necessary metadata from data sources (such as relational databases, NoSQL databases, message queues, etc.), including database schemas, field attributes, etc., to ensure that Flink jobs can adapt to different data sources and data formats, and can be flexibly configured to meet different metadata extraction requirements.

[0052] 2. Data feature collection:

[0053] For the original metadata, data feature extraction logic is defined in Flink to adapt to different data types and business requirements, such as calculating the average, maximum, minimum, and standard deviation of the field. And based on this feature extraction logic, key data features are collected from the metadata, that is, the first feature data mentioned above, such as data distribution, statistical summary, etc.

[0054] According to different data types (numeric, string, Boolean, time series, binary / categorical, text data, spatial data), use Flink's AggregateFunction, ProcessFunction, MapFunction, etc. to extract features, as shown in Table 1.

[0055] Table 1 The first feature data extraction method

[0056]

[0057]

[0058] 3. Data cleaning:

[0059] Define clear data quality rules, such as data format, data scope, data integrity, etc., to guide the data cleaning process.

[0060] Furthermore, the map or flatMap function of Flink is used to clean the metadata and the extracted first feature data to remove invalid, erroneous or inconsistent data.

[0061] 4. Data features and metadata fusion:

[0062] Use Flink to operate the cleaned metadata and the first feature data to ensure that the data features are correctly associated with the corresponding metadata. Bind and merge the metadata and data features from different data sources to form a unified and structured data set, namely the second feature data mentioned above.

[0063] 5. Data Verification:

[0064] Verify the integrated second feature data and add verification logic to the Flink job, such as checking the consistency and completeness of the data set to ensure the quality and applicability of the data set.

[0065] In the above step S204, feature extraction is performed on the metadata to obtain target feature data corresponding to the metadata, including: determining a feature extraction operation of the second feature data, wherein the feature extraction operation includes at least one of the following: missing value processing, feature conversion, feature selection, feature construction, feature dimensionality reduction, feature persistence and feature structuring; and feature extraction is performed on the second feature data according to the feature extraction operation to obtain target feature data.

[0066] In the embodiment of the present application, feature extraction of the integrated metadata (i.e., the second feature data) is a key step, which helps to deeply understand the structure and attributes of the data, thereby providing a more accurate data basis for data standardization (i.e., the target feature data). The following are the detailed steps and methods for feature extraction of the second feature data:

[0067] 1. Missing value processing:

[0068] Identify missing values ​​in metadata and choose an appropriate filling strategy based on the characteristics of the field. For example:

[0069] Missing values ​​in field length and precision are filled with the mean of the column; missing values ​​in field name and description are filled with the string "Unknown"; missing values ​​in numerical features (such as mean, maximum, minimum) are filled with the median.

[0070] 2. Feature conversion:

[0071] By transforming features, we can improve the distribution characteristics of data and make it easier for the model to capture the potential rules in the data.

[0072] For example:

[0073] Standardize the numerical features in the metadata, such as Z-score standardization, to eliminate the impact of different dimensions; encode the categorical data, such as One-Hot Encoding or Label Encoding, to facilitate processing by machine learning models.

[0074] 3. Feature Selection:

[0075] Feature selection is used to select the features that are most relevant to the target variable from the original data. For example:

[0076] Statistical-based methods (such as Pearson's correlation coefficient) are used to evaluate the relevance of features to data criteria, and features with correlations below a certain threshold are removed, retaining the most relevant feature subset.

[0077] 4. Feature construction:

[0078] Combining business knowledge and data understanding, construct new features that may better reflect the intrinsic attributes of the data and business needs. For example:

[0079] The ratio of field length to average value is calculated by length / average_value.

[0080] 5. Feature Dimensionality Reduction:

[0081] For categorical features containing multiple categories (such as field types), use dimensionality reduction techniques such as principal component analysis (PCA) and t-SNE to reduce the number of features while retaining the information of the original data as much as possible to improve data processing efficiency and model performance.

[0082] 6. Feature persistence:

[0083] During the feature encoding process, the encoding mapping relationship is recorded, and a mapping table is used to store the original value before label encoding and one-hot encoding and the encoded value. For example:

[0084] The feature_storage_mapping table is used to store the mapping during feature engineering. Its data structure is as follows:

[0085] FeatureID (INT, PK): feature ID;

[0086] TableID (INT, FK->tables.TableID): the table ID (tables structure is omitted);

[0087] FieldID (INT, FK->fields.FieldID): the field ID (the fields table structure is omitted);

[0088] FeatureName (VARCHAR): feature name;

[0089] FeatureValue (VARCHAR): feature encoding value;

[0090] FeatureDataType (VARCHAR): feature data type, such as numeric type, string type, etc.;

[0091] TransformParamID(INT,FK->fields.FieldID): Use transformation parameter ID.

[0092] After feature conversion and encoding, save the conversion type (such as mean, standard deviation, encoding mapping, feature construction, PCA feature dimensionality reduction, Z-score, etc.) and specific parameters (JSON) for subsequent analysis. For example:

[0093] Use feature_transform_log to store feature transformation logs. Its data structure is as follows:

[0094] TransformLogID(INT,PK): transformation log ID;

[0095] FeatureID (INT, FK): feature ID (foreign key);

[0096] TransformDesc(VARCHAR): feature transformation summary;

[0097] TransformType(VARCHAR): transformation type;

[0098] TransformParams (JSON): transformation parameters;

[0099] OriginalValue(VARCHAR): The original value of the feature.

[0100] 7. Feature structuring:

[0101] Organize the extracted features into a structured form, such as a table or vector, to facilitate subsequent data processing and model training, thereby ensuring the consistency and integrity of the features to support the decision-making process of data standardization.

[0102] In the above step S206, a knowledge graph corresponding to the metadata is generated based on the target feature data, including: processing the target feature data through a clustering analysis algorithm to obtain a clustering analysis result, wherein the clustering analysis result is used to represent the grouping of data points with similar features in the target feature data; processing the clustering analysis result through preset association rules to obtain an association learning result, wherein the association learning result is used to represent frequent item sets and association rules in the clustering analysis result; determining a knowledge graph based on the association learning result, wherein the nodes in the knowledge graph are used to represent the fields of the association learning result, and the edges in the knowledge graph are used to represent the field relationships of the association learning result.

[0103] Before determining the knowledge graph based on the association learning results, the method also includes: determining frequent item sets and association rules in the association learning results; determining conversion parameters for reverse feature conversion of the association learning results; reversely converting the frequent item sets and association rules based on the conversion parameters; and performing data structure adaptation on the association learning results, wherein the data structure adaptation is used to adjust the data structure of the association learning results.

[0104] In the embodiments of the present application, through operations such as clustering algorithm analysis, association rule mining, feature reverse conversion, and data structure adaptation, the complex data features and relationships in the target feature data can be converted into a knowledge graph that is easy to understand and operate, thereby providing a more effective tool for data analysis and decision-making. The specific process can be as follows:

[0105] 1. Cluster Analysis

[0106] Cluster analysis is an important technology in data mining. It aims to divide the samples in the data set into several clusters, so that the samples in the same cluster have high similarity and the samples in different clusters have low similarity. In the process of data standard formulation, cluster analysis can help us identify natural groupings in the data, thus providing a basis for data classification and standard formulation. The details are as follows:

[0107] First, select a suitable clustering algorithm according to the characteristics of the target feature data, such as K-means, hierarchical clustering, etc. Among them, the K-means algorithm is simple and efficient, suitable for large-scale data sets; hierarchical clustering can discover the hierarchical structure of data and is suitable for smaller data sets.

[0108] Secondly, determine the appropriate number of clusters. For example, the optimal number of clusters can be determined by methods such as the elbow rule and the silhouette coefficient. The elbow rule determines the number of clusters by observing the downward trend of intra-cluster differences, while the silhouette coefficient determines the number of clusters by evaluating intra-cluster similarity and inter-cluster differences.

[0109] Furthermore, through multiple iterations, clustering parameters and algorithms are continuously adjusted to obtain the best clustering analysis result, which is used to represent the grouping of data points with similar characteristics in the target feature data.

[0110] Finally, the cluster analysis results are stored in the database, such as the field_cluster_mapping table, for subsequent data standard formulation and analysis. For example, the structure of the field_cluster_mapping table is as follows:

[0111] ClusterID (INT, PK): cluster ID, primary key;

[0112] ClusterName (VARCHAR): cluster name, used to describe the characteristics of the cluster;

[0113] Fields(List[INT]): Aggregate field ID list, store field IDs belonging to the same cluster;

[0114] Features(List[String]): takes a feature ID list and stores the field feature IDs used for clustering.

[0115] Through the above clustering analysis process, data fields with similar characteristics can be grouped, providing a scientific basis for data standard formulation and improving the efficiency and quality of data governance.

[0116] 2. Association Rule Mining

[0117] Apply association rule mining in each cluster in the above cluster analysis results, such as using Apriori or FP-Growth algorithm to mine association rules between fields, find frequent item sets and meaningful association rules between field feature values, and obtain association learning results to represent more specific and accurate associations. The specific process can be as follows:

[0118] Through the Features field in the field_cluster_mapping table, query the FeatureDataType and FeatureValue (encoded result, not original value) in the feature_storage_mapping table to build a data set for mining association rules, as follows:

[0119] [{"avg:1","count:5","null_count:2",……},

[0120] {"avg:1","count:5","null_count:1",……},

[0121] {"avg:1","count:4","null_count:1",……},

[0122] {"avg:0","count:5","null_count:2",…},…]

[0123] By setting the minimum support and minimum confidence, you can output frequent item sets as follows:

[0124] [{'avg:1'},{'count:5'},{'avg:1','count:5'},{'avg:1','null_count:1'},……]

[0125] At the same time, the association rules will be output, along with the improvement of the results. The better association rules will be analyzed through automatic iteration, and the associated antecedents, consequents, support, confidence, and improvement will be written into the storage, as follows:

[0126] {'avg:1'}->{'count:5'}(support=0.6,confidence=1.0,lift=1.67)

[0127] {'avg:1'}->{'null_count:1'}(support=0.6,confidence=1.0,lift=1.67)

[0128] {'avg:1','count:5'}->{'null_count:1'}(support=0.4, confidence=0.67, lift=2.22)……

[0129] For example, the field_rule_mapping table is used to store the mapping relationship between clustering results and association rule mining results. Its data structure is as follows:

[0130] RuleId (INT, PK): Rule ID, primary key, used to uniquely identify each rule. The integer type ensures the uniqueness and automatic increment of the rule ID.

[0131] ClusterID (INT, FK–>field_cluster_mapping.ClusterID): cluster ID, used to associate a specific cluster;

[0132] Fields (List[INT]): associated field ID (fields table structure omitted);

[0133] Antecedent (VARCHAR): rule antecedent;

[0134] Consequent (VARCHAR): rule consequence;

[0135] Support (FLOAT): support of the rule;

[0136] Confidence (FLOAT): confidence of the rule;

[0137] Lift(FLOAT): The lifting degree of the rule.

[0138] 3. Feature Reverse Conversion

[0139] After completing cluster analysis and association rule mining, the association learning results are reversely transformed to restore the feature data to its original state. The details are as follows:

[0140] 1. Feature restoration of fields within clusters:

[0141] Get all the field IDs in the same cluster from the field_cluster_mapping table. For these fields, query the feature_storage_mapping table through the ID in the Fields field to find the corresponding feature type. Use the transformation parameters (such as mean, standard deviation, encoding mapping, etc.) in the feature_transform_log table to reverse the feature value back to its original value.

[0142] 2. Feature reduction in association rules:

[0143] Get the detailed information of the association rule from the field_rule_mapping table, including the rule ID, the associated field ID, etc. For each association rule, query the feature_storage_mapping table by the ID in the Fields field to find the feature types of the rule antecedent and consequent. Use the transformation parameters in the feature_transform_log table to reverse-convert these feature values ​​back to their original values.

[0144] 3. Store the reverse conversion results:

[0145] Create a new data table rule_original_value to store the results of the reverse conversion. The table structure includes: RuleOriginalId (primary key), RuleOriginalType (association type, 0 indicates frequent set items, 1 indicates association rules), RuleId (foreign key, associated with field_rule_mapping.RuleId), FiledTuple (tuple associated with field ID), FeatureID (foreign key, associated with feature ID), OriginalValue (original value of the feature).

[0146] 4. Database structure adaptation

[0147] In the data structure adaptation phase, the goal is to ensure that the data after feature reverse conversion can be seamlessly connected with the knowledge graph construction and subsequent analysis process. The core tasks of this phase include adjusting the data format to adapt it to the graph database or other storage solutions, and processing different types of data features through built-in summary generation rules to subsequently generate descriptive information such as rule summaries and feature dimensions. The details are as follows:

[0148] 1. Data format standardization:

[0149] Identify the data formats supported by the graph database and convert the reverse-converted data into these formats.

[0150] 2. Application of summary generation rules:

[0151] According to the type of data features (such as numerical, text, time series, etc.), select an appropriate summary generation strategy and generate summaries based on the summary generation strategy. These summaries should be able to accurately reflect the key information of the data features.

[0152] 3. Feature dimension mapping:

[0153] Each data feature is mapped to a node or edge of the knowledge graph to ensure that the dimension and description information of the feature are preserved during the mapping process.

[0154] 4. Data storage adaptation:

[0155] Store the reformatted data in a graph database or other storage solution to ensure that the data is stored in a structure that supports efficient query and analysis operations.

[0156] 5. Knowledge Graph Construction

[0157] By converting the above association learning results into a graph structure, the relationships and features between data can be made more intuitive and easier to understand. The details are as follows:

[0158] 1. Clustering results are converted into nodes:

[0159] Node definition: Each data field is represented as a node. The attributes of the node include basic information such as the field name, data type, and description.

[0160] Cluster mapping: Map the clusters obtained by the clustering algorithm to node groups in the graph. The field nodes in each cluster are clustered together to represent their similarity or correlation in data features.

[0161] 2. Association rules are converted into edges:

[0162] Edge definition: The relationship between fields is represented by edges, and the attributes of edges include the type, support, confidence, and lift of association rules.

[0163] Rule mapping: Based on the rules obtained by the association rule learning algorithm, fields with high confidence and support are connected through edges. The attribute values ​​of the edges reflect the strength and reliability of the rules.

[0164] 3. Visual representation of attribute values:

[0165] Node size: The size of the node can be adjusted according to the support of the association rule. The nodes corresponding to the rules with high support are displayed larger to highlight their importance in the data.

[0166] Edge thickness: The edge thickness can be adjusted according to the confidence of the association rule. Rules with high confidence have thicker edges, indicating stronger association.

[0167] Color coding: Different colors can be used to represent different types of association rules or field features to increase the readability and intuitiveness of the graph.

[0168] 4. Optimization and adjustment of the graph:

[0169] Layout optimization: Use graph theory algorithms (such as force-directed layout) to optimize the layout of nodes and edges to make the graph clearer and easier to understand.

[0170] Interaction design: Add interactive functions to the front-end display of the graph, such as clicking on a node to view detailed information, dragging a node to adjust the view, etc., to enhance the user experience.

[0171] Through the application of the above-mentioned knowledge graph, not only the visual expression of data is enhanced, but also the efficiency and accuracy of data standard formulation are improved, making data governance work more scientific and systematic.

[0172] In the above step S208, after determining the data standard corresponding to the metadata based on the knowledge graph, the method also includes: sending the data standard to the target object; receiving confirmation information returned by the target object, or receiving feedback information returned by the target object, and updating the data standard based on the feedback information.

[0173] In the embodiment of the present application, the method for formulating data standards by intelligent knowledge graph visualization assistance can be implemented through the following steps:

[0174] 1. Graph algorithm development:

[0175] Flink Gelly is used to analyze graph data, extract key information from cluster analysis and association rule mining, and further screen out potential data standards.

[0176] 2. Recommended Standard Visualization:

[0177] Through a graphical user interface (GUI), data standard recommendations are intuitively displayed, including central field standards and related fields, as well as the visual relationship between them. The visual representation (such as size, color, distance) helps users understand the importance and correlation between fields.

[0178] 3. Association rule display:

[0179] The above data standards are sent to the target object, that is, the specific association rules and their confidence and support scores are displayed on the user (i.e., the above target object) interface, such as the "unique value number matching rule", and detailed data feature dimensions and data standard recommendations are provided, as shown in Table 2.

[0180] Table 2 Association rules and their confidence and support scores

[0181]

[0182]

[0183] 4. User feedback and evaluation:

[0184] Receive user confirmation information, or collect and analyze user feedback on recommendation data standards. This feedback information is used to optimize and adjust the recommendation model to improve the accuracy and applicability of future recommendations.

[0185] 5. Data standard confirmation:

[0186] Users can review the recommended data standards, accept them or make updates or modifications as needed, and then formally save and implement them.

[0187] 6. Log Audit:

[0188] Recording all recommended data standards and their modification history ensures transparency and auditability of the entire process, which is critical for data governance and compliance.

[0189] In the above interactive process, not only the automation and intelligence level of data standardization is improved, but also the applicability and acceptability of the standard are enhanced through the user feedback mechanism. Among them, through graphical display and intelligent recommendation, the data standardization process is made more intuitive and user-friendly, while maintaining a high degree of flexibility and customizability.

[0190] In the embodiments of the present application, by constructing an intelligent knowledge graph, deeply extracting and analyzing data features, applying association rule learning algorithms, using graph theory algorithms to optimize decision-making processes, developing automated data standard recommendation processes, and designing user interaction and feedback mechanisms, the automation, intelligence, and efficiency of data standard formulation are achieved. The integrated application of these technical points not only improves the interpretability, adaptability, and accuracy of data standards, but also reduces manual intervention, enhances the logic and intuitiveness of decision-making, and improves decision-making efficiency. At the same time, through user interaction and feedback mechanisms, the recommendation algorithm is continuously optimized to ensure the practicality and user satisfaction of data standards, and promote the progress of data governance practices.

[0191] According to an embodiment of the present application, a device for determining a data standard is provided. It should be noted that the device for determining a data standard in the embodiment of the present application can be used to execute the method for determining a data standard provided in the embodiment of the present application. The device for determining a data standard provided in the embodiment of the present application is introduced below.

[0192] Figure 3 is a structural diagram of a data standard determination device provided according to an embodiment of the present application. Figure 3 As shown, the device comprises:

[0193] An acquisition module 30, configured to acquire metadata in a data source through a stream processing framework, wherein the metadata includes at least a database schema and field attributes to which the data belongs;

[0194] An extraction module 32, used to extract features from the metadata to obtain target feature data corresponding to the metadata;

[0195] A generation module 34, used to generate a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent the data relationship in the metadata;

[0196] The determination module 36 is used to determine the data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to standardize the data in the metadata.

[0197] Through the acquisition module 30, extraction module 32, generation module 34 and determination module 36 in the above-mentioned data standard determination device, the purpose of automation and intelligence from data feature extraction to data standard formulation is achieved, thereby realizing the technical effect of improving the accuracy of data standardization and enhancing the intuitiveness of recommendation results, thereby solving the technical problem of low data standardization efficiency due to the fact that the data standard recommendation method in the related technology usually relies on manual calibration and cannot fully extract the data features of metadata.

[0198] In the data standard determination device provided in the embodiment of the present application, the extraction module is also used to determine the feature extraction logic of the metadata; extract key data features in the metadata according to the feature extraction logic to obtain first feature data; clean the metadata and the first feature data, and combine the cleaned metadata and the first feature data to obtain second feature data.

[0199] In the data standard determination device provided in the embodiment of the present application, the extraction module is also used to determine the feature extraction operation of the second feature data, wherein the feature extraction operation includes at least one of the following: missing value processing, feature conversion, feature selection, feature construction, feature dimensionality reduction, feature persistence and feature structuring; feature extraction is performed on the second feature data according to the feature extraction operation to obtain target feature data.

[0200] In the data standard determination device provided in the embodiment of the present application, the generation module is also used to process the target feature data through a clustering analysis algorithm to obtain a clustering analysis result, wherein the clustering analysis result is used to represent the grouping of data points with similar characteristics in the target feature data; the clustering analysis results are processed through preset association rules to obtain an association learning result, wherein the association learning result is used to represent frequent item sets and association rules in the clustering analysis results; a knowledge graph is determined based on the association learning results, wherein the nodes in the knowledge graph are used to represent the fields of the association learning results, and the edges in the knowledge graph are used to represent the field relationships of the association learning results.

[0201] In the data standard determination device provided in the embodiment of the present application, the generation module is also used to determine the frequent item sets and association rules in the association learning results; determine the conversion parameters for reverse feature conversion of the association learning results; and reversely convert the frequent item sets and association rules based on the conversion parameters.

[0202] In the data standard determination device provided in the embodiment of the present application, the generation module is also used to perform data structure adaptation on the associated learning result, wherein the data structure adaptation is used to adjust the data structure of the associated learning result.

[0203] In the data standard determination device provided in the embodiment of the present application, the determination module is further used to receive confirmation information returned by the target object, or receive feedback information returned by the target object, and update the data standard according to the feedback information.

[0204] An embodiment of the present application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; the processor is connected to the memory and is used to execute a determination method for implementing the above-mentioned data standard.

[0205] It should be noted that the above electronic equipment is used to execute Figure 2 The method for determining the data standard shown in the figure, therefore the relevant explanations and instructions in the above method for determining the data standard are also applicable to the electronic device and will not be repeated here.

[0206] An embodiment of the present application further provides a non-volatile storage medium, which includes a stored computer program, wherein a device where the non-volatile storage medium is located executes the above-mentioned method for determining the data standard by running the computer program.

[0207] It should be noted that the above non-volatile storage medium is used to execute Figure 2 The method for determining the data standard shown in the figure, therefore the relevant explanations in the above method for determining the data standard are also applicable to the non-volatile storage medium and will not be repeated here.

[0208] An embodiment of the present application also provides a computer program product, including computer instructions, which implement the above-mentioned method for determining the data standard when executed by a processor.

[0209] It should be noted that the above-mentioned computer program product is used to execute Figure 2 The method for determining the data standard shown in the figure, therefore the relevant explanations and instructions in the above method for determining the data standard are also applicable to the computer program product and will not be repeated here.

[0210] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0211] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0212] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0213] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0214] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0215] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk and other media that can store program codes.

[0216] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining a data standard, characterized in that: include: Obtaining metadata from a data source through a stream processing framework, wherein the metadata includes at least a database schema and field attributes to which the data belongs; Extracting features from the metadata to obtain target feature data corresponding to the metadata; Generating a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent data relationships in the metadata; A data standard corresponding to the metadata is determined based on the knowledge graph, wherein the data standard is a data rule for regulating the data in the metadata.

2. The method according to claim 1, characterized in that After obtaining metadata from the data source through the stream processing framework, the method further includes: Determining feature extraction logic for the metadata; Extracting key data features in the metadata according to the feature extraction logic to obtain first feature data; The metadata and the first characteristic data are cleaned, and the cleaned metadata and the first characteristic data are combined to obtain second characteristic data.

3. The method according to claim 2, characterized in that Extracting features from the metadata to obtain target feature data corresponding to the metadata includes: Determining a feature extraction operation for the second feature data, wherein the feature extraction operation includes at least one of the following: missing value processing, feature conversion, feature selection, feature construction, feature dimensionality reduction, feature persistence, and feature structuring; Feature extraction is performed on the second feature data according to the feature extraction operation to obtain the target feature data.

4. The method according to claim 1, characterized in that: Generating a knowledge graph corresponding to the metadata according to the target feature data includes: Processing the target feature data by a cluster analysis algorithm to obtain a cluster analysis result, wherein the cluster analysis result is used to represent a grouping of data points with similar features in the target feature data; Processing the cluster analysis results by using preset association rules to obtain association learning results, wherein the association learning results are used to represent frequent item sets and association rules in the cluster analysis results; The knowledge graph is determined based on the associated learning results, wherein the nodes in the knowledge graph are used to represent the fields of the associated learning results, and the edges in the knowledge graph are used to represent the field relationships of the associated learning results.

5. The method according to claim 4, characterized in that Before determining the knowledge graph according to the association learning result, the method further includes: Determining frequent item sets and association rules in the association learning results; Determining conversion parameters for performing feature inverse conversion on the association learning result; The frequent item sets and the association rules are reversely converted according to the conversion parameters.

6. The method according to claim 4, characterized in that Before determining the knowledge graph according to the association learning result, the method further includes: Data structure adaptation is performed on the associated learning result, wherein the data structure adaptation is used to adjust the data structure of the associated learning result.

7. The method according to claim 1, characterized in that The method further comprises: Sending the data standard to a target object; Receive confirmation information returned by the target object, or receive feedback information returned by the target object, and update the data standard according to the feedback information.

8. A device for determining a data standard, characterized in that: include: An acquisition module, used to acquire metadata in a data source through a stream processing framework, wherein the metadata at least includes a database mode and field attributes to which the data belongs; An extraction module, used to extract features from the metadata to obtain target feature data corresponding to the metadata; A generation module, used to generate a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent the data relationship in the metadata; A determination module is used to determine a data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to regulate the data in the metadata.

9. An electronic device, characterized in that: include: A memory and a processor, wherein the memory is used to store program instructions; The processor is connected to the memory and is used to execute the method for determining the data standard as described in any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the method for determining the data standard according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that: When the computer instructions are executed by a processor, the method for determining the data standard described in any one of claims 1 to 7 is implemented.

Images