Method, device and electronic equipment for determining data standards

Abstract

This application discloses a method, apparatus, and electronic device for determining data standards. The method includes: acquiring metadata from a data source using a stream processing framework, wherein the metadata includes at least the database schema and field attributes to which the data belongs; extracting features from the metadata to obtain target feature data corresponding to the metadata; generating a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent data relationships in the metadata; and determining a data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to standardize the data in the metadata. This application solves the technical problem that data standard recommendation methods in related technologies typically rely on manual calibration and cannot fully extract data features from metadata, resulting in low data standardization efficiency.

Description

Technical Field

[0001] This application relates to the field of data governance, and more specifically, to a method, apparatus, and electronic device for determining data standards. Background Technology

[0002] Data governance is the core of data management. Its goal is to eliminate data silos, improve data quality, and provide reliable, accurate, and timely data to meet business needs by establishing a data management system. Data governance involves the development of data standards, which are the standardized requirements for data throughout its entire lifecycle, including collection, storage, use, exchange, and destruction. These standards form the foundation of data governance.

[0003] In related technologies, the formulation of data standards usually relies on manual methods based on business needs and industry norms. This is a huge workload, inefficient, and prone to omissions or errors. It also fails to effectively extract the data features. Existing data governance tools or systems also struggle to provide intelligent data standard formulation capabilities.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, and electronic device for determining data standards, in order to at least solve the technical problem that data standard recommendation methods in related technologies usually rely on manual calibration and cannot fully extract the data features of metadata, resulting in low data standardization efficiency.

[0006] According to one aspect of the embodiments of this application, a method for determining a data standard is provided, comprising: obtaining metadata from a data source through a stream processing framework, wherein the metadata includes at least the database schema to which the data belongs and field attributes; performing feature extraction on the metadata to obtain target feature data corresponding to the metadata; generating a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent data relationships in the metadata; and determining a data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to standardize the data in the metadata.

[0007] Optionally, after obtaining metadata from the data source through the stream processing framework, the method further includes: determining the feature extraction logic of the metadata; extracting key data features from the metadata based on the feature extraction logic to obtain first feature data; cleaning the metadata and the first feature data, and combining the cleaned metadata and the first feature data to obtain second feature data.

[0008] Optionally, feature extraction is performed on the metadata to obtain target feature data corresponding to the metadata, including: determining the feature extraction operation of the second feature data, wherein the feature extraction operation includes at least one of the following: missing value handling, feature transformation, feature selection, feature construction, feature dimensionality reduction, feature persistence, and feature structuring; and performing feature extraction on the second feature data according to the feature extraction operation to obtain the target feature data.

[0009] Optionally, generating a knowledge graph corresponding to the metadata based on the target feature data includes: processing the target feature data using a clustering analysis algorithm to obtain clustering analysis results, wherein the clustering analysis results are used to represent groupings of data points with similar features in the target feature data; processing the clustering analysis results using preset association rules to obtain association learning results, wherein the association learning results are used to represent frequent itemsets and association rules in the clustering analysis results; and determining the knowledge graph based on the association learning results, wherein nodes in the knowledge graph are used to represent fields in the association learning results, and edges in the knowledge graph are used to represent field relationships in the association learning results.

[0010] Optionally, before determining the knowledge graph based on the association learning results, the method further includes: determining frequent itemsets and association rules in the association learning results; determining transformation parameters for inverse feature transformation of the association learning results; and performing inverse transformation on frequent itemsets and association rules based on the transformation parameters.

[0011] Optionally, before determining the knowledge graph based on the association learning results, the method further includes: performing data structure adaptation on the association learning results, wherein the data structure adaptation is used to adjust the data structure of the association learning results.

[0012] Optionally, the method further includes: sending the data standard to the target object; receiving confirmation information returned by the target object, or receiving feedback information returned by the target object, and updating the data standard based on the feedback information.

[0013] According to another aspect of the embodiments of this application, a data standard determination apparatus is also provided, comprising: an acquisition module, configured to acquire metadata from a data source through a stream processing framework, wherein the metadata includes at least the database schema to which the data belongs and field attributes; an extraction module, configured to perform feature extraction on the metadata to obtain target feature data corresponding to the metadata; a generation module, configured to generate a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent data relationships in the metadata; and a determination module, configured to determine the data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to standardize the data in the metadata.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and used to execute the method for determining the data standard described above.

[0015] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, the non-volatile storage medium including a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned data standard determination method by running the computer program.

[0016] According to another aspect of the embodiments of this application, a computer program product is also provided, including computer instructions, wherein the computer instructions, when executed by a processor, implement the method for determining the above-mentioned data standard.

[0017] In this embodiment, metadata from a data source is obtained through a stream processing framework. This metadata includes at least the database schema and field attributes to which the data belongs. Feature extraction is performed on the metadata to obtain target feature data corresponding to the metadata. A knowledge graph corresponding to the metadata is generated based on the target feature data, where the knowledge graph represents the data relationships within the metadata. A data standard corresponding to the metadata is determined based on the knowledge graph, where the data standard is a data rule used to regulate the data in the metadata. This achieves automation and intelligence from data feature extraction to data standard formulation, thereby improving the accuracy of data standardization and enhancing the intuitiveness of recommendation results. This solves the technical problem that data standard recommendation methods in related technologies typically rely on manual calibration and cannot fully extract data features from metadata, resulting in low data standardization efficiency. Attached Figure Description

[0018] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0019] Figure 1 This is a hardware structure diagram of a computer terminal for implementing a method for determining data standards according to an embodiment of this application;

[0020] Figure 2 This is a flowchart of a method for determining a data standard according to an embodiment of this application;

[0021] Figure 3 This is a structural diagram of a data standard determination device according to an embodiment of this application. Detailed Implementation

[0022] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0023] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0024] First, some nouns or terms that appear in the explanation of the embodiments of this application shall be interpreted as follows:

[0025] Apache Flink: An open-source framework for large-scale data stream processing, supporting both batch and stream processing for data feature extraction.

[0026] Flink Gelly: A graph computation library in Apache Flink that provides a rich set of graph algorithms and operations, simplifies the development of graph analysis applications, and is used to compute data feature relationships and assist in the generation of data standards.

[0027] Clustering algorithms: An unsupervised learning method that divides samples in a dataset into several clusters, such that samples within the same cluster have high similarity, while samples between different clusters have low similarity. It is used to identify natural groupings in data, thus providing a basis for data classification and standard setting.

[0028] Association rule learning: In large database systems, it rapidly identifies potential and valuable relationships between various entities, represents them as rules, and, through reasoning and accumulation, forms knowledge to draw important conclusions about these relationships. It is used to discover association rules between fields and provides interpretability for generating standards to support decision-making.

[0029] To address the issue of poor data standardization efficiency in related technologies, this application provides a method for determining data standards, which can be implemented in... Figure 1 The computer terminal shown is described below.

[0030] The data standard determination method provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal for implementing a method for determining data standards is shown. Figure 1 As shown, the computer terminal 10 may include one or more processors (shown as 102a, 102b, ..., 102n in the figure) (the processor may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission module 106 for communication functions connected via wired and / or wireless networks. In addition, it may also include: a display, a keyboard, a cursor control device, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, and a BUS bus. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0031] It should be noted that the aforementioned one or more processors and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be wholly or partially integrated into any other element in the computer terminal 10. As involved in the embodiments of this application, the data processing circuits serve as processor control (e.g., selection of a variable resistor termination path connected to an interface).

[0032] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the data standard determination method in this embodiment. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned data standard determination method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0033] The transmission module 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission module 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission module 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0034] The display may be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10.

[0035] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer terminal shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance, and is intended to illustrate the types of components that may exist in the aforementioned computer terminal.

[0036] In the above operating environment, this application provides an embodiment of a method for determining data standards. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0037] Figure 2 This is a flowchart of a method for determining a data standard according to an embodiment of this application, such as... Figure 2As shown, the method includes the following steps:

[0038] Step S202: Obtain metadata from the data source through the stream processing framework. The metadata includes at least the database schema to which the data belongs and field attributes.

[0039] In step S202 above, Apache Flink can be used as a stream processing framework to extract metadata from the data source (such as a database, data lake, etc.). This metadata includes, but is not limited to, the database schema and field attributes (such as data type, length, and whether nullables are allowed). This step forms the basis for data standardization, ensuring that subsequent steps are based on accurate data source information.

[0040] Step S204: Extract features from the metadata to obtain target feature data corresponding to the metadata.

[0041] In step S204 above, the purpose of feature extraction is to identify and extract key information from metadata. This information may include the data type of the field, field length, field uniqueness, and statistical characteristics of the field (such as mean, standard deviation, etc.), which are used for subsequent knowledge graph construction and data standard generation. Feature extraction methods may include, for example, missing value handling, feature transformation, feature selection, feature construction, feature dimensionality reduction, feature persistence, and feature structuring.

[0042] Step S206: Generate a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent the data relationships in the metadata.

[0043] In step S206 above, a knowledge graph can be constructed using target feature data extracted from metadata. A knowledge graph is a graphical data structure used to represent entities (data fields in this scenario) and the relationships between them. In a knowledge graph, each data field is considered a node, and the relationships between fields (such as dependencies, relevance, etc.) are considered edges. Constructing a knowledge graph helps to intuitively display the relationships between data, providing visual support for the development of data standards.

[0044] Step S208: Determine the data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is the data rule used to standardize the data in the metadata.

[0045] In step S208 above, data standards corresponding to the metadata can be determined based on the knowledge graph using relevant graph algorithms. These data standards are a set of data rules used to standardize the format, type, and processing methods of data to ensure data consistency and comparability. In this step, the data relationships displayed in the knowledge graph can be used to identify which fields require unified data standards and which fields have dependencies, thereby formulating corresponding data rules. For example, if two fields show a high degree of correlation in the graph, rules may need to be formulated to ensure that the data format and processing methods of these two fields remain consistent.

[0046] Through steps S202 to S208, the automation and intelligence of data feature extraction and data standard setting are achieved, thereby improving the accuracy of data standardization and enhancing the intuitiveness of recommendation results. This solves the technical problem of low data standardization efficiency caused by the reliance on manual calibration and the inability to fully extract metadata features in related technologies. A detailed explanation follows.

[0047] In step S202 above, after obtaining metadata from the data source through the stream processing framework, the method further includes: determining the feature extraction logic of the metadata; extracting key data features from the metadata according to the feature extraction logic to obtain the first feature data; cleaning the metadata and the first feature data, and combining the cleaned metadata and the first feature data to obtain the second feature data.

[0048] In this embodiment of the application, the data preparation stage includes obtaining metadata from the data source through a stream processing framework (such as Apache Flink) and performing a series of data processing steps to ensure the quality and applicability of the data.

[0049] Specifically as follows:

[0050] 1. Metadata extraction:

[0051] Use Apache Flink to extract necessary metadata from data sources (such as relational databases, NoSQL databases, message queues, etc.), including database schemas, field attributes, etc., to ensure that Flink jobs can adapt to different data sources and data formats, and can be flexibly configured to meet different metadata extraction needs.

[0052] 2. Data Feature Collection:

[0053] For the raw metadata, Flink defines data feature extraction logic to adapt to different data types and business needs, such as calculating the average, maximum, minimum, and standard deviation of fields. Based on this feature extraction logic, key data features, namely the aforementioned first feature data, such as data distribution and statistical summaries, are collected from the metadata.

[0054] Based on different data types (numerical, string, boolean, time series, binary / categorical, text data, spatial data), Flink's AggregateFunction, ProcessFunction, MapFunction, etc., are used for feature extraction, as shown in Table 1.

[0055] Table 1. Methods for extracting the first feature data

[0056]

[0057]

[0058] 3. Data cleaning:

[0059] Define clear data quality rules, such as data format, data range, and data integrity, to guide the data cleaning process.

[0060] Furthermore, Flink's map or flatMap function is used to clean the metadata and extracted first feature data, removing invalid, erroneous, or inconsistent data.

[0061] 4. Integration of data features and metadata:

[0062] Flink is used to manipulate the cleaned metadata and the first feature data to ensure that the data features and the corresponding metadata can be correctly associated. Metadata and data features from different data sources are bound and merged to form a unified, structured dataset, namely the second feature data mentioned above.

[0063] 5. Data Validation:

[0064] The integrated second feature data is validated by adding validation logic to the Flink job, such as checking the consistency and integrity of the dataset, to ensure the quality and applicability of the dataset.

[0065] In step S204 above, feature extraction is performed on the metadata to obtain the target feature data corresponding to the metadata, including: determining the feature extraction operation of the second feature data, wherein the feature extraction operation includes at least one of the following: missing value processing, feature transformation, feature selection, feature construction, feature dimensionality reduction, feature persistence and feature structuring; and performing feature extraction on the second feature data according to the feature extraction operation to obtain the target feature data.

[0066] In this embodiment of the application, feature extraction of the integrated metadata (i.e., the aforementioned second feature data) is a crucial step. It helps to gain a deeper understanding of the data's structure and attributes, thereby providing a more accurate data foundation for data standardization (i.e., the aforementioned target feature data). The following are the detailed steps and methods for feature extraction of the second feature data:

[0067] 1. Handling missing values:

[0068] Identify missing values ​​in the metadata and select an appropriate imputation strategy based on the characteristics of the field. For example:

[0069] Missing values ​​in field length and precision are filled with the mean of the column; missing values ​​in field name and description are filled with the string "Unknown"; missing values ​​in numerical features (such as mean, maximum, minimum) are filled with the median.

[0070] 2. Feature transformation:

[0071] By transforming features, the distribution characteristics of the data can be improved, making it easier for the model to capture the potential patterns in the data.

[0072] For example:

[0073] Standardize numerical features in metadata, such as Z-score standardization, to eliminate the influence of different units of measurement; encode categorical data, such as one-hot encoding or label encoding, to facilitate processing by machine learning models.

[0074] 3. Feature selection:

[0075] Feature selection identifies the features most relevant to the target variable from the original data. For example:

[0076] Statistical methods (such as Pearson correlation coefficient) are used to evaluate the correlation between features and data standards, and features with correlation below a certain threshold are removed, retaining the most relevant subset of features.

[0077] 4. Feature Construction:

[0078] By combining business knowledge and data understanding, new features can be constructed that may better reflect the inherent attributes of the data and business needs. For example:

[0079] The ratio of field length to average value is calculated using length / average_value.

[0080] 5. Feature dimensionality reduction:

[0081] For categorical features containing multiple categories (such as field types), use dimensionality reduction techniques such as principal component analysis (PCA) and t-SNE to reduce the number of features while preserving as much information as possible from the original data, thereby improving data processing efficiency and model performance.

[0082] 6. Feature persistence:

[0083] During feature encoding, the encoding mapping relationship is recorded, and a mapping table is used to store the original values ​​before and after one-hot encoding of the label. For example:

[0084] The feature_storage_mapping table is used to store the mappings during the feature engineering process. Its data structure is as follows:

[0085] FeatureID(INT,PK): Feature ID;

[0086] TableID(INT,FK->tables.TableID): The ID of the table to which it belongs (the structure of the tables is omitted);

[0087] FieldID(INT,FK->fields.FieldID): The ID of the field to which it belongs (the structure of the fields table is omitted);

[0088] FeatureName(VARCHAR): Feature name;

[0089] FeatureValue(VARCHAR): Feature encoding value;

[0090] FeatureDataType(VARCHAR): Feature data type, such as numeric, string, etc.;

[0091] TransformParamID(INT,FK->fields.FieldID): Uses the transformation parameter ID.

[0092] After feature transformation and encoding, save the transformation type (e.g., mean, standard deviation, encoding mapping, feature construction, PCA feature reduction, Z-score, etc.) and specific parameters (JSON) for subsequent analysis. For example:

[0093] The feature transformation log is stored using feature_transform_log, and its data structure is as follows:

[0094] TransformLogID(INT,PK): Transform Log ID;

[0095] FeatureID(INT,FK): Feature ID (foreign key);

[0096] TransformDesc(VARCHAR): Feature transformation summary;

[0097] TransformType(VARCHAR): The conversion type;

[0098] TransformParams(JSON): Transformation parameters;

[0099] OriginalValue(VARCHAR): Original value of the feature.

[0100] 7. Feature structuring:

[0101] The extracted features are organized into a structured form, such as tables or vectors, to facilitate subsequent data processing and model training, thereby ensuring the consistency and integrity of the features to support the data-standardized decision-making process.

[0102] In step S206 above, generating a knowledge graph corresponding to the metadata based on the target feature data includes: processing the target feature data using a clustering analysis algorithm to obtain clustering analysis results, wherein the clustering analysis results are used to represent the grouping of data points with similar features in the target feature data; processing the clustering analysis results using preset association rules to obtain association learning results, wherein the association learning results are used to represent frequent itemsets and association rules in the clustering analysis results; and determining the knowledge graph based on the association learning results, wherein the nodes in the knowledge graph are used to represent the fields of the association learning results, and the edges in the knowledge graph are used to represent the field relationships of the association learning results.

[0103] Before determining the knowledge graph based on the association learning results, the method also includes: determining the frequent itemsets and association rules in the association learning results; determining the transformation parameters for inverse feature transformation of the association learning results; performing inverse transformation of the frequent itemsets and association rules based on the transformation parameters; and performing data structure adaptation on the association learning results, wherein the data structure adaptation is used to adjust the data structure of the association learning results.

[0104] In this embodiment, through clustering algorithm analysis, association rule mining, feature inverse transformation, and data structure adaptation, complex data features and relationships in the target feature data can be transformed into a knowledge graph that is easy to understand and manipulate, thereby providing a more effective tool for data analysis and decision-making. The specific process is as follows:

[0105] I. Cluster Analysis

[0106] Cluster analysis is an important technique in data mining. It aims to divide samples in a dataset into several clusters, ensuring high similarity among samples within the same cluster and low similarity among samples in different clusters. In the process of data standardization, cluster analysis can help us identify natural groupings in the data, thus providing a basis for data classification and standardization. Specifically:

[0107] First, select a suitable clustering algorithm based on the characteristics of the target feature data, such as K-means or hierarchical clustering. Among them, the K-means algorithm is simple and efficient, and suitable for large-scale datasets; hierarchical clustering can discover the hierarchical structure of the data and is suitable for smaller datasets.

[0108] Secondly, determine the appropriate number of clusters. For example, the elbow rule or silhouette coefficient can be used to determine the optimal number of clusters. The elbow rule determines the number of clusters by observing the decreasing trend of intra-cluster dissimilarity, while the silhouette coefficient determines the number of clusters by evaluating intra-cluster similarity and inter-cluster dissimilarity.

[0109] Furthermore, through multiple iterations, the clustering parameters and algorithms are continuously adjusted to obtain the optimal clustering analysis results, which are used to represent the grouping of data points with similar features in the target feature data.

[0110] Finally, the clustering analysis results are stored in a database, such as the `field_cluster_mapping` table, for subsequent data standardization and analysis. For example, the structure of the `field_cluster_mapping` table is as follows:

[0111] ClusterID(INT,PK): Cluster ID, primary key;

[0112] ClusterName(VARCHAR): Cluster name, used to describe the characteristics of the cluster;

[0113] Fields(List[INT]): A list of clustered field IDs, storing field IDs belonging to the same cluster;

[0114] Features(List[String]): Uses a list of feature IDs to store the field feature IDs used for clustering.

[0115] Through the clustering analysis process described above, data fields with similar characteristics can be grouped, providing a scientific basis for the formulation of data standards and improving the efficiency and quality of data governance.

[0116] II. Association Rule Mining

[0117] Within each cluster of the above clustering analysis results, association rule mining is applied. For example, the Apriori or FP-Growth algorithms are used to mine association rules between fields, finding frequent itemsets and meaningful association rules between field feature values ​​to obtain association learning results, which are used to represent more specific and accurate associations. The specific process can be as follows:

[0118] By querying the FeatureDataType and FeatureValue (encoded results, not the original values) from the Feature_storage_mapping table using the Features field in the field_cluster_mapping table, a dataset for mining association rules is constructed, as follows:

[0119] [{"avg:1","count:5","null_count:2",……},

[0120] {"avg:1","count:5","null_count:1",……},

[0121] {"avg:1","count:4","null_count:1",……},

[0122] {"avg:0","count:5","null_count:2",…},…]

[0123] By setting the minimum support and minimum confidence, frequent itemsets can be output, as follows:

[0124] [{'avg:1'},{'count:5'},{'avg:1','count:5'},{'avg:1','null_count:1'},……]

[0125] It also outputs association rules along with the lift of the results. Through automatic iterative analysis, it identifies the optimal association rules and writes the antecedents, consequents, support, confidence, and lift of the associations into storage, as follows:

[0126] {'avg:1'}->{'count:5'}(support=0.6,confidence=1.0,lift=1.67)

[0127] {'avg:1'}->{'null_count:1'}(support=0.6,confidence=1.0,lift=1.67)

[0128] {'avg:1','count:5'}->{'null_count:1'}(support=0.4, confidence=0.67, lift=2.22)……

[0129] For example, the `field_rule_mapping` table stores the mapping relationship between clustering results and association rule mining results. Its data structure is as follows:

[0130] RuleId(INT,PK): Rule ID, primary key, used to uniquely identify each rule. The integer type ensures the uniqueness and auto-incrementing nature of the rule ID.

[0131] ClusterID(INT,FK–>field_cluster_mapping.ClusterID): Cluster ID, used to associate a specific cluster;

[0132] Fields(List[INT]): Associated field ID (fields table structure omitted);

[0133] Antecedent (VARCHAR): Rule antecedent;

[0134] Consequent (VARCHAR): Rule consequent;

[0135] Support (FLOAT): The degree of support for the rule;

[0136] Confidence (FLOAT): The confidence level of a rule;

[0137] Lift (FLOAT): The degree of lift of a rule.

[0138] III. Feature Reverse Conversion

[0139] After completing cluster analysis and association rule mining, the association learning results are reverse-transformed to restore the feature data to its original state. Specifically:

[0140] 1. Feature restoration of fields within clusters:

[0141] Retrieve all field IDs within the same cluster from the `field_cluster_mapping` table. For these fields, query the `feature_storage_mapping` table using the ID from the `Fields` field to find the corresponding feature type. Use the transformation parameters (such as mean, standard deviation, encoding mapping, etc.) from the `feature_transform_log` table to reverse-transform the feature values ​​back to their original values.

[0142] 2. Feature restoration in association rules:

[0143] Retrieve detailed information about the association rules from the `field_rule_mapping` table, including the rule ID and the ID of the associated field. For each association rule, query the `feature_storage_mapping` table using the ID from the `Fields` field to find the feature types of the rule's antecedent and consequent. Use the transformation parameters from the `feature_transform_log` table to reverse-transform these feature values ​​back to their original values.

[0144] 3. Store the reverse conversion result:

[0145] Create a new table, `rule_original_value`, to store the results of the reverse transformation. The table structure includes: `RuleOriginalId` (primary key), `RuleOriginalType` (association type, 0 for frequent items, 1 for association rule), `RuleId` (foreign key, associated with `field_rule_mapping.RuleId`), `FieldTuple` (tuple of associated field IDs), `FeatureID` (foreign key, associated with feature IDs), and `OriginalValue` (original value of the feature).

[0146] IV. Database Structure Adaptation

[0147] In the data structure adaptation phase, the goal is to ensure that the data, after feature inverse transformation, can seamlessly integrate with knowledge graph construction and subsequent analysis processes. The core tasks of this phase include adjusting the data format to adapt to graph databases or other storage solutions, and processing different types of data features through built-in summary generation rules to facilitate the subsequent generation of descriptive information such as rule summaries and feature dimensions. Specifically:

[0148] 1. Data format standardization:

[0149] Identify the data formats supported by the graph database and convert the reverse-engineered data into these formats.

[0150] 2. Application of summary generation rules:

[0151] Based on the type of data characteristics (such as numerical, textual, time series, etc.), select an appropriate summary generation strategy and generate summaries according to the strategy. These summaries should accurately reflect the key information of the data characteristics.

[0152] 3. Feature dimension mapping:

[0153] Each data feature is mapped to a node or edge in the knowledge graph to ensure that the dimensionality and descriptive information of the feature are preserved during the mapping process.

[0154] 4. Data storage adaptation:

[0155] Store the formatted data in a graph database or other storage solution to ensure that the data storage structure supports efficient query and analysis operations.

[0156] V. Knowledge Graph Construction

[0157] By transforming the above association learning results into a graphical structure, the relationships and features between data can be made more intuitive and easier to understand. Specifically:

[0158] 1. Clustering results are converted into nodes:

[0159] Node definition: Each data field is represented as a node, and the attributes of a node include basic information such as the field's name, data type, and description.

[0160] Clustering mapping: The clusters obtained by the clustering algorithm are mapped to groups of nodes in the graph. Field nodes in each cluster are grouped together to represent their similarity or correlation in data features.

[0161] 2. Association rules are transformed into edges:

[0162] Edge definition: The relationship between fields is represented by edges. The attributes of an edge include the type of association rule, support, confidence, and lift.

[0163] Rule mapping: Based on the rules obtained by the association rule learning algorithm, fields with high confidence and support are connected by edges. The attribute values ​​of the edges reflect the strength and reliability of the rules.

[0164] 3. Visual representation of attribute values:

[0165] Node size: The size of a node can be adjusted based on the support of the association rule. Rules with higher support will have larger nodes to highlight their importance in the data.

[0166] Edge thickness: The thickness of the edges can be adjusted based on the confidence level of the association rule. Rules with higher confidence levels correspond to thicker edges, indicating a stronger association.

[0167] Color coding: Different colors can be used to represent different types of association rules or field features, increasing the readability and intuitiveness of the graph.

[0168] 4. Optimization and adjustment of the atlas:

[0169] Layout optimization: Use graph theory algorithms (such as force-directed layout) to optimize the layout of nodes and edges, making the graph clearer and easier to understand.

[0170] Interaction design: Add interactive features to the front-end display of the graph, such as clicking on nodes to view detailed information and dragging nodes to adjust the view, to enhance the user experience.

[0171] The application of knowledge graphs not only enhances the visualization of data but also improves the efficiency and accuracy of data standard setting, making data governance more scientific and systematic.

[0172] In step S208 above, after determining the data standard corresponding to the metadata based on the knowledge graph, the method further includes: sending the data standard to the target object; receiving confirmation information returned by the target object, or receiving feedback information returned by the target object, and updating the data standard based on the feedback information.

[0173] In this embodiment of the application, the method for formulating intelligent knowledge graph visualization-assisted data standards can be implemented through the following steps:

[0174] 1. Graph Algorithm Development:

[0175] Flink Gelly was used to analyze the graph data, extracting key information from cluster analysis and association rule mining, and further filtering out potential data standards.

[0176] 2. Visualization of recommendation criteria:

[0177] The graphical user interface (GUI) visually presents data standard recommendations, including core field standards and related fields, as well as the visual relationships between them. This visual representation (such as size, color, and distance) helps users understand the importance and relevance between fields.

[0178] 3. Display of association rules:

[0179] The aforementioned data standards are sent to the target object, that is, the specific association rules and their confidence and support scores are displayed on the user's (i.e. the target object's) interface, such as the "unique value quantity matching rule", and detailed data feature dimensions and data standard suggestions are provided, as shown in Table 2.

[0180] Table 2 Association rules and their confidence and support scores

[0181]

[0182]

[0183] 4. User feedback and evaluation:

[0184] It receives user confirmation information or collects and analyzes user feedback on recommendation data standards. This feedback information is used to optimize and adjust the recommendation model to improve the accuracy and applicability of future recommendations.

[0185] 5. Data standard confirmation:

[0186] Users can review the recommended data standards, accept them, or update or modify them as needed, and then officially save and implement them.

[0187] 6. Log auditing:

[0188] Recording all recommended data standards and their modification history to ensure transparency and auditability throughout the process is crucial for data governance and compliance.

[0189] The aforementioned interactive process not only improved the automation and intelligence of data standardization but also enhanced the applicability and acceptability of the standards through user feedback mechanisms. Specifically, graphical displays and intelligent recommendations made the data standardization process more intuitive and user-friendly, while maintaining a high degree of flexibility and customizability.

[0190] In this embodiment, by constructing an intelligent knowledge graph, deeply extracting and analyzing data features, applying association rule learning algorithms, employing graph theory algorithms to optimize the decision-making process, developing an automated data standard recommendation process, and designing user interaction and feedback mechanisms, the automation, intelligence, and efficiency of data standard formulation are achieved. The integrated application of these key technologies not only improves the interpretability, adaptability, and accuracy of data standards but also reduces human intervention, enhances the logic and intuitiveness of decision-making, and improves decision-making efficiency. Simultaneously, through user interaction and feedback mechanisms, the recommendation algorithm is continuously optimized, ensuring the practicality of data standards and user satisfaction, and promoting the progress of data governance practices.

[0191] According to embodiments of this application, a data standard determination apparatus is provided. It should be noted that the data standard determination apparatus of this application can be used to execute the data standard determination method provided in the embodiments of this application. The data standard determination apparatus provided in the embodiments of this application will be described below.

[0192] Figure 3 This is a structural diagram of a data standard determination device provided according to an embodiment of this application. Figure 3 As shown, the device includes:

[0193] The acquisition module 30 is used to acquire metadata from the data source through the stream processing framework. The metadata includes at least the database schema to which the data belongs and field attributes.

[0194] Extraction module 32 is used to extract features from metadata to obtain target feature data corresponding to the metadata;

[0195] The generation module 34 is used to generate a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent the data relationships in the metadata;

[0196] Module 36 is used to determine the data standard corresponding to the metadata based on the knowledge graph. The data standard is a data rule used to standardize the data in the metadata.

[0197] Through the acquisition module 30, extraction module 32, generation module 34 and determination module 36 in the aforementioned data standard determination device, the goal of automation and intelligence from data feature extraction to data standard formulation is achieved. This realizes the technical effects of improving the accuracy of data standardization and enhancing the intuitiveness of recommendation results. In turn, it solves the technical problem that data standard recommendation methods in related technologies usually rely on manual calibration and cannot fully extract the data features of metadata, resulting in low data standardization efficiency.

[0198] In the data standard determination device provided in this application embodiment, the extraction module is further used to determine the feature extraction logic of metadata; extract key data features in the metadata according to the feature extraction logic to obtain first feature data; clean the metadata and the first feature data, and combine the cleaned metadata and the first feature data to obtain second feature data.

[0199] In the data standard determination device provided in this application embodiment, the extraction module is further used to determine the feature extraction operation of the second feature data, wherein the feature extraction operation includes at least one of the following: missing value processing, feature transformation, feature selection, feature construction, feature dimensionality reduction, feature persistence, and feature structuring; and to extract features from the second feature data according to the feature extraction operation to obtain target feature data.

[0200] In the data standard determination apparatus provided in this application embodiment, the generation module is further configured to process the target feature data through a clustering analysis algorithm to obtain clustering analysis results, wherein the clustering analysis results are used to represent the grouping of data points with similar features in the target feature data; process the clustering analysis results through preset association rules to obtain association learning results, wherein the association learning results are used to represent frequent itemsets and association rules in the clustering analysis results; and determine a knowledge graph based on the association learning results, wherein the nodes in the knowledge graph are used to represent the fields of the association learning results, and the edges in the knowledge graph are used to represent the field relationships of the association learning results.

[0201] In the data standard determination device provided in this application embodiment, the generation module is further used to determine the frequent itemsets and association rules in the association learning results; determine the transformation parameters for performing feature inverse transformation on the association learning results; and perform inverse transformation on the frequent itemsets and association rules according to the transformation parameters.

[0202] In the data standard determination device provided in the embodiments of this application, the generation module is further used to perform data structure adaptation on the association learning results, wherein the data structure adaptation is used to adjust the data structure of the association learning results.

[0203] In the data standard determination device provided in the embodiments of this application, the determination module is further configured to receive confirmation information returned by the target object, or receive feedback information returned by the target object, and update the data standard based on the feedback information.

[0204] This application also provides an electronic device, including: a memory and a processor, wherein the memory is used to store program instructions; and the processor is connected to the memory and used to execute the method for determining the data standard described above.

[0205] It should be noted that the aforementioned electronic equipment is used to perform Figure 2 The data standard determination method shown above is also applicable to this electronic device, and will not be repeated here.

[0206] This application also provides a non-volatile storage medium including a stored computer program, wherein the device containing the non-volatile storage medium executes the above-mentioned data standard determination method by running the computer program.

[0207] It should be noted that the aforementioned non-volatile storage media is used for execution. Figure 2 The method for determining the data standard shown above is also applicable to this non-volatile storage medium, and will not be repeated here.

[0208] This application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the method for determining the above-mentioned data standard.

[0209] It should be noted that the above-mentioned computer program product is used to execute... Figure 2 The method for determining the data standard shown above is also applicable to this computer program product, and will not be repeated here.

[0210] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0211] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0212] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0213] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0214] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0215] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0216] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for determining data standards, characterized in that, include: Metadata from a data source is obtained through a stream processing framework. The metadata includes at least the database schema and field attributes to which the data belongs. The data source includes at least one of the following: relational database, non-relational database, and message queue. Feature extraction is performed on the metadata to obtain target feature data corresponding to the metadata; A knowledge graph corresponding to the metadata is generated based on the target feature data, wherein the knowledge graph is used to represent the data relationships in the metadata; Based on the knowledge graph, a data standard corresponding to the metadata is determined, wherein the data standard is a data rule used to regulate the data in the metadata; Generating a knowledge graph corresponding to the metadata based on the target feature data includes at least: performing cluster analysis on the target feature data to obtain cluster analysis results; performing association rule mining on the cluster analysis results to obtain association learning results, wherein the association learning results are used to represent frequent itemsets and association rules in the cluster analysis results; performing feature inverse transformation on the association learning results, and generating a knowledge graph corresponding to the metadata.

2. The method according to claim 1, characterized in that, After obtaining metadata from the data source through a stream processing framework, the method further includes: Determine the feature extraction logic for the metadata; Based on the feature extraction logic, key data features are extracted from the metadata to obtain the first feature data; The metadata and the first feature data are cleaned, and the cleaned metadata and the first feature data are combined to obtain the second feature data.

3. The method according to claim 2, characterized in that, Feature extraction is performed on the metadata to obtain target feature data corresponding to the metadata, including: The feature extraction operation for the second feature data is determined, wherein the feature extraction operation includes at least one of the following: missing value handling, feature transformation, feature selection, feature construction, feature dimensionality reduction, feature persistence, and feature structuring; The target feature data is obtained by performing feature extraction on the second feature data according to the feature extraction operation.

4. The method according to claim 1, characterized in that, Generate a knowledge graph corresponding to the metadata based on the target feature data, including: The target feature data is processed by a clustering analysis algorithm to obtain the clustering analysis result, wherein the clustering analysis result is used to represent the grouping of data points with similar features in the target feature data; The clustering analysis results are processed by preset association rules to obtain the association learning results; The knowledge graph is determined based on the association learning results, wherein the nodes in the knowledge graph are used to represent the fields of the association learning results, and the edges in the knowledge graph are used to represent the field relationships of the association learning results.

5. The method according to claim 4, characterized in that, Before determining the knowledge graph based on the association learning results, the method further includes: Determine the frequent itemsets and association rules in the association learning results; Determine the transformation parameters for inverse feature transformation of the association learning results; The frequent itemsets and the association rules are reverse-transformed according to the transformation parameters.

6. The method according to claim 4, characterized in that, Before determining the knowledge graph based on the association learning results, the method further includes: The association learning results are subjected to data structure adaptation, wherein the data structure adaptation is used to adjust the data structure of the association learning results.

7. The method according to claim 1, characterized in that, The method further includes: Send the data standard to the target object; Receive confirmation information returned by the target object, or receive feedback information returned by the target object, and update the data standard based on the feedback information.

8. A data standard determination device, characterized in that, include: The acquisition module is used to acquire metadata from a data source through a stream processing framework. The metadata includes at least the database schema and field attributes to which the data belongs. The data source includes at least one of the following: a relational database, a non-relational database, and a message queue. The extraction module is used to extract features from the metadata to obtain target feature data corresponding to the metadata; A generation module is used to generate a knowledge graph corresponding to the metadata based on the target feature data, wherein the knowledge graph is used to represent the data relationships in the metadata; generating a knowledge graph corresponding to the metadata based on the target feature data includes at least: performing cluster analysis on the target feature data to obtain cluster analysis results; performing association rule mining on the cluster analysis results to obtain association learning results, wherein the association learning results are used to represent frequent itemsets and association rules in the cluster analysis results; performing feature inverse transformation on the association learning results, and generating a knowledge graph corresponding to the metadata; The determination module is used to determine the data standard corresponding to the metadata based on the knowledge graph, wherein the data standard is a data rule used to standardize the data in the metadata.

9. An electronic device, characterized in that, include: A memory and a processor, wherein the memory is used to store program instructions; The processor, connected to the memory, is used to execute a method for determining the data standard according to any one of claims 1 to 7.

10. A non-volatile storage medium, characterized in that, The non-volatile storage medium includes a stored computer program, wherein the device containing the non-volatile storage medium executes the method for determining the data standard according to any one of claims 1 to 7 by running the computer program.

11. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method for determining the data standard as described in any one of claims 1 to 7.

Images